Crowd-sourced competitions and benchmarks can be used to tackle virtually any machine learning problems, such as, among others, classification, regression, reinforcement learning and automated machine learning. Any field of application can be studied too. The main requirement is to define a quantitative metric that can be used to rank participants’ submissions.

Although we are platform-agnostic, we strongly recommend using Codabench.

The time needed to organize a challenge greatly vary from one project to another. An usual time is around 3 months for the preparation, and around 2 months of open participation phase.

The prize pool destinated to the winners of the challenge are usually founded by sponsors or by the organizers.

A competition is an event limited in duration, where participants compete for a prize. A benchmark is a cooperative process aiming at identifying the best methods to solve a problem. While the objectives are different, the practical process is fairly similar in both cases.

The cost of our service can vary depending on the work time required by the project. For instance, a simple result submissions competition on Codabench’s main server is quicker to organize than a code submissions competition on a dedicated server. Please reach us to discuss your own project.