This software service enables you to configure a benchmark challenge and make it available for participants to submit their method prepared in GitHub to be run on the holdout data. You can publish the challenge leaderboard both blinded or open on the Next platform.
Features
The following pages are optional:
About page to inform participants about your challenge
Consent page for informed consent
Helpdesk page
Privacy statement (pdf file upload)
When the above pages are added, these are all visible for participants when clicking the "i" at the bottom corner. The about and consent pages are part of the onboarding of respondents and will also be shown before respondents enter the data donation workflow.
Add custom branding to your benchmark, by changing the header image and adding a logo, title and subtitle.
Add tasks to your benchmark challenge, the following tasks are available. You can add multiple tasks and edit the title of a task any time:
Prepare method: provide a link to the template GitHub repository that participants need to use to prepare their method for submission (see example repo).
Download data: upload a zip file with the data that participants can use to tune or train their method.
Submit method: add a method submission task to enable participants to submit their method for your challenge and set the submission deadline. After the deadline passed the submission will be automatically closed. When participants enter the submission task, they see the deadline in their own time zone.
Each submit method task has the option to create a leaderboard with the submissions. A card is then added to your project with the list of submissions that can be exported to run on the holdout data. Once the methods are run on the holdout data the leaderboard can be published either blinded or open. With an open leaderboard participants can see eachothers submissions with corresponding GitHub links. With a blinded leaderboard, participants see the scores of other participants, but the names and GitHub links are blinded. They do see their own submission on the leaderboard.
Get an invite link to your benchmark challenge to invite participants to join the challenge.
Monitor how many participants started and finished the tasks.
Since benchmark challenges are typically international the default language is set to English. All features can be configured in the content management system (CMS) on the Next platform.
Content Management System (CMS) for benchmark challenges on the Next platform.
What can we do for you?
Publish a benchmark challenge on the Next platform: By creating an account on the Next platform, you can easily configure your benchmark challenge in the Next platform CMS and publish it on our high-availability environment by simply clicking publish, after this it will be readily available for your participants. We take care of the availability and security.
Support with developing your benchmark challenge template repository: To be able to run the submitted methods on the holdout data, we use a docker setup. A template GitHub repository needs to be developed that participants can fork to prepare their method for submission(see example repo). We can support you in developing the template repository.
Advise on how to set up a benchmark challenge: If you are not sure how to set up a benchmark challenge. Contact us to see what we can do for you.
Benchmark challenge with LISS panel data: Since we are a trusted partner of Centerdata you can use the Next platform to organize a challenge with LISS panel data in collaboration with Centerdata.
Pricing
The benchmark challenge software service is currently subject to custom pricing, depending on your requirements. Interested? Fill out our contact form and we will get back to you to schedule an appointment.
Collaboration and Funding
The benchmark challenge software service was mostly developed in collaboration with:
Tom Emery and Paulina Pankowska (ODISSEI)
Gert Stulp and Elizaveta Sivak (University of Groningen)
Joris Mulder (Centerdata)
This software service is mostly funded by ODISSEI, University of Groningen and Eyra. For an example of a challenge that used this software service visit the PreFer challenge website.
