How Community Judging Works
Upon the resolution of the submission period of a competitive audit, the judging phase will commence. At this point all eligible community judges for a contest will receive an email notification advising them of the beginning of the judging phase and their eligibility to participate.
By navigating to CodeHawks, users can see a 'Judging' option available in the menus at the top of the page if they're eligible.
Judging Dashboard
From here, a judge will be able to see any contests for which they are eligible to judge. We also see a Start and End date. The Community Judging period will be time locked before submissions are handed to a Lead Judge for final validation.
The duration of this period will be determined by a number of factors including, but not limited to:
Number of submissions
Number of eligible judges
By hovering over the eligibility information icon, a judge can see the requirements to be eligible to participant in each contest's judgement.
See Eligibility for more information.
Community Judging Submission View
Clicking 'Judge' will bring one to the Community Judging submission view. The community Judge will be faced with the Community Judging view. In this view, a judge will be randomly assigned 1 submission at a time, with another being assigned after each judgement. There is no limit to the number of submissions a community judge can complete.
A community judge's role is to determine if each submission assigned is valid or invalid and categorize them as such using the radial buttons available in the top right.
Following the conclusion of the community judging period, an assigned Lead Judge will assess the validity of judgements and ultimately determine the performance of community judges.
Performance
A community judge's performance is determined by comparing the accuracy of their submissions versus the Lead Judge's final verdicts.
A community judge will receive +1 point if their judgement matches that of the Lead Judge, and -1 point if the assessments do not match. This simple system provides a grade of how accurate a community judge's assessments are and ultimately determines rewards for participants.
For example:
Community Judge determines 10 valid findings and 5 invalid
Lead Judge's verdict matches 8 of CJs valid and 4 of the CJs invalid judgements.
The equation looks like:
Valid Judgement Matches - Valid Judgement Mismatches + Invalid Judgement Matches - Invalid Judgement Mismatches
For example: 8 - 2 + 4 - 1 = 9
The community judge in this example would receive 9 points for their 15 judged submissions.
Last updated
