rankmydoodle00WHAT THE RESEARCH SAYS
A score is a judgement, not a measurement.
Studies of song contests, figure skating, wine medals and orchestra auditions show the same thing: when people rank creative work, the order, the judge and what they can see all leave a mark. A few simple habits make a contest fairer.
What tips a score01Principles
02THE SHORT ANSWER
Three ways a ranking can go wrong,
each shown in published research.
RUNNING ORDER
In the Eurovision Song Contest and figure skating contests, ratings increased with serial position: later performers earned higher marks.
What tips a scoreCONSISTENCY
When a wine competition slipped the same wine to judges three times, only about 10 percent scored it within one medal group each time.
See the numbersPAIRS
Comparative judgement asks judges only which of two is better. A 2025 analysis of 101 datasets found ten comparisons per item is generally enough.
Run a fair contest03A QUICK SCORE CHECK
Pick a situation to see what research found. The bar shows how much each one has been shown to sway or steady a ranking.
Findings come from published studies of song contests, figure skating, a US wine competition, orchestra auditions and comparative judgement datasets.
04TWO NUMBERS WORTH KNOWING
One figure on judges' consistency,
one on what happens when names are hidden.
≈ 10%
Judges who repeated their own score
Between 65 and 70 expert judges a year were tested from 2005 to 2008 with the same wine poured three times in a flight of 30. About 10 percent stayed within a single medal group.
WINE JUDGES · HODGSON 2008
50%
Rise in a woman's chance of advancing
Goldin and Rouse found the audition screen increased by 50% the probability a woman would be advanced out of certain preliminary rounds.
BLIND AUDITIONS · NBER 1997
Hodgson also found judges were more consistent about what they did not like than what they did, and only about half of panels gave awards based solely on wine quality.
05FOUR HABITS FOR A FAIRER CONTEST
None of these needs special software. They borrow from the research on order effects, blind judging and comparative judgement.
Randomising cannot remove order effects, Bruine de Bruin notes, but it gives every entry an equal chance of a good slot.
Number the entries and keep names out of sight, as orchestras did with an audition screen.
Show judges two entries at a time instead of asking for a score out of ten, and give each entry around ten comparisons.
Even expert judges rarely repeat their own marks. Pool several judges' views rather than relying on a single one.
06COMPARATIVE JUDGEMENT
People often interpret a 'score out of ten' differently. Comparative judgement avoids that by asking only which of two pieces is better, then fitting a statistical model, usually the Bradley–Terry model, to all the choices to give each piece a score.
Researchers have used it for samples of students' writing, cold-brew coffee and even images taken by spacecraft. It is especially useful for qualities that are hard to pin down, which is exactly what judging a drawing is.
07Questions
In Bruine de Bruin's study of the Eurovision Song Contest and figure skating, ratings increased with serial position, so later performers tended to score higher.
Not fully. The study says randomisation cannot reduce order effects, but it does give every candidate an equal chance of a preferred position.
Not always. In a US wine competition, only about 10 percent of judges scored the same wine within one medal group across three blind pourings.
A 1997 NBER working paper found an audition screen increased by 50% the probability a woman advanced out of certain preliminary rounds, and made her several times likelier to win the final.
A 2025 meta-analysis of 101 comparative judgement datasets found that collecting ten comparisons for every item is generally sufficient.
08Sources
ABOUT THIS NAME
rankmydoodle.com is part of the DoodleSale portfolio. It isn't in an auction right now, but you can ask about leasing it.
