At Wikimania 2026 in Paris I gave a short community lightning talk called How We Judge: Building Trust in Wikimedia Campaign Evaluation. A lightning talk leaves room for one idea and a few examples. This post is the longer version, with the experience behind it.
Where the talk came from
Over the last two years I have sat on both sides of the jury table.
As an organiser, I was National Lead for the Wiki Science Competition 2025 in India, where I set up the Montage campaigns and managed a jury process for 3,706 entries across two tracks.
As a juror, I have judged Wiki Loves Folklore 2025 in India, the Wiki Loves Folklore 2026 international round, the Wiki Loves Monuments international jury, Wiki Loves Birds Philippines, and Wiki Loves Earth 2026 in the Philippines.
Seeing the same process from both sides made one thing obvious to me. The jury is not only selecting winners. It is deciding whether photographers come back next year.
The core argument: trust outlasts prizes
Most Wiki Loves prizes are modest. People enter because they love photographing their monuments, festivals, birds or landscapes, and because they want those images to be used. If they come to believe the judging is careless, inconsistent or decided by familiarity, they stop entering. A campaign can survive a small prize fund. It cannot survive a jury that photographers do not trust.
Trust is built from ordinary, visible practices. These are the five I think matter most.
1. Published criteria, applied the same way
The criteria should be on the campaign page before uploads open, not only in the jurors' instructions. Wiki Loves Earth Philippines told jurors exactly what to weigh: technical quality, originality and usefulness on Wikipedia. When the criteria are public, a photographer who did not win can at least see what the jury was looking for.
The harder part is applying them consistently. Over hundreds of images, a juror's standard drifts with fatigue and mood. Good juries plan for that: shorter judging sessions, full-size viewing before any decision, and a second look at early decisions once the full range of the batch is known.
2. Several independent eyes
No single juror's taste should decide an outcome. In a well-designed process each image is seen by several jurors who cannot see each other's votes. In the Wiki Loves Birds Philippines jury, the second round ensured that every qualifying image reached at least five jurors.
Round design matters too. A yes/no screening round, then ratings, then a final ranking, narrows the field while keeping each step simple enough to do carefully.
3. Mixed juries and fair comparisons
A local juror knows which sites are rare and which are photographed every year. An outside juror brings a consistent standard and no local loyalties. The best juries have both.
Fair comparison also means comparing like with like. For the Wiki Science Competition in India we split the jury into two panels, one for phone photographs and one for camera photographs. A student with a phone should never be measured against a DSLR with a macro lens. Our jury also came from three countries, India, the United States and Nigeria, which reduced the chance of the result reflecting one circle's taste.
4. Handling tool failures fairly
Jury tools are built and maintained by volunteers, and they fail. I have had Montage show a round as finished while 30 of my images were still unrated, and I have reported loading and performance problems in another jury tool during Wiki Loves Folklore.
What matters is the response. The fairest responses I have seen widen the next round, so that no entry loses its chance because of a bug. The worst possible response is a silent workaround that nobody documents. Jurors also have a duty here: report problems early, clearly and to the organiser, not in a side chat.
5. Discretion
Results are public. Deliberations are not. Jurors should never discuss individual votes, never hint at who scored what, and never comment publicly on specific entries during judging. This protects photographers from feeling targeted and protects jurors from pressure. It is also why I write about my jury work only in general terms.
What I look for as a juror
For each image I ask three plain questions.
Is it technically sound at full size? Focus, exposure, noise and framing, not just how it looks as a thumbnail.
Would an article be better with it? A clear photograph of a little-documented species or monument often teaches more than a dramatic but generic one.
Does the file page do its job? A precise title, a real description and the right categories turn a good photograph into usable knowledge.
The third question is the one photographers most often skip, and the easiest to fix.
What organisers can do tomorrow
A short checklist I now use for any campaign:
Recruit jurors before uploads close, not after.
Run a test round in the jury tool so first-time jurors learn the interface on practice images.
Keep a buffer of two weeks in the judging schedule. Something always slips.
Publish the criteria and the round structure on the campaign page from the first day.
Decide in advance how tool failures will be handled, and say so to the jury.
Why I keep doing it
Judging is unpaid, slow and mostly invisible. But every round I finish, I have looked carefully at places and lives I would otherwise never see, from Indian folk festivals to Philippine reefs. And every published result tells photographers that someone took their work seriously. That is worth the evenings it takes.