How to Judge a Hackathon

At the end of a hackathon, when hacking is done, and presentations have just finished, there’s a pause, and a giddy excitement fills the air. The judges leave to some private room and begin to talk about which hacks were the best and which deserve prizes. As an evangelist, I’ve taken part in many of these sessions, and sat in on many more. I’ve seen great judging sessions, and terrible ones. At the end of the day, everyone wants the right person to win, but selecting the winner can get tricky. However, if the judges are picked well and good criteria can be established, this decision becomes easier.

What Makes a Good “Hack”

To understand what makes a good hack, you must determine the purpose of a hackathon. At the end of the day, a hackathon is meant to bring developers together and have them build something. Hopefully, the developers learn something, meet new people, and have a good time in the process. By understanding that, determining what makes a good hack becomes clear: it’s something that helps developers achieve the purpose of the hackathon. To me, a good hack is one that is technically impressive and receives some sort of exclamation after seeing it, “nifty!” That’s it.

With what makes a good hack in mind, the criteria becomes simple:

• Technical Difficulty
• Impressiveness
• If your hackathon has a theme, you may also have the theme as part of the criteria.

Frequently, organizers will add additional criteria, that can prevent the right person winning first prize. This criteria includes:

• Usefulness/Practicality – The purpose of a hackathon is not to create useful software, and some of the best most exciting hacks are utterly useless (e.g. The Homework Machine, Virtual Reality Nerf Gun, and a traffic cam tracker). While it’s cool if a hack can be used after a hackathon, it should not be a judging criteria. Furthermore, almost no hacks ever continue after a hackathon, making this criteria pretty pointless.
• Business Potential – The purpose of a hackathon is not to build a business, that should be left for startup competitions like Startup Weekend or StartupBus. Judging based on business potential will lead to a bunch of pitch decks and lies about roadmaps rather than demos of hacks.

Selecting Your Judges

Another vital part of judging is selecting the right judges. This can be difficult. However, this again can be made easier by taking the purpose of a hackathon into account. As a rule, hackathon judges should all be technical (with an exception granted to subject matter experts in a themed hackathon). The judges should be able to ask questions like, “Why did you choose LISP?!” or “What problems did you run into while implementing the Google Maps API?” Investors and startup reporters (read: non-technical folks) tend to ask the wrong questions, and select pitch decks over hacks, this serves to counteract the purpose of a hackathon.

Commonly, judges will come from sponsoring companies, but if you need additional judges you can use judging as a way to build relationships. Invite people you want to know better, movers and shakers in the community, or people from companies you want to sponsor the event.

The Judging Process

All this brings us back to the event itself, and the process of judging. Preparation for judging truly starts at the beginning of the event, and continues until the very end.

Before Demos

Before demos start, make sure to brief the judges on what to look for, explain your criteria and your expectations. Make it clear to judges that they are to be looking for hacks, not businesses, and that they should ask questions pertinent to the hack and not pertinent to the future of the hack. If you plan to have rating scales, provide the judges with these and further encourage them to take notes on each hack during demos.

You must also make expectations clear to participants from the beginning of the event. Make sure participants know that they will be judged based on their code and that they should show a demo, not PowerPoint slides.

During Demos

Keep demos short, no more than five minutes per hack including questions. As stated earlier, encourage judges to take notes on the demos, as these will come in handy later and prevent the tendency of remembering only the first and last demo.

After Demos

After demos are done, and the judges are in session, you may start the “real judging.” If you chose to have a ranking scale for each criteria, tally these up, however do not allow it to be the final decider of the winner. Often discussion brings much more to the table. Ask the judges if any hacks stuck out as the clear winner, there will often be a few which do. From there, talk out which hacks really deserve it, often the folks who spent the weekend with the hackers will have some good insight. Further, determine the sponsor prizes and try to make sure that no team gets all the prizes.

Once decisions are made, go announce it, and make the winners feel special. Make sure they come to the front of the stage and present them with each prize as if they were accepting an Olympic medal. They earned it.

---

This post deals primarily with smaller hackathon judging where “science fair judging” isn’t required. However, many of the lessons from this post can be taken and applied to other methods of judging including science fair style and other distributed methods.

If you ever want to discuss judging or anything else hackathon related please, don’t hesitate to get in contact with SendGrid’s Developer Relations team.