June 30th 2023

Measuring safer systems

Machine Learning Benchmarks Hackathon

Join us for a weekend of intense and focused research work where we think about how to measure the safety of AI systems.
Join the newsletter and the community on Discord to stay updated and notified about future events.
Marina T
Johanna Einsiedler
and others!
You have successfully been signed up! You should receive an email with further information.
Oops! Something went wrong while submitting the form.


Read up on the topic before we start!
Join the reading group

Eval Harness: LLM benchmark tool

MMLU: Measuring language ability

RL Benchmarks list for many tasks

Language Model Evaluation Harness

The LMEH is a set of over 200 tasks that you can automatically run your models through. You can easily use it by writing pip install lm-eval at the top of your script.

See a Colab notebook shortly introducing how to use it here.

Check out the Github repository and the guide to adding a new benchmark so you can test your own tasks using their easy interface.

Join & organize a jam site

The in-person hubs for the Alignment Jams are run by passionate individuals just like you! We organize the schedule, speakers, and starter templates, and you can focus on engaging your local research and engineering community. Read more about organizing and use the media and social media content below to set up your event.

Registered sites

No events registered at the moment
The event starts at 19:00 CEST and the submission deadline is at 4 AM Monday CEST. We recommend that you end your event Sunday evening.
Upload failed. Max size for files is 10 MB.
Thank you! Your submission has been received! Your event will show up on this page.
Oops! Something went wrong while submitting the form.
Social media message

We're hosting a hackathon to find the best benchmarks for safety in large language models! 

Large models are becoming increasingly important and we want to make sure that we understand the safety of these systems.

With Alignment Jams, we get a chance to create impactful and real research on this problem together with people from across the world.

Don't miss this opportunity to explore machine learning deeper, network, and challenge yourself!

Register now: https://alignmentjam.com/jam/benchmarks

[or add your event link here]

Liked by
Event cover image
Social media 1

Past experiences

See what our great hackathon participants have said
Jason Hoelscher-Obermaier
Interpretability hackathon
The hackathon was a really great way to try out research on AI interpretability and getting in touch with other people working on this. The input, resources and feedback provided by the team organizers and in particular by Neel Nanda were super helpful and very motivating!
Luca De Leo
AI Trends hackathon
I found the hackaton very cool, I think it lowered my hesitance in participating in stuff like this in the future significantly. A whole bunch of lessons learned and Jaime and Pablo were very kind and helpful through the whole process.

Alejandro González
Interpretabiity hackathon
I was not that interested in AI safety and didn't know that much about machine learning before, but I heard from this hackathon thanks to a friend, and I don't regret participating! I've learned a ton, and it was a refreshing weekend for me.
Alex Foote
Interpretability hackathon
A great experience! A fun and welcoming event with some really useful resources for starting to do interpretability research. And a lot of interesting projects to explore at the end!
Sam Glendenning
Interpretability hackathon
Was great to hear directly from accomplished AI safety researchers and try investigating some of the questions they thought were high impact.