← Back to post

Edit history

Most recent

ARC Prize maintains multiple tracks around their benchmarks. They have “verified” leaderboards, “community” leaderboards, and they also run the ARC Prize competition.

They update the “verified” leaderboards when they test raw LLMs without sophisticated harnesses. They seem to update this sporadically and only occasionally do press releases or blog posts about new scores. For example, the latest score from Claude Opus 5 (High) is 30% at $20,000, and they didn’t post about that as far as I know. Again, this just the raw LLM without an agentic or world-model harness.

The ARC Prize competition has a harder set of criteria. Participants have to use smaller, open models with a limited compute budget, with open source code, and of course the solutions are verified by ARC Prize at the end of the competition.

The “community” leaderboards, which is what this post is about, are self-reported and not verified by ARC Prize. There are no restrictions on what model is used or limitations on compute. So naturally they aren’t going to make official news releases about those, unless they decide to verify them at some point.

The only reason I chose to post this is that the top solutions seem legitimate, with source code released, and two of them have associated papers.

Edited

ARC Prize maintains multiple tracks around their benchmarks. They have “verified” leaderboards, “community” leaderboards, and they also run the ARC Prize competition.

They update the “verified” leaderboards when they test raw LLMs without sophisticated harnesses. They seem to update this sporadically and only occasionally do press releases or blog posts about new scores. For example, the latest score from Claude Opus 5 (High) is 30% at $20,000. Again, this just the raw LLM without

The ARC Prize competition has a harder set of criteria. Participants have to use smaller, open models with a limited compute budget, with open source code, and of course the solutions are verified by ARC Prize at the end of the competition.

The “community” leaderboards, which is what this post is about, are self-reported and not verified by ARC Prize. There are no restrictions on what model is used or limitations on compute. So naturally they aren’t going to make official news releases about those, unless they decide to verify them at some point.

The only reason I chose to post this is that the top solutions seem legitimate, with source code released, and two of them have associated papers.

Original

ARC Prize maintains multiple tracks around their benchmark. They “verified” leaderboards, “community” leaderboards, and they also run the ARC Prize competition.

They update the “verified” leaderboards when they test raw LLMs without sophisticated harnesses. They seem to update this sporadically and only occasionally do press releases or blog posts about new scores. For example, the latest score from Claude Opus 5 (High) is 30% at $20,000. Again, this just the raw LLM without

The ARC Prize competition has a harder set of criteria. Participants have to use smaller, open models with a limited compute budget, with open source code, and of course the solutions are verified by ARC Prize at the end of the competition.

The “community” leaderboards, which is what this post is about, are self-reported and not verified by ARC Prize. There are no restrictions on what model is used or limitations on compute. So naturally they aren’t going to make official news releases about those, unless they decide to verify them at some point.

The only reason I chose to post this is that the top solutions seem legitimate, with source code released, and two of them have associated papers.