The problem
I wanted to compare models on the same prompts without seeing their names first, then look closely at where their answers failed. BlindBench grew out of that interest in making model choices easier to examine.
What I built
I built BlindBench as an open-source evaluation arena. I use real prompts from 4 Kaggle datasets, run those prompts through 100+ models, and score responses on correctness, reasoning depth, and failure patterns. I keep model names hidden during evaluation so judgment is less biased.
Key design choices:
- Blind testing: Model identities hidden during evaluation to reduce brand bias.
- Truth scoring: Composite score based on correctness, reasoning quality, and consistency.
- Failure classification: 10 distinct failure types (hallucination, logic errors, refusal bias, etc.) tracked per model.
- Bring-a-key mode: Keys are sent with the request to a Supabase Edge Function, which calls the model provider. Free-tier models work without a user-provided key.
Architecture
React + Vite frontend deployed to GitHub Pages. Supabase backend with Edge Functions that proxy LLM calls server-side. Materialized views power the leaderboard and failure analytics. Four seeded Kaggle datasets provide the evaluation corpus.
Distribution
- Live arena: test models immediately
- GitHub repo: full source, seed scripts, deployment guide