Leaderboard
Loaded from results.json. The shipped file is a
mock provider run — not a live model ranking.
| Category | Passed | Total | Accuracy |
|---|---|---|---|
| Loading results.json… | |||
| ID | Category | Scoring | Pass |
|---|
Quickstart
Mock run needs no API key. Copy, paste, get a labelled results file.
pip install -e .
darija-eval validate
darija-eval run
How it works
JSONL in, Pydantic in CI, provider behind env vars, exact / normalized / fuzzy
scoring, resumable results.json.
- Validate every line against the schema.
- Call mock or OpenAI-compatible chat completions (with retries + disk cache).
- Score; optionally ask an LLM judge only on misses.
- Write JSON + Markdown/HTML tables; this page reads the JSON.
Submit your model’s results
Run against a real provider, keep the raw results.json, and open a PR
that replaces site/results.json only if you document provider, model,
date, and that it is not mock. Prefer adding a second file under
site/runs/ so the mock baseline stays.
Honest limits
- All 150 items are still
draft— no native-speaker sign-off yet. - Mock accuracy is a hash bucket, not linguistic quality.
- Safety items score a fixed refusal phrase, not free-form refusal quality.
- Regional Darija and Arabizi-heavy chat are under-covered.
How I built it
- Default provider is mock so CI and Pages never need secrets.
- Fuzzy scoring uses difflib after Arabic-aware normalization, not a BLEU dependency.
- Resume writes after every item so a killed process can continue.
- Dataset card lives in
DATASHEET.mdnext to the JSONL.
Status
Roadmap issues: github.com/mohamedtaqysalmi/darija-eval-set/issues