darija-eval-set

I built this because most public Arabic evals ignore Moroccan Darija chat— mixed script, French loans, short answers. Here: 150 prompts, a Typer runner, and scores you can reproduce offline.

Leaderboard

Loaded from results.json. The shipped file is a mock provider run — not a live model ranking.

Accuracy by category
Category Passed Total Accuracy
Loading results.json…
Per-item results
ID Category Scoring Pass

Quickstart

Mock run needs no API key. Copy, paste, get a labelled results file.

pip install -e .
darija-eval validate
darija-eval run

How it works

JSONL in, Pydantic in CI, provider behind env vars, exact / normalized / fuzzy scoring, resumable results.json.

  1. Validate every line against the schema.
  2. Call mock or OpenAI-compatible chat completions (with retries + disk cache).
  3. Score; optionally ask an LLM judge only on misses.
  4. Write JSON + Markdown/HTML tables; this page reads the JSON.

Submit your model’s results

Run against a real provider, keep the raw results.json, and open a PR that replaces site/results.json only if you document provider, model, date, and that it is not mock. Prefer adding a second file under site/runs/ so the mock baseline stays.

Use the PR template →

Honest limits

How I built it

Status

CI status   CodeQL status

Roadmap issues: github.com/mohamedtaqysalmi/darija-eval-set/issues