2026 · LISN · LLM Evaluation

France Travail Job Description Challenge

France Travail, France's public employment service, publishes millions of job offers. Some are unattractive, and some contain illegal clauses. This Codabench challenge asks for LLM judges that score an offer, find its specific flaws, point to where they are in the text, and explain them.

140control / degraded offer pairs

10rubrics: 4 legality, 6 attractiveness

4competition metrics

Surgical-edits protocol

We start from real, high-performing offers (more applications than the average for their job category). For each, we create a degraded copy that differs by one targeted edit, leaving everything else identical. A good judge must score the degraded version lower.

Critical omission: remove a key piece of information, for example the salary, to test the Compensation & Benefits rubric.
Toxic injection: insert a flaw, for example a discriminatory requirement. We store the exact span of the inserted text so we can check whether the judge locates it.

The legality rubrics cover discriminatory requirements, illegal financial demands, privacy and freedoms, and labour-law violations. The attractiveness rubrics cover pay, career growth, culture, work–life balance, nature of the work, and clarity.

Metrics

S₁ Sensitivity: does the degraded offer score lower than the original?
S₂ Feedback quality: does a reference LLM judge accept the explanation?
S₃ Localisation: does the judge name the omission, or locate the injected span with IoU > 0.7?
Final score: S₁ × (1 + S₂ × S₃) / 3. Sensitivity is a gate: if S₁ is 0, everything else counts for nothing.

Baseline results

Model	S₁	S₂	S₃	Final
Gemini 2.5 Flash	0.65	0.55	0.55	0.38
Gemini 2.5 Flash-Lite	0.45	0.35	0.40	0.27
Gemini 2.0 Flash	0.30	0.20	0.20	0.17

Debug run on 20 pairs. On the full 140 pairs, open-weight Mistral-Small 3.2 (24B) reached 32.9% sensitivity. It is strong on illegal financial demands (61%) and blind to subjective rubrics such as work–life balance and culture (0%).

Sensitivity is the bottleneck. A judge that doesn't notice a degradation can't explain it, so beating 65% will likely need larger models or models fine-tuned for evaluation.

My contributions

Built the LLM-as-a-judge baseline: one rubric at a time, temperature 0.1, and a fallback parser (JSON, then regex, then a numeric heuristic).
Implemented the multi-metric scoring program for Codabench.
Benchmarked inference for 24–31B open models on T4 GPUs (llama.cpp with flash attention, vLLM, multi-GPU, quantized Gemma).