Methodology
Your regulatory dependencies, with a changelog. emendrix watches EU legislation, computes provision-level diffs when it is amended, and explains what changed in plain English where every sentence cites a provision you can click.
Finding which provisions an amendment touched scores micro-F1 0.883 over 18 transitions against labels the legislation publishes about itself — that is agreement with a reference set at article-or-annex granularity, and it is not a measure of whether any explanation is good. Every figure below is generated from a committed report; none is typed in by hand.
Measured, not asserted
Every row carries what it means and what it does not. The rows are the same ones the repository's README publishes, rendered from the same committed report, so the two cannot disagree.
| Measure | Result | n | What it means — and what it does not |
|---|---|---|---|
| Localisation (which provisions changed) | P 0.806 / R 0.975 / F1 0.883 | 18 transitions | Structural diff against the corpus's own modification metadata, at article-or-annex granularity. Not a measure of whether the explanation is any good; precision is dragged down by blanket amendments the reference set annotates only once. |
| Instruction-parse agreement (cross-check) | P 0.875 / R 0.940 / F1 0.906 | 13 transitions | A third, independent reading of the same question, from the amending act's own prose. Weaker by construction: it cannot read a range or an instruction that delegates to an annex, and those are counted as unread, never approximated. |
| Disputed changes (signals disagree) | 0.308 | 104 changes | Changes at least one signal named and another did not. They ship marked disputed; a high rate is a data-quality finding, not a hidden failure. |
| Change classification | accuracy 1.000 | 79 units | Insert / modify / delete against the metadata's role codes, on units both signals named. The role semantics are empirical — the authority tables publish no labels. |
| Citation grounding (gate pass, no retry) | 1.000 | 55 changes | Of the changes with an explanation to check, the share whose first answer cited only provisions it had been offered. Citation validity, not explanation quality — a wrong sentence with a good citation passes. |
| Quote-fallback rate | 0.000 | 55 changes | The share where the gate replaced the model's prose with a verbatim quotation of the provision after one failed retry. Correct by construction and visibly marked; this is where grounding losses go, and no change is ever dropped. |
| Explanation faithfulness (sampled) | 0.850 | n = 20, LLM judge + spot review (reviewed 20/20 on 2026-08-11 by Martins Erts: 18 faithful, 2 not faithful, judge disagreed on 1 (strict). Same question as 2026-08-09; the sample moved by one triple, re-recorded after the instruction-scope fix, and both failures were also flagged by the judge.) | Whether the shipped sentences follow from the evidence the writer was given: the prompt's deterministic-facts header and the two capped texts (the header joined the evidence base on 2026-08-09, so earlier rates answer a narrower question and are not comparable). The weakest number here: a sampled judgement by openrouter:openai/gpt-5.6-sol, which is a different and stronger model than the explainer but not an independent one. The raw fraction, no confidence interval — at this n an interval would be wider than the number is useful. |
Measured on 2026-08-11 at revision 9a8d53b, from reports/eval/2026-08-11-9a8d53b.md. The deterministic rows cover every transition in the committed corpus (18 of 18 scored); the model rows cover the pinned explanation subset only, because each change in it is one recorded call to a provider.
How it works
- 1 · WatchA watchlist of acts, matched against the Publications Office notification feed by identifier. Deterministic.
- 2 · FetchBoth consolidated versions and the amending act, as structured Formex 4 XML, through a disk cache. Deterministic.
- 3 · DeltaA structural diff over the provision trees: which articles and annexes were inserted, modified, deleted. No model.
- 4 · ExplainThe only model call in the loop, constrained to the two verbatim texts it is handed. It never decides what changed.
- 5 · GateEvery citation must resolve to a provision that was offered, or the sentence is replaced by a verbatim quotation. Deterministic.
Between Delta and Explain the changes are corroborated against two independent signals — the corpus's own modification metadata and a parse of the amending act's instruction prose — and anything they disagree about ships marked disputed rather than dropped. After Gate the result is committed to a git repository as Markdown and JSON. Every stage except Explain is deterministic Python.
How this site is built
A directory of static files, generated by emendrix site build from artifacts committed in the emendrix repository and in a changelog repository the operator owns: the changelog documents this tool wrote and the dated evaluation report it was scored against. Where that repository lives is deliberately not printed here.
Every page is a rendering of things already committed, so the site can answer no question its inputs do not already contain, which is what makes no number here lacks provenance structural rather than aspirational. Two builds of one repository state produce the same bytes: nothing on the site reads a clock, reaches the network, or calls a model. One small script adds search; the only links that leave the site go to EUR-Lex.