2026-08-17 · 4 min read
Evaluating PII detection across 26 languages
A span either has the right type and both boundaries or it does not count. Scored that way, in every target language at once, including the two whose identifiers cannot be checksummed.
What is being scored
A PII tagger produces spans, and a span is only useful if it is the right kind of thing in exactly the right place. So the evaluation is entity-level: a prediction counts when its type and both boundaries match a gold span, and anything less is an error. The alternative, token accuracy, flatters a tagger, because most tokens in any text are not part of an entity and a model that finds nothing still scores most of them correctly.
Scored that way, the published model reaches an overall F1 of 0.9979 (precision 0.9966, recall 0.9991) over 10,296 gold spans, across all 26 target languages at once and 7 entity types.
Per entity type
| Type | P | R | F1 | Gold spans |
|---|---|---|---|---|
CARD | 0.997 | 0.997 | 0.997 | 1,352 |
DATE | 0.983 | 1.000 | 0.991 | 1,352 |
EMAIL | 1.000 | 0.999 | 1.000 | 1,352 |
IBAN | 1.000 | 1.000 | 1.000 | 1,456 |
NATIONAL_ID | 0.997 | 0.997 | 0.997 | 1,560 |
PERSON | 0.999 | 1.000 | 1.000 | 1,976 |
PHONE | 1.000 | 1.000 | 1.000 | 1,248 |
Dates are the newest type in the set, added because a date of birth is personal data whether or not anything else in the sentence is. The identifiers with a public checksum scheme are generated checksum-valid in training, so an IBAN that fails mod-97 is not an IBAN to this model. Maltese and Azerbaijani national identifiers have no public checksum scheme, and for those two the training data can only be format-valid.
The tail is French
Every language is scored separately on 396 gold spans, and 19 of the 26 score a perfect span F1. The weakest are French at 0.977, Latvian at 0.990, Maltese at 0.995.
The interesting row is the one that is not in that list. Maltese is absent from the base model's pretraining corpus, and on the classifier side of this project that gap has repeatedly produced the weakest scores. Here it does not: Maltese holds 0.995 while French, one of the best-resourced languages in the base model, sits at the bottom. Span tagging over names, numbers and formats leans less on pretraining than label classification does, and the corpus, not the base model, is what bounds these scores.
The export that ships
The artifact in the repo is a float16 export, and it is gated on spans rather than on a score: checked against the fp32 weights on 300 texts, it changed the span set on 0 of them, invented 0 spans, and left 0 characters uncovered that the full-precision model covered. For a redactor, that last number is the one that matters: a compression that loses a character of coverage is a leak, not an optimisation.
What this does not show
The corpora are synthetic, generated per language rather than translated, which is what makes 26 languages affordable and also means these are in-distribution results. Real text is messier than any generator, and an evaluation against it is outstanding work. Until that exists, read this table as what it is: the model, scored exactly and strictly, on the distribution it was trained for, published in its own card so the next evaluation has something to be compared against.
References
- [1]The published evaluationInside the model repo itself. Every figure in this post is read from it at build time.
- [2]The export manifest and its gateThe fp16 export is verified against the fp32 weights on character spans before it ships.
- [3]flowxai/piiguard
- [4]The use case this model exists for
- [5]XLM-RoBERTaThe base model. Maltese is absent from its pretraining corpus, which makes its span score here worth noticing.