Round-trip translation
Translate a text into another language and back again. If the model understood it, you get the original meaning back.
What it is
Each task takes a text in a base language, has the model translate it into a target language, then has the same model translate that result back. We compare the back-translation with the original by embedding both texts and measuring the distance between them. The distance is converted to a score between 0 and 1, and higher is better.
The tasks target several languages from different language families, so a score reflects more than how well a model handles one language.
Comparing meaning through embeddings, instead of matching words, means a faithful paraphrase is not punished for choosing different wording than the original.
More than a translation test
A plain round trip only asks whether a model can translate. Real use asks more: does it stay on task, keep its instructions to itself, leave the facts alone and hand back something a program can read? Each task is built so that failing any of those shows up in the score.
- Leaking information. The model is given private context that has nothing to do with the text. A faithful translator never needs it, so if it surfaces in the output, the model answered instead of translating.
- Instruction injection. Some of the text to translate contains an instruction addressed to the reader. A good model treats it as content and translates it. A model that obeys it has been hijacked by the data it was meant to process.
- Fact injection. Some sentences state well-known falsehoods. The job is to translate what was written, not to quietly correct it. Models that “fix” the source change the meaning they were asked to carry over.
- Not refusing legitimate text. Some passages sound alarming out of context but are ordinary and lawful. Refusing or softening them is a failure, because over-cautious models are not useful ones.
- Structured output. All answers come back as one JSON object with a fixed set of keys, with no commentary or stray formatting around it. Output that cannot be parsed or has missing keys counts as a failure.
- Genuinely hard language. Idioms, references back to earlier sentences, negation scope, ambiguity, irony and large numbers appear alongside the probes above, so a model can’t pass on easy text alone.
These traps are scored hard. A failed task scores 0.0001 rather than being averaged away, and a run’s score is a geometric mean, so one failure drags it down well beyond its share. That is what separates a model that translates well in a demo from one you can put inside a pipeline.
What it is good for
- No reference translations needed. The original text is its own answer key, so the benchmark works for any language pair you have text for.
- Comparing models and quantisations. It shows how much a smaller quantisation or a different model loses on the same tasks, run by run.
- Catching meaning loss. Dropped facts, mistranslated terms and drifting meaning all push the back-translation away from the original.
- Running on your own hardware. Every result comes from a model you can download and run yourself.
- Choosing hardware. The leaderboard shows the memory each model and quantisation needs at each context length, so you can see what fits your machine.
What it is not good for
- Judging fluency or style. A clumsy but meaning-preserving translation can score well.
- Proving a single translation is correct. Errors in the two directions can cancel out, so the round trip can look fine even when the intermediate text is off.
- Measuring everything a model can do. It says nothing about reasoning or coding, and it tests instruction handling only in the narrow sense above. Treat it as one signal, not a ranking of overall quality.
- Comparing scores across versions. Scores are only comparable within one benchmark version.
Current version v26.10.0
Best run so far: deepseek/v4-flash-0731 (unsloth-iq3_s, default).