Quality Estimation: Everything You Wanted to Know

Introduction

A couple of weeks ago, I had the pleasure of moderating the innovation track at the GenAI in Localization conference, hosted by my friends at Custom.MT. I walked away with several interesting insights, but one trend in particular inspired me to write this piece: the increasing number of quality estimation (QE) and automated post-editing (APE) solutions being developed by dedicated langtech companies, AI startups, and LSPs of all sizes.

In today's AI landscape, the rise of LLM-powered post-editing makes perfect sense since content operations are what large language models are built to do. Historically, quality estimation has proven to be a much tougher and perhaps less glamorous nut to crack. This article is my attempt to break down everything you wanted to know (or not) about quality estimation — for the linguist trying to make sense out of a score in a CAT tool, the engineer wiring a QE endpoint into a pipeline, and the C-suite executive trying to work out whether any of this actually moves the P&L.

Neural Machine Translation

The story begins with the rise of neural machine translation in 2016-17. The transformer, the architecture behind modern language models, was introduced by Google specifically for translation before DeepL turned it into a consumer phenomenon, and the language services industry witnessed the breakthrough of a technology that would deeply reshape the work of localization professionals across the supply chain. For the first time in history, we asked ourselves whether machine translation would ever replace human linguists.

At the same time, it marked the beginning of a new era in the language technology space. Open-source toolkits such as Marian and OpenNMT became staples of academic research, and dedicated MT companies like Globalese, Kantan, and Pangeanic accelerated the transition from statistical (Moses, anyone?) to neural. SYSTRAN, the industry's oldest MT name, was still reporting €20 million in revenue in 2022, before ChapsVision absorbed it in 2024. Enterprise localization programs as well as LSPs (especially the big ones) established shiny MT departments that dominated industry events with case studies and keynotes.

Due to big tech's dominance (Amazon, Google, Microsoft, IBM) in the MT field, machine translation became a commodity before we even realized it. The meteoric rise of DeepL, which quickly became the most requested integration in TMS, accelerated the adoption of neural machine translation even further.

memoQ blog | July 2018 | https://blog.memoq.com/wait-no-more-deepl-is-now-available-in-memoq

The service has yet to see its first birthday, and still, our product team has been bombarded with requests for DeepL support in memoQ. The sense of urgency became even stronger after their paid service and API became available publicly in March earlier this year [2018]. It might be one of the most requested memoQ features ever.

Nevertheless, the first cracks in the MT bubble also started to appear; consistency, out-of-context translations, and terminology adherence soon turned out to be genuine challenges.

The Dawn of Quality Estimation

The language industry experimented with different approaches to NMT customization, but none truly moved the needle. Production-grade training on parallel corpora never quite materialized — partly because it's a slow, expensive endeavor, but mostly because we overestimated the suitability of translation memories as training data. Other techniques such as MT glossaries helped at the margins but turned out to be drops in a very large ocean. In the end, most MT labs spent their energy comparing baseline models and having sample sets scored by human linguists — a practice that very often crowned stock DeepL the winner.

Part of the problem ran deeper as TMS became the main NMT hub. The single-sentence paradigm was imposed, sprinkling neural machine translation over XLIFF units and exposing it as an additional translation result alongside TM matches, thereby amplifying its primary weak spots.

While the MT investments didn't result in further sophistication of the MTPE model, the prospect of an intelligence layer that could help determine when, where, and how human intervention was needed would outlast every customization bet. Rather than altering a translation model's parameters directly, MT specialists turned their attention to the operational infrastructure around the model — automation, integration, orchestration, routing.

> Enter quality estimation.

Early quality estimation was driven by models that were trained on millions of translation pairs to assess and predict the accuracy of machine-translated content. The earliest neural QE models didn't simply resemble NMT systems, they were built from them: an encoder-decoder trained on parallel data, with an estimator layer bolted on top to predict and quantify quality instead of generating a translation.

Later initiatives would move away from this approach and lean on pretrained multilingual encoders instead, but the underlying idea didn't change. Since evaluating MT quality is never entirely unbiased — not even in the case of human evaluation — quality estimation was introduced to predict how much post-editing a specific machine translation requires, thus identifying potential risks associated with the translation.

Among some of its use cases:

• Focusing post-editing efforts on poorly translated segments.

• Processing translations that require minor editing in lightweight workflows.

• Pushing low-risk translations into production, no humans in the loop.

• Making informed decisions about content that needs to be prioritized for localization.

According to ModelFront's framing, the high-value use case is approving or rejecting translations without compromising quality, the ultimate goal being to establish a level of efficiency that enables companies to translate more content into more languages — content that would otherwise never receive human translation at all. Of course, "without compromising quality" only means something once "quality" is defined.

The Score is Not the Quality

So what does "quality" in "quality estimation" actually mean? For a long time, we've associated translation quality with getting every dot, comma, and grammatical nuance right. Contrarily, in today's technology climate, quality is increasingly connected to how multilingual content performs, i.e., its impact and intent proximity, contributing to higher-level business goals, KPIs, and ROI.

Here's my favorite illustrative example: if 30% of your website consists of high-traffic content that drives customer acquisition and sales, that portion deserves to be a work of art, delivered by a skilled human linguist. But if a raw machine translation of the remaining 70% of low-traffic content helps attract more visitors or keeps them engaged a bit longer, you'll have achieved the exact same level of quality — from a business perspective, not linguistically.

And here's my favorite real-world example: Uber allows me to book a ride in a matter of minutes and take care of all the post-ride admin with a consistently positive user experience, despite occasional translation glitches in Dutch, my mother tongue. I simply don't care about them (and dear readers, neither do you). Uber gives me everything I need to get from A to B.

Defining quality is one thing, measuring it is a different challenge. The language industry has no shortage of quality algorithms and metrics, but the problem is that context-poor, static evaluation methods are often mistaken for quality representations of both human and machine translation. The notorious BLEU score (or more advanced metric families such as LEPOR) is certainly useful, but is intended to measure to which extent a model can replicate a human reference translation. The even more notorious edit distance statistic calculates the amount of deletions, insertions, and substitutions in a machine-generated translation, but doesn't look at the cognitive effort necessary to finalize a translation, yet several companies used it to compensate linguists or negotiate lower rates.

Then what sets quality estimation apart from other algorithms and metrics? Perhaps less than we think or hope. Personally, I love the idea of generating instant quality signals in order to estimate when, where and how human intervention is needed, and subsequently route localized content in sophisticated, conditional workflows (if > then > else). However, my experience with low-entry, off-the-shelf QE solutions returning blackbox scores to assess "quality" is pretty negative and comes with a set of side effects that, according to me, shouldn't be considered solved problems — not even when throwing LLMs in the mix:

1. Most QE services operate at the translation-unit level, but fail to produce a document-level (or "entry-level") quality assessment that still catches and reports errors in isolated sentences.

2. Other than setting thresholds, QE scores are difficult to operationalize. A score of 0.85 doesn't necessarily imply that 15% of the translation is compromised like a fuzzy match does. It's a guess dressed as a policy.

3. A small mistake might still produce a high mechanical score. Imagine the source sentence "Take 2 pills every 6 hours" being translated as "Take 6 pills every 2 hours"? A raw edit distance calculation will make it look almost perfect.

Levenshtein edit distance

At GenAI in Localization's innovation track, Adam Bittlingmayer rightfully debunked the myth that raw QE scores are usable as-is.

Dumping a number into a workflow forces the localization team or CAT user to guess what it means and decide what to do next.

A production-grade implementation should translate probabilistic scores into deterministic decisions under transparent monitoring. In other words, the problem isn't that a score exists, it's that the score gets shipped to the consumer instead of the decision. Uncalibrated scores never help delete anything from the invoice, since they can't be relied upon.

The Technology Market

The theory is clear, the market is less so. The growing number of QE and APE solutions reflects a fragmented ecosystem with many different approaches. Sizing the QE market cleanly is close to impossible as most of it is embedded in TMS subscriptions, package deals, homegrown solutions, and service contracts rather than sold as a standalone line item.

Using our own $0.4 billion estimate of the TMS market as the primary proxy, and assuming QE features don't even account for a modest 5% of that spend, dedicated revenue from quality estimation is probably still below $20 million once factoring in what pure-play vendors and LSPs deliver outside the TMS channel. ModelFront and TAUS, the two names most associated with quality estimation, remain relatively small operations.

Next to the roughly $73 billion language services market (cf. Nimdzi Insights' estimate), $20 million is negligible. The real point: QE revenue is small because it's sold as a feature, not because the problem is small. The value shows up in the buyer's P&L, not in the vendor's revenue report.

Quality estimation market snapshot

The Pioneers

• Unbabel: Released OpenKiwi (2019) and then COMET (2020), a neural framework that became one of the de facto standard metrics for automated MT evaluation in the research community. Pioneering tech and establishing industry standards turned out to be insufficient to resist the cracks in the MT bubble — in 2025, TransPerfect acquired Unbabel's AI assets, including COMET and the TowerLLM model set, mashing them straight into GlobalLink.

• TAUS: Spent years collecting and selling translation data, likely sitting on the largest data vault in the industry — a resource that directly underpins its own quality estimation framework, which is now also widely distributed via TMS.

• ModelFront: Arguably one of the few companies that successfully productized quality estimation, drawing on founder Adam Bittlingmayer's experience at Google Translate. ModelFront now supports Booking.com's localization program, among others, automating hundreds of millions of translated words.

• What many people may not know is that Translated's ModernMT also exposed a QE endpoint through its API. The confidence scoring capability carries into Lara, ModernMT's successor.

Finally, Pangeanic is characterized by its ability to continuously evolve in response to fast-changing technology and market demands. Its QE offering, grounded in its deep adaptive AI philosophy, scores translated segments without a reference translation and routes them for publication, light review, deep post-editing, or rejection.

Translation Management Systems

• Phrase: Phrase's internal ML team (back in the Memsource days) built one of the industry's earliest segment-level QE features, expressing MT confidence on the same percentage scale as TM matches so it could slot into existing analysis and pricing workflows.

• wxrks: Released one of the first modern, LLM-powered semantic analysis features — a clean break from the rigid, score-driven quality estimation that came before it. Instead of calculating numbers, it surfaces meaning-level red flags directly to the linguist, treating quality as something to reason about. Recent efforts include agentic autonomy, configurable confidence scoring, and routing control.

• Smartling: Was already experimenting with forward-looking quality indications long before the LLM era — a 2017 proprietary ML model predicted the likelihood a target-market human would rate a translation as high quality. With their newly released LQA agent, Smartling now plays the agentic card.

• XTM: After years of research into proprietary model training and numerous pre-LLM AI pilots, XTM pivoted to LLMs with its own QE > APE pair, offering a combination of context-aware translation, scoring, risk-based routing, and automatic edits. Expected to be deployed across XTM's acquired product portfolio.

• Crowdin: Integrates quality estimation into the multilingual content ecosystem through third-party add-ons available in its marketplace of 700+ apps, including ModelFront and TAUS, alongside its own deeply context-aware AI infrastructure.

• memoQ: Chose to build direct integrations with ModelFront and TAUS rather than building its own model, exposing quality scores directly in the editor and supporting threshold-based auto-validation. memoQ acquired Globalese in 2024 and is now quietly repositioning it in the shadow of its own adaptive translation solution.

Beyond the TMS layer, aggregators close the circle. Custom.MT layers quality estimation (and APE) from different providers on top of whatever translation model a customer already uses, Intento builds tailored agentic pipelines, and broader workflow orchestrators like Blackbird go further, gluing the many content systems and file formats in the typical localization tech stack together in end-to-end workflows. Most quality estimation APIs are segment-in, score-out and can't natively parse the formats localization actually runs on. One thing the aggregators typically add is the ability to take away the corresponding complexity and write results directly in dashboards, XLIFF, and TMS.

What about DeepL?

A full market overview and no sign of DeepL? DeepL now offers its own translation quality evaluation capabilities as a beta feature in its enterprise tier. The system analyzes documents, produces quality reports with MQM-inspired scores as well as error classifications and corresponding routing options. Add to this that DeepL now also exposes translation units in a CAT-style editor and one thing becomes clear: DeepL is gradually evolving beyond the transactional MT engine it once was, and starts taking on the shape of a reverse-engineered TMS with an increasing focus on humans in the loop.

The Harness

Based on what I hear and see at industry events, the massive MT programs born in the NMT era are now being complemented or even replaced by quality programs that face growing complexity amid the AI revolution. Quality in localization goes much deeper than simply distinguishing the business-critical 30% from the second-tier 70% described earlier. A modern quality program must be able to scale dynamically with unbudgeted content and target market expansion on the one hand, and on demand on the other. These are fundamentally different challenges. A Black Friday localization sprint isn't a bigger version of a quiet summer release; it's a different quality regime with higher business stakes per word, and perhaps a temporary tolerance for rougher edges. None of this is something a translation model can understand on its own, which is why the center of AI gravity has shifted.

Where the industry once focused almost entirely on the models and the MT step itself, the heavy lifting now happens before and after. Today, practicing AI in localization means perfecting context supply and implementing guardrails — the so-called "harness." Every AI program I've built and seen indicates that the real value comes from the scaffolding rather than the engine; a harness that survives a model swap is an asset, a prompt tuned to a vendor's model is a liability. Something that should matter to any CTO deciding where to spend the next engineering quarter, and to any CFO wondering why two vendors running the same model produce different quality and cost outcomes.

Within the harness, quality estimation and automated post-editing are likely to have a permanent seat for a number of reasons. First, in my experience, language models are often better at evaluating than generating — evaluating a translation against well-designed contextual materials plays more to their strengths than producing one from scratch. Evaluation starts from a candidate that already exists; generation is unconstrained search. Second, LLMs will inevitably make mistakes in automated pipelines and deserve a second shot, same as human linguists do, guided by the same kind of supporting metadata available to human linguists — much of which still lives in CAT comments, emails, spreadsheets, tickets, and shared documents. In my opinion, this is the foundation of modern AI-driven localization pipelines.

AI-driven localization pipeline example

Conclusion

The catalyst for the real breakthrough of quality estimation, with or without automated post-editing, may be more economic than technical. The language services industry has ended up in an uncomfortable yet predictable situation where global businesses need more multilingual content than traditional localization models can economically deliver. As Adam puts it: translating with today's legacy technology and processes is so painfully slow and expensive that companies can only afford human-quality translation for "less than one percent of their content, into a handful of language markets." Making translation more efficient doesn't reduce demand; it reveals how much unprocessed demand there is — yes, Jevons paradox.

Therefore, next-gen quality estimation will not be defined by better scores, but by better decisions. A number between zero and one produced by a model nobody can interrogate is a vibe with decimals. A real quality estimate is anchored to a decision the business has already agreed to make and can be audited afterwards. This is also where automated post-editing kicks in. APE that shaves edit distance is worth very little as the expensive part isn't the typing, it's the reading. It succeeds only when it moves segments across the approval line reliably enough to reserve human attention for where it matters most.

Technically, GPT wrappers with clever prompts aren't the future of quality estimation. QE belongs in a comprehensive harness that connects the fragmented business context required to make decisions — from communication channels, spreadsheets, style guides and all kinds of internal documents to online content, research, legislation, institutional knowledge, and market signals. The defining capability is the ability to continuously learn from outcomes, automatically enrich context, and dynamically refine guardrails.

The persistent gap between what companies translate and what they would translate if translation were efficient enough to be a default rather than a business decision narrows when deciding whether a translation is good enough stops being an expense, and becomes trustworthy enough to happen thousands of times a day. Without anyone watching.

> Enter quality estimation.