Generative Pre-trained Transformer (GPT) systems are not designed to be translation engines.  So it is surprising that they succeed so well at doing simple translations.  Some articles have claimed that they can translate better than existing translation engines.  How true are those claims?

The best comprehensive evaluation was done by scientists at Microsoft Research, unsurprising because they are among the leaders in both machine translation and GPT models.  The brief summary is that while GPT models have competitive quality when translating usual sentences from a major (see high-resource below) language to English, they are less good at other types of translation.

The evaluation uses three GPT systems that are known for translation quality, and compares them with neural machine translation engines (NMT), either Microsoft Azure API or the best-performing commercial systems or research prototype.  Quality scoring uses either algorithmic scoring or human evaluation.

How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation 

In most languages and for language directions, MS Azure Translator and other NMT engines outperform GPT for most measures of quality.  However, GPT does have the ability to improve after being given a few examples of correct translations, and to outperform NMT in some language directions after 5 tries.  This is the case for translations to English from German, Chinese and Japanese, languages for which there are a lot of examples in the GPT training set.  These are called “high-resource” languages.  On the other hand, it does not do particularly well for low-resource languages like Czech or Icelandic, or for English to other languages.  GPT’s training set had less text in those languages.  In the chart below, orange is GPT and blue is NMT.  The lines are algorithmic evaluation and the bars are human evaluation.

Those experiments above are for sentence-level translations.  For multi-sentence translations, those with more context that can be found in other sentences, GPT improves relative to NMT.  Not enough to beat the best NMT systems, but sometimes enough to match or beat the normal Azure API.  That is not very surprising: Azure Translator was optimized for sentence-level translation, while GPT is trained for multi-sentence context, up to thousands of words.  Other Azure translation APIs like Document Translator and Custom Translator are better at longer context windows, but this is not what was tested here. 

Neural Machine Translation models have another big advantage over GPT: they can be re-trained for a particular domain rather than with general domains, and this significantly improves their quality score for that domain.  For example, by giving it many training examples of automotive documents, Azure Custom Translator (a re-trainable version of Azure Translator) can increase its translation quality for documents in the automotive domain by a large factor.

Looking at other measures of performance gives a better understanding of what is different in the performance of the GPT vs. NMT.  Measuring fluency, essentially how natural is the sentence, how similar it is to other sentences out in the world, tells you something about the quality of the prose.  GPT is more fluent in English, it sounds more natural.  That doesn’t mean that it is a more accurate translation, far from it.  In fact, GPT has a greater tendency to add words, concepts, and punctuation that do not correspond to the original, or to omit some.  So it’s good prose, but it’s not necessarily what the original said.  For example, it does better at figures of speech, by not translating them literally, but also does not necessarily replace it with a term that means exactly the same thing.  It does not wander far from the original with completely made-up things, but it’s often not quite correct.  However GPT also does hallucinate words or concepts that were not in the original.

A literal and faithful translation is sometimes required, but that often leads to what is called “translationese”.  This refers to a set of common issues with text generated by human translators.  Translationese can refer to excessive precision or wordiness, or excessive vagueness in translated text, or syntax that is uncommon in the target language.  What translators are compensating for is the fact that different languages are specific about different things.  For example, English has the term “uncle”, which does not differentiate between paternal and maternal uncle or uncles by blood or marriage, but many other languages are much more specific.  Translating from those other languages to English, Translationese would not say “uncle” but might say “maternal uncle by marriage”, a term that is unusual in English, but which avoids losing information that was contained in the original.  In terms of translation quality, humans who are not translators might rate the translation with “uncle” higher because it sounds more natural, but translators would rate the awkward translation higher because it is more accurate.

Sometimes We Want Translationese

The design of the two types of models is also different. The “T” in GPT stands for “Transformer”.  Transformer is an attention-based neural network model that when looking at a word within a sentence or even longer text, determines which other words are the most relevant ones to pay attention to.  NMT also uses Transformer models.  However, there are big differences.  One is that GPT uses Decoder models, while NMT uses Encoder-decoder models.  What does that mean?  Decoder models focus on the output, the next word to be spit out.  Encoder-decoder models try to extract features from the input before feeding it to the part of the model that predicts the output.  It focuses separately on the input and on the output.  It tries to be robust to small changes in the input.

GPT only outputs one word at a time.  It starts with the text of the prompt plus other information in the input, then outputs a single word.  Then it adds that word to the end of the prompt in the input and puts this new input through again to get the next word.  It is unidirectional, that is to say when it is generating text it only looks at the previous words that are already generated, it doesn’t consider what it will say next because it hasn’t said it yet. Like a lot of humans, GPT is more concerned with what it wants to say next than with what you’re saying.  NMT is bidirectional.  It considers the rest of the sentence and the next sentence in both the source and the translation while it is generating the text.  It generates entire sentences at once.

Because it is a generative model, GPT is biased towards what is usual.  If the original text in the other language is commonplace and expected, then GPT will find good ways to express that text in English in ways that are commonplace and expected, because that is what it is trained to do.  If the original says something that is unexpected or expresses it in unexpected ways, GPT’s translation is likely to replace it with something more usual using some of the same words.  GPT does well at translation essentially because most things that require translation are predictable and unoriginal.

NMTs have a whole bag of tricks to deal with translation tasks that GPT does not, including specialized knowledge about the structure of languages, and tricks to deal with numbers, capitalization, and non-standard spacing correctly and efficiently.  They are also trained to preserve information.  You know that trick that people sometimes use, translating a sentence to another language then back to English so they can laugh at the result?  NMTs include that round-trip in their training, to make sure that none of the meaning gets lost in the translation.  Other tricks include having the neural network teach another neural network how to translate, detecting errors in the training data, and other tricks that address common translation errors.  There are also tricks to reduce gender bias, a problem that still plagues GPT.

Current Azure Translator NMT uses models of about 50 million parameters, which can run 4 language pairs in a Docker container on a host having a 2-core CPU with 2 GB memory.  Even the next generation of NMTs, Z-code MoE, which have 100 languages (10,000 language pairs) in a single model, can fit on a single GPU even though they have billions or hundreds of billions of parameters.  These are sizes for querying, what is required for training is much bigger.  GPT-4 uses 100 trillion parameters.  Training requires hundreds of thousands of CPUs and tens of thousands of GPUs, but to query them, it looks like a single cluster of 8 GPUs and a dozen or two CPUs is what is required.  Microsoft is very good at shrinking by orders of magnitude the size of machines required to run AI models so direct comparison is difficult, but NMTs deliver translations at much lower computational cost.  Microsoft’s DeepSpeed library in particular increases speed and reduces latency by a large factor.

The computing power required also has a potential impact on security.  NMTs, even the bigger potential NMTs can be run on a single processor, while GPT requires many processors.  Using GPT you are probably sharing hardware with strangers, while for NMT it is possible to have dedicated resources.  Because of its recurrent architecture, where the output is fed back into the input, GPT probably has some static storage of your data, while NMT can be architected with a pipeline where neither the input nor output text is ever stored.  I don't know how it is implemented by anyone, but I notice that for Azure, NMT has a no-trace option by default while GPT limited access previews do not.  Because of ethical concerns, data is probably retained for abuse monitoring.  I'm sure the security is good, but the architecture reduces the options for security.

The blanket claim that GPT is better at translation in not generally true.  However GPT is surprisingly good, considering that translation is a task that it was neither designed nor trained for.  It is unexpected that it is sometimes equal to or better than the highly specialized NMTs.  There is a fair bit of work being done on hybrid systems that combine the accuracy and specialized training of NMT with the fluency of GPT and will deliver the best of both. The next generation of NMT (see How to get a higher level of machine translation quality) will also allow the model to transfer language knowledge obtained from one language to other related languages, and in that way vastly improve the quality for low-resource languages such as southern Slavic languages.  That innate knowledge of what is common to languages in the same family can then be used to improve the quality of both NMT and GPT.

Let us stand back and try to understand what the term “human parity” means and what it doesn’t. 

First, the humans in question are not professional translators.  Professional human translators are still better.  The tests that justified the use of that term came from an annual competition where both humans and machines translated sentences from news stories.  Journalistic style is relatively simple, and the machines had trained on other news stories, so that makes it easier.  The two sets of translations, human and machine, were then scored independently for quality.  Microsoft Azure Translator obtained as good a score as the average human.  But remember, the humans in this comparison were probably bilingual computer science students, as were the evaluators.  If the testing had been done with professional translators doing the translation and doing the rating, it would not have been a tie, as other studies have shown.  “Human parity” is a good benchmark, but it is far from perfection.  There are still errors.

1. Implement a post-editing step that looks particularly for certain kinds of common translation errors before publishing
2. Switch to Azure Custom Translator and train it on a corpus of your own documents and phrases so that it learns your vocabulary and your style
3. Participate in the by-invitation pilot of the Z-Code MoE translation engine
4. Change the style guide of source documents so that they are written in a way that is easier to translate

First you need to have at least 10,000 professionally translated sentences or terms, in a supported format and following the naming conventions.  This is the learning set.  You can upload parallel documents in both languages, and it has some tools to help you align the sentences, but it’s not as good as you’d think and you have to revise carefully.  If an extra line is inserted, every sentence after that will be mis-aligned.  You also have to remove the sentences that are not quite the same.  Translators often split or combine sentences when they translate, it may be more natural sounding that way, but it will confuse the translation engine training, best to remove those.  Similarly, some translations, although correct, are not a good example for a machine to train on. For example, if the text is talking about pangrams,  sentences that use all 26 letters, “the quick brown fox jumped over the lazy dog” can be translated to “Portez ce vieux whisky au juge blond qui fume” (Carry this old whiskey to the blonde judge who smokes).  While it’s correct in this context, having a model learn from this translation would probably interfere with the model’s ability to translate other text about canids.  Training sets have to curated to be most effective.  How about idioms, expressions that do not translate literally?  Those are good, training sets should include them so that it will know later how to translate those in a way that is not literal.

Version 2 of the Custom Translator, which includes “human parity” models that PointFire uses, improved even further particularly for languages where Version 1 had not performed as well like Korean and Hindi, although less so for languages where it was already good like German.

Depending on the language and the field, particularly how your specialized vocabulary differs from the vocabulary that Microsoft used in its training, typically government documents crawled from the web, you can get small to significant improvements in translation quality by training your own model in Azure Custom Translator. However, it requires significant investment in time and resources to set up, and Microsoft charges 4-5 times what it charges for the regular translation engine.  It is not for everyone.

Z-code is part of Microsoft’s larger ambitious “XYZ-code” initiative.  It takes advantage of new technology for massive, massive neural networks of a scale barely envisioned a few years ago, billions or even hundreds of billions of parameters, made possible with the Microsoft DeepSpeed library https://www.deepspeed.ai/.  It’s thousands of times bigger than current translation models.  Rather than separately training individual models for English-French, English-Hungarian, English-Chinese, etc., it trains one massive model that learns about all language pairs at once.  In order to avoid duplication, the model teaches itself about features that are common to families of languages, and features that are common to all written human languages.  That way, even if it was not given enough examples from a particular language pair, it can extrapolate from examples in other related languages, and most likely get it right.  It is called MoE (mixture of experts) because it incorporates specialized competing smaller models, called “experts”, each of which may propose an answer but each of which specializes in a type of problems, and it has another component that is trained to be good at determining which of the experts will be right under different circumstances.

In the diagram above, the percent improvement seems to be improvement in BLEU score.  According to other research, this class of models is particularly good for correcting negation errors in German.  Note that 2) and 3) cannot be used together.

MTS is a back-end SharePoint service application that works in SharePoint Online and in SharePoint 2013, 2016, 2019, and SE on premise.  It is able to carry out machine translation of several different file types including Office files and html pages, and of plain text.  Because it is built into SharePoint, using it is free.  Since it is a back-end service, you have to look at the different front-end functionality that may be using it.

There are three places where the Machine Translation Service is available for you to use in SharePoint: translation of Variations, translation of Term Sets, and translation via the API or PowerShell.  Let’s look at these one by one.

If you have been using Variations on Classic sites for a while, you may realize that it used to be possible to set up machine translation for a Variation label, which could machine translate pages either automatically or on demand.  This uses MTS as a back-end.  In September 2018, the user interface to machine translate Variation labels or classic pages was removed from SharePoint Online.  The functionality itself persisted, that is to say if you had previously set up a Variation label with automatic translation before then, MTS continued translating any new pages automatically, but you simply couldn’t set it up for any new Variation labels.

If you have created a term set in the term store, and you have set it up to have working languages in addition to the default language, then you have the option to machine translate the terms in a term set on demand, using the MTS.

In the SharePoint Admin Centre, under Content Services, select “Term store”

At any point, you can add working languages to a term store.

Inside the term store, individual term sets provide the option to create translations to the other working languages.  Under Translation, select Manage

This will bring up the translation options.

One of the options is Machine translation.  Press Start.  You will then have to select one language to translate to.  You can come back later and choose the other languages one by one.

Translation is often quite slow.  It is using the MTS in the back-end.  You may have to wait a while and if you are the first person to use it that day, it might time out before the service has a chance to fully start up and you may have to start again.

Once the term set is translated, the text will be fully translated to the other languages and the translated version of the term site is (often) what will be shown to the user based on their language.

Without going into too much detail, the translation capability of the MTS is available via CSOM or REST.  That means it can be called from C#, JavaScript, or PowerShell.  The MTS API lets you translate either a short text or a supported document or even an entire folder or library, either using a stream or a SharePoint file URL.  In the back-end it is using MTS.  It is possible that one of the customizations or products that you use are calling this API.  If you have custom code or apps that generate some translation, there are good chances that it is using the API to call MTS.