The tricky part is that you need to keep all those files in sync, so it gets more complicated with the more languages you need to support. Our PointFire Localizer solution helps you localize your SPFx solution, so you do not have to worry about missing translations.

PointFire Localizer is a GitHub Action that helps you translate your localization files for your SPFx solutions. The Azure AI Translator is used to translate your localization files to the desired languages.

The GitHub Action adds the missing translations to your localization files so you can be sure that all key/value pairs are included in all languages. When a human has already translated a key, it will not be overwritten by machine translation. The GitHub Action favors human translations.

Our GitHub Action is designed to be easily integrated into your existing build pipeline. Thus, you can automatically translate your localization files when you build your SPFx solution.

As our builds are already running on GitHub Actions, we created a GitHub Action that can be used in your workflows. This way, you can easily integrate the localization process into your existing build pipeline, and you are sure that on release, all localization files will include all key/value pairs for all languages.

Here you can see an example of a GitHub Actions workflow for packaging a SharePoint Framework solution:

The prerequisite for using PointFire Localizer is that you have an Azure AI Translator service. You can create a new service in the Azure Portal. Once you have created the service, either the free or the paid tier, you can get the API key from the Azure Portal. You can find more information in the get your authentication keys and endpoint article.

You can make use of the free tier from Azure AI Translator.

Follow the next steps to add your Azure AI Translator API key to your GitHub repository:

• Go to your GitHub repository
• Go to the `Settings` tab
• Go to the `Secrets and variables` section and click on `Actions`
• Click on the `New repository secret` button
• Add a new secret with the name `TRANSLATOR_API_KEY` and the value of your Azure AI Translator API key

Once you have added the Azure AI Translator API key to your GitHub repository, you can add the PointFire Localizer GitHub Action to your workflow.

There are two ways how you can use PointFire Localizer in your workflow:

1.  By a predefined list of locales

2.  By automatically detecting the locales

If you want to use a predefined list of locales, you can add the following step to your workflow file right after the `install dependencies` step:

```yaml

- name: PointFire Localizer

 uses: IceFireStudios/pointfire-localizer-action@v1.0.0

 with:

 api-key: ${{ secrets.TRANSLATOR_API_KEY }}

 api-region: "westeurope"

 default-locale: "en-us"

 locales: "nl-nl,fr-fr,de-de"

 summary: true

```

In this example, we use the predefined list of locales `nl-nl,fr-fr,de-de` and the default locale `en-us`.

The GitHub Action will do the following:

· It will first look for all the `en-us.js` files in the SPFx solution

· It will then translate all the missing or empty key/value pairs to the locales `nl-nl`, `fr-fr`, and `de-de`

o If a localization file does not exist, it will create a new one

o If a localization file already exists, it will add the missing key/value pairs

The advantage of this approach is that those localization files can be created during the build process, so you do not have to make them manually if there are no human translations available.

If you want to detect the locales automatically, you can add the following step to your workflow file right after the `install dependencies` step:

```yaml

- name: PointFire Localizer

 uses: IceFireStudios/pointfire-localizer-action@v1.0.0

 with:

 api-key: ${{ secrets.TRANSLATOR_API_KEY }}

 api-region: "westeurope"

 default-locale: "en-us"

 summary: true

```

In this example, we are using the default locale `en-us`. As we did not specify any locales, the GitHub Action will automatically detect them based on the existing localization files.

The GitHub Action will do the following:

· It will first look for all the `en-us.js` files in the SPFx solution

· It will retrieve the linked locales per the default locale

· It will then translate all the missing or empty key/value pairs to the linked locales

In this case, the GitHub Action will only translate the existing locale files in the SPFx solution.

Generative Pre-trained Transformer (GPT) systems are not designed to be translation engines.  So it is surprising that they succeed so well at doing simple translations.  Some articles have claimed that they can translate better than existing translation engines.  How true are those claims?

The best comprehensive evaluation was done by scientists at Microsoft Research, unsurprising because they are among the leaders in both machine translation and GPT models.  The brief summary is that while GPT models have competitive quality when translating usual sentences from a major (see high-resource below) language to English, they are less good at other types of translation.

The evaluation uses three GPT systems that are known for translation quality, and compares them with neural machine translation engines (NMT), either Microsoft Azure API or the best-performing commercial systems or research prototype.  Quality scoring uses either algorithmic scoring or human evaluation.

How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation 

In most languages and for language directions, MS Azure Translator and other NMT engines outperform GPT for most measures of quality.  However, GPT does have the ability to improve after being given a few examples of correct translations, and to outperform NMT in some language directions after 5 tries.  This is the case for translations to English from German, Chinese and Japanese, languages for which there are a lot of examples in the GPT training set.  These are called “high-resource” languages.  On the other hand, it does not do particularly well for low-resource languages like Czech or Icelandic, or for English to other languages.  GPT’s training set had less text in those languages.  In the chart below, orange is GPT and blue is NMT.  The lines are algorithmic evaluation and the bars are human evaluation.

Those experiments above are for sentence-level translations.  For multi-sentence translations, those with more context that can be found in other sentences, GPT improves relative to NMT.  Not enough to beat the best NMT systems, but sometimes enough to match or beat the normal Azure API.  That is not very surprising: Azure Translator was optimized for sentence-level translation, while GPT is trained for multi-sentence context, up to thousands of words.  Other Azure translation APIs like Document Translator and Custom Translator are better at longer context windows, but this is not what was tested here. 

Neural Machine Translation models have another big advantage over GPT: they can be re-trained for a particular domain rather than with general domains, and this significantly improves their quality score for that domain.  For example, by giving it many training examples of automotive documents, Azure Custom Translator (a re-trainable version of Azure Translator) can increase its translation quality for documents in the automotive domain by a large factor.

Looking at other measures of performance gives a better understanding of what is different in the performance of the GPT vs. NMT.  Measuring fluency, essentially how natural is the sentence, how similar it is to other sentences out in the world, tells you something about the quality of the prose.  GPT is more fluent in English, it sounds more natural.  That doesn’t mean that it is a more accurate translation, far from it.  In fact, GPT has a greater tendency to add words, concepts, and punctuation that do not correspond to the original, or to omit some.  So it’s good prose, but it’s not necessarily what the original said.  For example, it does better at figures of speech, by not translating them literally, but also does not necessarily replace it with a term that means exactly the same thing.  It does not wander far from the original with completely made-up things, but it’s often not quite correct.  However GPT also does hallucinate words or concepts that were not in the original.

A literal and faithful translation is sometimes required, but that often leads to what is called “translationese”.  This refers to a set of common issues with text generated by human translators.  Translationese can refer to excessive precision or wordiness, or excessive vagueness in translated text, or syntax that is uncommon in the target language.  What translators are compensating for is the fact that different languages are specific about different things.  For example, English has the term “uncle”, which does not differentiate between paternal and maternal uncle or uncles by blood or marriage, but many other languages are much more specific.  Translating from those other languages to English, Translationese would not say “uncle” but might say “maternal uncle by marriage”, a term that is unusual in English, but which avoids losing information that was contained in the original.  In terms of translation quality, humans who are not translators might rate the translation with “uncle” higher because it sounds more natural, but translators would rate the awkward translation higher because it is more accurate.

Sometimes We Want Translationese

The design of the two types of models is also different. The “T” in GPT stands for “Transformer”.  Transformer is an attention-based neural network model that when looking at a word within a sentence or even longer text, determines which other words are the most relevant ones to pay attention to.  NMT also uses Transformer models.  However, there are big differences.  One is that GPT uses Decoder models, while NMT uses Encoder-decoder models.  What does that mean?  Decoder models focus on the output, the next word to be spit out.  Encoder-decoder models try to extract features from the input before feeding it to the part of the model that predicts the output.  It focuses separately on the input and on the output.  It tries to be robust to small changes in the input.

GPT only outputs one word at a time.  It starts with the text of the prompt plus other information in the input, then outputs a single word.  Then it adds that word to the end of the prompt in the input and puts this new input through again to get the next word.  It is unidirectional, that is to say when it is generating text it only looks at the previous words that are already generated, it doesn’t consider what it will say next because it hasn’t said it yet. Like a lot of humans, GPT is more concerned with what it wants to say next than with what you’re saying.  NMT is bidirectional.  It considers the rest of the sentence and the next sentence in both the source and the translation while it is generating the text.  It generates entire sentences at once.

Because it is a generative model, GPT is biased towards what is usual.  If the original text in the other language is commonplace and expected, then GPT will find good ways to express that text in English in ways that are commonplace and expected, because that is what it is trained to do.  If the original says something that is unexpected or expresses it in unexpected ways, GPT’s translation is likely to replace it with something more usual using some of the same words.  GPT does well at translation essentially because most things that require translation are predictable and unoriginal.

NMTs have a whole bag of tricks to deal with translation tasks that GPT does not, including specialized knowledge about the structure of languages, and tricks to deal with numbers, capitalization, and non-standard spacing correctly and efficiently.  They are also trained to preserve information.  You know that trick that people sometimes use, translating a sentence to another language then back to English so they can laugh at the result?  NMTs include that round-trip in their training, to make sure that none of the meaning gets lost in the translation.  Other tricks include having the neural network teach another neural network how to translate, detecting errors in the training data, and other tricks that address common translation errors.  There are also tricks to reduce gender bias, a problem that still plagues GPT.

Current Azure Translator NMT uses models of about 50 million parameters, which can run 4 language pairs in a Docker container on a host having a 2-core CPU with 2 GB memory.  Even the next generation of NMTs, Z-code MoE, which have 100 languages (10,000 language pairs) in a single model, can fit on a single GPU even though they have billions or hundreds of billions of parameters.  These are sizes for querying, what is required for training is much bigger.  GPT-4 uses 100 trillion parameters.  Training requires hundreds of thousands of CPUs and tens of thousands of GPUs, but to query them, it looks like a single cluster of 8 GPUs and a dozen or two CPUs is what is required.  Microsoft is very good at shrinking by orders of magnitude the size of machines required to run AI models so direct comparison is difficult, but NMTs deliver translations at much lower computational cost.  Microsoft’s DeepSpeed library in particular increases speed and reduces latency by a large factor.

The computing power required also has a potential impact on security.  NMTs, even the bigger potential NMTs can be run on a single processor, while GPT requires many processors.  Using GPT you are probably sharing hardware with strangers, while for NMT it is possible to have dedicated resources.  Because of its recurrent architecture, where the output is fed back into the input, GPT probably has some static storage of your data, while NMT can be architected with a pipeline where neither the input nor output text is ever stored.  I don't know how it is implemented by anyone, but I notice that for Azure, NMT has a no-trace option by default while GPT limited access previews do not.  Because of ethical concerns, data is probably retained for abuse monitoring.  I'm sure the security is good, but the architecture reduces the options for security.

The blanket claim that GPT is better at translation in not generally true.  However GPT is surprisingly good, considering that translation is a task that it was neither designed nor trained for.  It is unexpected that it is sometimes equal to or better than the highly specialized NMTs.  There is a fair bit of work being done on hybrid systems that combine the accuracy and specialized training of NMT with the fluency of GPT and will deliver the best of both. The next generation of NMT (see How to get a higher level of machine translation quality) will also allow the model to transfer language knowledge obtained from one language to other related languages, and in that way vastly improve the quality for low-resource languages such as southern Slavic languages.  That innate knowledge of what is common to languages in the same family can then be used to improve the quality of both NMT and GPT.

For some apps “From your organization”, particularly the ones using the Add-in model in my case, there are some “You can’t add this app here” messages.

Usually “Find out why” is not that helpful, but for one such app, it gives an actual reason

We have applied the technique from the Language-dependent JSON column formatting blog post so that the choice column values are localized into French and German.  However as we can see below, this does not extend to the “group by” headers, those remain in English.  The column value, in blue, is localized, but the same value in the header is not.

The display of values in the columns can be customized by column formatting JSON, but display of values in the group-by headers must be customized by view formatting JSON.  Let’s have a look at how we can change that formatting.  In the View menu, we can find “Format current view” in the bottom.  In this case we are going to format the AllItems (default) list view.

By default it offers the graphical “design mode”.

Let’s select “Advanced mode” to see the JSON.  It is initially empty other than the schema declaration, so it uses some default formatting.

We see “Progress=New application” But in French we get “Progrès=New application”

The “columnDisplayName” is localized by the MUI, but the column value is not localized by our column formatting.  We can put this all together to take advantage of the display names.  In the nested ifs below, we check the value of “@group.columnDisplayName” to determine the current language, then the current value, and then display the appropriate translation of that value.

Here is the result, in French

And in German

Language-dependent JSON column formatting: this time it's the @UIlcid token

Language-dependent JSON column formatting using the @lcid token

MTS is a back-end SharePoint service application that works in SharePoint Online and in SharePoint 2013, 2016, 2019, and SE on premise.  It is able to carry out machine translation of several different file types including Office files and html pages, and of plain text.  Because it is built into SharePoint, using it is free.  Since it is a back-end service, you have to look at the different front-end functionality that may be using it.

There are three places where the Machine Translation Service is available for you to use in SharePoint: translation of Variations, translation of Term Sets, and translation via the API or PowerShell.  Let’s look at these one by one.

If you have been using Variations on Classic sites for a while, you may realize that it used to be possible to set up machine translation for a Variation label, which could machine translate pages either automatically or on demand.  This uses MTS as a back-end.  In September 2018, the user interface to machine translate Variation labels or classic pages was removed from SharePoint Online.  The functionality itself persisted, that is to say if you had previously set up a Variation label with automatic translation before then, MTS continued translating any new pages automatically, but you simply couldn’t set it up for any new Variation labels.

If you have created a term set in the term store, and you have set it up to have working languages in addition to the default language, then you have the option to machine translate the terms in a term set on demand, using the MTS.

In the SharePoint Admin Centre, under Content Services, select “Term store”

At any point, you can add working languages to a term store.

Inside the term store, individual term sets provide the option to create translations to the other working languages.  Under Translation, select Manage

This will bring up the translation options.

One of the options is Machine translation.  Press Start.  You will then have to select one language to translate to.  You can come back later and choose the other languages one by one.

Translation is often quite slow.  It is using the MTS in the back-end.  You may have to wait a while and if you are the first person to use it that day, it might time out before the service has a chance to fully start up and you may have to start again.

Once the term set is translated, the text will be fully translated to the other languages and the translated version of the term site is (often) what will be shown to the user based on their language.

Without going into too much detail, the translation capability of the MTS is available via CSOM or REST.  That means it can be called from C#, JavaScript, or PowerShell.  The MTS API lets you translate either a short text or a supported document or even an entire folder or library, either using a stream or a SharePoint file URL.  In the back-end it is using MTS.  It is possible that one of the customizations or products that you use are calling this API.  If you have custom code or apps that generate some translation, there are good chances that it is using the API to call MTS.