Machine translation (MT) has been one of the grand problems in computing for the last seventy years. The dream that you could instantly read an accurate translation between any two languages has attracted many great minds and prodigious research. As anyone who uses current MT systems, even Google Translate, knows, we’re still a long way from that goal. In this little demonstration, I’m going one step further in requiring the initial text to be recognised from an image. And just to make this even more challenging, I’ll go from Japanese to English.

Throughout this article, I use screenshots from Japanese Wikipedia.

I first open the screenshot in Preview. As I pass the pointer over blocks of Japanese text, it changes to an I-beam, so I can select some of the content that Live Text has recognised in the pixels. I then select the block I want to translate, and Control-click to bring up the contextual menu. At the top are two useful options, to look the recognised text up, or translate it.

Translation isn’t as impressive as Live Text, though. In fairness, the original Japanese is a minefield for any translator, as it’s about different types of leukaemia, so it’s technical, and the terms it uses are easily misunderstood.

Here’s an even tougher task: vertical Japanese. Modern Japanese is most commonly written horizontally, using what’s called yokogaki. More traditionally, characters can be arranged in vertical columns, so-called tategaki. Instead of trying an old scroll, kanji arranged in columns on these buildings in Akihabara, Tokyo, present a serious challenge.

In this case, Live Text can’t cope with columns of kanji, but sideways-vertical Roman text is recognised perfectly.

Is there anything we can do to improve Live Text recognition or translation?

The only controls we have are in the Language & Region pane.

To help Live Text recognise tategaki, I tried adding Japanese as a secondary language, without success. Maybe if you can make it your primary language, that might help, but of course, if your understanding of Japanese is sufficient to run macOS in the language, you don’t need these aids.

Translation has two options: the default is online, with the selected text being translated remotely. Click on the Translation Languages… button and you can bring it offline instead.

Tick the On-Device Mode box, then download at least the two languages you want to support. Once they’re ready, click Done and try them out.

This definitely improved the quality of the translation, although it still struggled with those technical terms.

Apple clearly hasn’t solved the grand problem of machine translation, but what we have on our Macs and devices is getting really useful, even when pushed to its limits. Live Text copes well with modern text layout, but after seventy years of work, machine translation still can’t match a skilled human translator.