hoakley Macs, Technology

How to enable, use and fix Visual Look Up

Visual Look Up is one of the features of macOS which uses Machine Learning (ML), and should just work. However, many users have reported that it doesn’t appear to be available on their Mac, or even stranger, that it works on one system but not others. This article explains what it requires, and how you should be able to use it.

VLU is capable of recognising paintings, different breeds of dogs and cats, well-known landmarks, and many flowers and plants. While it can recognise paintings within paintings, it isn’t able to recognise other objects like pets or landmarks within paintings. Unlike other visual recognition systems, images aren’t uploaded to a server, but perceptual hashes are used instead. This ensures good accuracy, low latency, and a high level of privacy.

Requirements

Visual Look Up (VLU) is only available in macOS Monterey (plus iOS/iPadOS), and its availability there is limited by version, language and location. In Monterey 12.0.1 and later, it should be available to Macs set to use the English language in the USA, at least. In Monterey 12.3 and later, it should be available to the following combinations:

  • English, in the USA, Australia, Canada, Indonesia, Singapore and the UK;
  • French, in France;
  • German, in Germany;
  • Italian, in Italy;
  • Spanish, in the USA, Mexico and Spain.

You may find that it works with other languages and countries, but those are the combinations officially supported at present.

According to Apple’s detailed listing, VLU should work on all Macs officially supported to run Monterey, although it works significantly quicker on M1 series models.

You don’t need to enable Siri for VLU to work, but one setting you should check is in the Spotlight pane. Select the Search Results tab, and ensure that Siri Suggestions is ticked. If it’s not, then VLU will be disabled.

vlu901

VLU relies on mediaanalysisd being able to make secure (TLS 1.3) outgoing connections to Apple’s servers through port 443. If you have a software firewall active, then you’ll need to allow those connections or VLU will fail.

Which apps work?

VLU only works in the following apps at present:

  • Preview, but not third-party image viewers or editors,
  • Photos, but not third-party equivalents,
  • Safari, but not third-party browsers which use their own web views,
  • Any third-party app which displays webpages or images in a WKWebKit view, including my own Mints.

How to access VLU

In Safari or a third-party WKWebKit view, Control-click on the image you want to look up, to see the contextual menu. After a moment’s delay, the bottom command to Look Up should be added.

vislookup1

Select that, and a new QuickLook Preview floating window should appear, with the image inside it. One or more white spots should be present on the image, and one of them should open automatically giving you the results of the look up. If there are multiple white spots, you can click on each to see the results for that object.

vislookup4

In Preview, open the image and wait until the Inspector tool in the window’s toolbar changes from a simple letter i in a circle, with the addition of a + sign at its upper left. Click on that to open the Inspector, and one or more white spots should appear on the image. Click on one of those to see the results of the look up on that object in the image. White spots will remain so long as the Inspector window is open.

vlu902

in Photos, open the image and wait until the Info tool in the window’s toolbar changes from a simple letter i in a circle, with the addition of a + sign at its upper left. Click on that to open the Inspector, and one or more white spots should appear on the image. Click on one of those to see the results of the look up on that object in the image. White spots will remain so long as the Info window is open.

Why no result?

Some images aren’t recognised by VLU, in which case the Look Up feature won’t be enabled. In Safari, that menu item doesn’t appear; in Preview or Photos, the Inspector/Info tool doesn’t gain the + sign.

Some images are recognised, but VLU returns no results. This occurs when the image isn’t identified, and no objects within it are found to be suitable for VLU.

Troubleshooting

Check that your system meets the requirements given above, and that your Mac has an internet connection. Further information can be obtained using my free utility Mints, which can display relevant log entries, and has its own WKWebKit views to test VLU with.

Full details are given of VLU’s internal processes in this article, and its failure here.

Thanks to hsavirndb and others who have kindly provided valuable information.