As computer storage grew during the late 1980s, the ability to search its contents became increasingly important. Macs had the Find File application, which was integrated with the first version of Sherlock in 1998, which added internet search. While local file search was retained in Sherlock 2, its most prominent advances were in adding more internet search channels.

This changed just over 21 years ago, in April 2005, when Spotlight replaced Sherlock 3, to keep pace with increasing demands being placed on search. If you had bought a newly introduced Mac mini that year, the largest hard disk it would have come with had a capacity of just 80 GB. Today’s Mac mini is offered with an internal SSD of up to 8 TB, 100 times that of its ancestor, and it’s not uncommon for a user’s Home folder to exceed 1 TB in over 250,000 files. It’s also estimated that in the previous year 2004, the number of websites on the internet rose above 50 million.

Information retrieval is now a fundamental requirement of computers, whether you want to check your next appointment on your watch, or to discover how APFS makes a snapshot. Successful retrieval depends on appropriate search methods, and that in turn needs you to choose how to attain each search goal. In broad terms, that can adopt one of two popular approaches, either to exhaustively list all items that meet a set of criteria, or to generate a ranked list selected as those most likely to satisfy the search.

For example, when I search in the Finder’s Find window for files whose content includes the term APFS, I get an exhaustive list of 2,925 items, and adding a filter requiring the Kind of file to be PDF narrows those to a more manageable 24, making it quick to select from among them.

Entering the same search term in Spotlight’s window offers me two text log files ranked as being best hits, nine additional documents without any PDFs, three articles from the Tips app, a link to an external website, and various others including conversations in Messages and emails. If I limit that search to include only PDFs it offers me just 20, ordered according to its ranking algorithm. Presumably the other four found in the exhaustive search weren’t ranked highly enough to merit inclusion here.

As shown in the screenshot above of the initial release of Spotlight, from the outset it has combined both approaches, there offering a Top Hit from its search ranking, and all 27 hits in its exhaustive list as another option.

Last week I started to dig deeper into how Spotlight does this in macOS Tahoe, partly in preparation for the changes anticipated in Golden Gate’s semantic search. Already that has revealed the qualities of what would be offered as a Top Hit in its ranked search, with scores such as
mrScore: 0.2516, mrScoreText: 0.6878, kMDItemDisplayName: 1.0000, mrScoreFreshness: 0.50000000, mrScoreResultType: 0.0700, freshnessDate: (null), pommesL1=0.0000, L2=-4999.0283
While most of those are opaque, it’s notable that ‘freshness’ is included in ranking. In contrast, searches made in the Finder’s Find window are unranked, and don’t calculate any of those scores, nor take into account freshness.

As I’ll describe in greater detail next week, log entries reveal fuller details of how macOS calculates its measure of freshness. This depends on a non-linear function of engagement_date, apparently expressed in hours since a time exactly 31 years ago. Those appear to be totalled for events such as file and website access, which must be recorded in a local database maintained for this purpose.

Unranked exhaustive search such the Finder’s Find is only as good as the user’s ability to add filters or predicates to reduce the number of hits, and that requires prior knowledge of the targets of that search. It can also mislead, as a document that contains a single use of a term, perhaps buried in a footnote, is given the same weight as another in which the same term appears on every page.

We have come to accept, for better or worse, that unranked exhaustive search of websites has long since become impossible. As the number and contents of local files continues to grow, that strategy is becoming increasingly difficult for them too. It’s not yet clear, though, whether Spotlight’s ranked search is up to the job, or whether interactive refinement using Siri and AI will prove any better. What I’m sure we’ll all agree is that Spotlight’s eternal quest has never been more important.