hoakley February 15, 2021 Macs, Technology, Updates

Full release of Mints 1.0 is now available, and diagnoses Big Sur’s Spotlight bug

I’m delighted at long last to release the first full version of Mints, my free collection of advanced diagnostic and system tools.

I have further enhanced its Spotlight diagnostic tools, and using those can now explain precisely what the bug is in late versions of Catalina and all released versions of Big Sur, which prevents Spotlight search of RTF documents.

Changes in this version are concentrated on those Spotlight diagnostics. In addition to performing search on test files generated by Mints itself:

I have added two more built-in test file formats, for Numbers spreadsheets and Pages documents.
You can add your own custom test files, enabling you to investigate problems with other document types and their Spotlight importers.
Check Search not only checks which files can be found using Spotlight, but also provides information on test file types and the Spotlight importers they use.

To illustrate their use, here’s how I have worked out why Spotlight can’t find content of Rich Text (RTF) files.

When you click on the Create Test button, Mints creates its special test folder, and copies the standard test documents, including a test RTF, to that. Click on Check Search, and it finds all the other test files, but not that in RTF format. It now confirms that the RTF test file is recognised as being of the correct document type, and uses the bundled Rich Text importer which is part of macOS.

Thanks to comments from John, who reports that RTFD documents do work correctly, I then opened the RTF test file and saved it in RTFD format, once using TextEdit, and a second time using Nisus Writer Pro. This time, Mints’ Check Search was able to find the RTFD versions correctly, although the RTFs remained absent from search results.

RTFD documents are actually folders which contain text and image content for the whole document. What I did next was to open the RTFD document in the Finder, and copy its TXT.rtf file containing the Rich Text content into Mints’ test folder. Check Search demonstrated that Spotlight was unable to find the set search text in the TXT.rtf file from an RTFD document.

This demonstrates clearly that the importer is able to parse the RTF file perfectly well, but due to a bug, it won’t attempt that unless the RTF file is embedded within an entire RTFD document. This is therefore a simple logic error in handling different file types, rather than anything more complex involving the parsing of Rich Text, and could probably be fixed in a few minutes. I believe that this was introduced in build 319.60.100 of the Rich Text mdimporter which shipped with macOS Catalina 10.15.6, and is perpetuated in build 326 in Big Sur 11.0.1.

It also confirms once again that Apple doesn’t test its software before releasing it. Anyone who attempted to test whether RTF content indexing worked would have immediately noticed this complete and abject failure. As a matter of principle, I will not therefore be encouraging Apple to continue releasing untested software, by filing a bug report. Such basic testing is the responsibility of Apple, not of users or third-party developers. Until Apple takes effective action to put its own house in order, I don’t see why we should help it perpetuate a broken engineering system and failed quality management.

Mints of course does a great deal more than just test Spotlight, and version 1.0 is now available from here: mints1
from Downloads above, from its Product Page, and through its auto-update mechanism.

15Comments

Add yours

1

Kurt J. Meyer on February 15, 2021 at 9:57 am

I was curious about Spotlight and .rtf documents. So I made a little test on my Big Sur 11.2.1 MacBook Pro. I used a search criterion from a document I just had saved. The new document (.rtfd) was found — no wonder, it contained the key word in the file name. But Spotlight did also find a .rtf document from 2013 where the key word appeared in the text only.

LikeLiked by 1 person
- 2
  
  hoakley on February 15, 2021 at 10:26 am
  
  Thank you.
  As we have seen, search of content in RTFD documents still works fine. The reason that your 2013 RTF was found is that it hasn’t been reindexed with the broken importer. That’s one of the clues which I’ve used to work out when this bug was introduced – all my previously indexed RTFs are still fully searchable, but none written to or updated since.
  Howard.
  
  LikeLike
3

EcleX on February 15, 2021 at 10:08 am

Many thanks! If possible, it would be also helpful if Mints showed a progress bar or animated icon while performing the Check Search, as well as showing in the Spotlight Query Search if it was successful. Now it shows four entry lines inside “/Users/~/Library/Caches/Metadata/Safari”. Tested with macOS 10.12.6 (16G2136) Sierra.

LikeLiked by 1 person
- 4
  
  hoakley on February 15, 2021 at 10:30 am
  
  Thank you.
  Unfortunately, Spotlight search doesn’t provide any estimate of duration at the outset, and that’s unpredictable, so a progress bar is not possible. I’ve looked at a spinner instead, but here at least, even with a very large Home folder, search takes just a very few seconds, so by the time the spinner appears, it will disappear again, making it somewhat pointless. Are you saying that search for such an unusual string takes a long time in Sierra still?
  Howard.
  
  LikeLike
  - 5
    
    EcleX on February 15, 2021 at 3:05 pm
    
    Thanks. In sierra it took more time. About 30 seconds or more. Maybe you could include something like “Searching… please wait” legend, or similar. I am using a fast iMac 27‑inch Retina 5K display at 4.2 GHz (iMac18,3 mid 2017), booting from internal 2TB SSD. No other application was running at the time. Actually, I thought that Mint was not doing the search, but I knew that something was running thanks to MenuMeters showing high CPU activity.
    
    LikeLiked by 1 person
    - 6
      
      hoakley on February 15, 2021 at 3:31 pm
      
      Wow – thirty seconds? On this iMac Pro with Big Sur, and a hefty Home folder, it takes three seconds before the window appears, and another second or so before all the results are shown.
      I’ll look at incorporating a circular busy spinner in the next update.
      When did you last rebuild your Spotlight indexes on that volume?
      The search performed by Spotlight should be blisteringly quick: I believe it hashes the search term, then looks in its indexes for that hash, which should be very quick indeed. The time is then taken retrieving the details for each of the hits.
      Howard.
      
      LikeLike
  - 7
    
    EcleX on February 15, 2021 at 4:02 pm
    
    Thanks. Spotlight searches are really fast using the Finder; virtually in real time. The Spotlight was re-indexed (since it got corrupted and Spotlight did not find some items that I was sure where in the disk) on December 2018 (maybe later as well, but in such a case I did not record it) and it took almost 8 h (34 GB index).
    
    LikeLiked by 1 person
    - 8
      
      hoakley on February 15, 2021 at 8:42 pm
      
      I think you may have a problem with your Spotlight indexes. Although it’s a long time since I last used Sierra, I can’t recall this type of search ever taking more than 30 seconds to complete. That’s slower than non-Spotlight searches like FAF, surely?
      Howard.
      
      LikeLike
  - 9
    
    EcleX on February 15, 2021 at 9:56 pm
    
    I have tried again several times and this is what happened:
    
    1. Clicking Create Test does nothing. No MintsSpotlightTest4syzFiles folder is created.
    2. Clicking Check Search sends the CPU over the roof and does not stop after several minutes, until I quit mints.
    3. Clicking Delete Test does nothing (as expected, since there is nothing to delete).
    
    I must say that I experienced the same the first time that I tried Mints this morning, but trying several times, it worked a and generated the previously reported experience of my previous posts. So, it seems that Mints is behaving in an erratic way in Sierra.
    
    LikeLiked by 1 person
    - 10
      
      hoakley on February 15, 2021 at 10:03 pm
      
      Thank you.
      I have absolutely no idea why the test folder isn’t being created. The code is really basic stuff, and works here every time.
      There isn’t even the folder being created at the top level of your Home folder?
      Bizarre. Has anyone else tried this on Sierra please?
      
      LikeLike
  - 11
    
    EcleX on February 15, 2021 at 11:24 pm
    
    I have tried again and this time the MintsSpotlightTest4syzFiles folder was created. After clicking Check Search, it took 1 minute and 4 seconds (CPU to the roof as shown by MenuMeters) to show the Mints General Information window with the information inside. So, it works sometimes, as I also experienced this morning. Weird.
    
    LikeLiked by 1 person
12

Thomas Tempelmann (@tempelorg) on February 15, 2021 at 12:32 pm

Have you explored replacing the importer with an older version? Does that help? If so, where does one place the importer?

LikeLiked by 1 person
- 13
  
  hoakley on February 15, 2021 at 1:34 pm
  
  No, Thomas, I haven’t.
  I think you’d have to remove the original importer to get an older build to work at all. As this only affects late versions of Catalina and Big Sur, with the latter in particular it would mean breaking the seal on the system and leaving the system unsealed, which is non-trivial and has several penalties (I think it disables FairPlay, for instance).
  Additionally, Apple doesn’t provide any older versions of Catalina, and I don’t keep them here either.
  If someone with time on their hands would like to try this, I’d be very interested to know if it works.
  Howard.
  
  LikeLike
14

Richard Glaser on February 16, 2021 at 6:21 pm

Hi:

Thanks for the mint tool. Question? Have you ever thought of supporting allowing the utility prompt for admin credentials and run when logged in a user session as a standard user? Similar to command line `sudo` or `su mac` would make it much easier using your tool if you run as a standard user or if you need to troubleshoot another user running as a standard user. If you can add that functionality, would be great to include with all your tools that require admin credentials.

LikeLiked by 1 person
- 15
  
  hoakley on February 16, 2021 at 7:27 pm
  
  Thank you. Yes, I’d love to, only it’s a bit more complicated to do that. It’s on my list of things to do, and once I’ve done it for one app, it should be easier for others.
  The snag is that it isn’t just a matter of using a sudo-like addition to the command – I do use that trick in LockRattler and SilentKnight, but it requires the calling of an AppleScript, and isn’t suitable for the sort of use here.
  The only way to do this in an app is through a privileged helper app, something you’ll see in many apps. And that’s far from trivial to do correctly. In fact, a lot of apps which have privileged helpers introduce a significant vulnerability, as writing such a helper app is so easy to get wrong.
  So one day, when I’m feeling up to it, I’ll try doing this. If I can get it right, then I’ll be able to reuse the code for my other apps. But that first hurdle is quite high.
  Howard.
  
  LikeLike

·Comments are closed.

Share this:

Related