hoakley August 16, 2020 Macs, Technology

Last Week on My Mac: Apple leak

There are some gaffes no operating system developer can afford to make: one of them is to leak kernel memory. If that were to happen, every one of your users would be running a time bomb, which ticks and leaks away until memory runs dry, and the only way out is for the kernel to panic and take the whole system down. Anyone relying on your operating system for 24/7 services will suffer repeated outages over which they have no control, which damage them and their reputation. Kernel memory leaks are a highly effective way of driving away customers present and future.

Yet on 15 July, Apple did just that in macOS Catalina 10.15.6 update. The combination of a kernel which had been built only ten days before that public release and a set of new graphics drivers set up conditions which on some Macs at least resulted in a large leak in Mach zones, ready-use pools of memory dedicated to the kernel and its extensions. Although first reported on Macs which were running specific virtualisation software, it has since become clear that many more users were suffering as a result.

Apple redeemed itself in part by its swift and effective response: within a couple of weeks of receiving first reports of the problem, it released a Supplemental Update which appears to have restored stability to the kernel memory zones.

Looking inside that update is another revelation: the suspect graphics drivers, which in 10.15.5SU were at version 3.9.5 and progressed to 3.10.5 in 10.15.6, had been ‘fixed’ in a minor patch to version 3.10.6 in the Supplemental Update to macOS 10.15.6. In contrast, the stable kernel to accompany them had gone back to an earlier version from more than two weeks before that in the 10.15.6 update, having been built on 18 June.

More worrying is the fact that Apple doesn’t appear to have recognised that this memory leak affected more Macs than just those running VMware or VirtualBox: its description in the terse note describing the Supplemental Update states that it “fixes a stability issue that could occur when running virtualization apps”. And in Apple’s growing dictionary of euphemisms, a kernel panic is now considered to be a “stability issue” as well as an “unexpected restart“. Perhaps if we returned to calling a spade a spade Apple might treat its latest failure with the seriousness which it deserves.

We’ll never know how this fatal flaw in the 10.15.6 update leaked from Apple, but it once again speaks forcefully of lack of testing and absence of quality control, which have bedevilled macOS of late.

Some features and apps in macOS are notoriously difficult to test adequately in betas. The Mail app and Time Machine backups are two recent cases which have also plagued Catalina. Both were radically revised for this major release. As far as Mail was concerned, this wasn’t an upgrade or update but involved the removal of popular features such as controls in Column View. For Time Machine it was the opposite, with a marked increase in complexity which has left some users simply unable to make stable backups any more. At least Big Sur promises to address these, with the exciting prospect of at last being able to store backups on APFS-formatted volumes.

When it comes to testing the kernel and its extensions, Apple holds all the cards with special debug versions of the kernel, and engineers for whom tools like zprint are second nature.

The deepest irony with this kernel memory leak, though, is Apple’s current campaign to drive third-parties away from using kernel extensions. Big Sur is still promised to allow them, but their days are clearly numbered, with the substitution of System Extensions and their relatives, which run in user rather than kernel space. One strong argument in favour of Apple’s new regime is that it will ensure that no third-party extensions will be able to perpetrate the kernel memory leak and inevitable panic which macOS 10.15.6 introduced.

Now’s the time for Apple to address its own shortcomings in improving its pre-beta testing and exercising much better quality control. Otherwise we could be looking forward to more leaks in the future.

26Comments

Add yours

1

Steven Kapplin on August 16, 2020 at 7:32 pm

I agree with your last paragraph, wholeheartedly. Most of the Beta seed testers spend time ragchewing instead of testing. Apple no longer chooses Beta seed testers based on expertise. This is one important reason, I think, for the lack of sufficient QC. And, if there were more effort on Apple’s part to spend more time squashing bugs rather than adding eye candy, then the OS might not be so buggy.

Just my personal opinion after many years of seed testing. I lost interest a few years ago when the seed forums became like the Apple Support Community.

LikeLiked by 1 person
- 2
  
  hoakley on August 16, 2020 at 8:39 pm
  
  Thank you. I’m sorry that Apple lost your diligent testing.
  Howard.
  
  LikeLike
- 3
  
  Tony on August 16, 2020 at 10:24 pm
  
  The extreme example of what you quote is the public beta programme. I cannot believe that Apple receives quality feedback from that nor that it can analyse the quantity of feedback, short of a few crude statistics like ‘it crashes a lot’. The timescales between beta releases and gold master/public release also suggest little opportunity to analyse feedback (visible in Howard’s comments about a kernel that was just 10 days old when released).
  
  Quality should trump quantity here, not least so Apple’s engineers can see the wood through the trees.
  
  LikeLiked by 1 person
4

Jeremy Gurofsky on August 16, 2020 at 9:48 pm

Howard, you mentioned VM Ware Fusion and Virtual Box; do you reckon Parallels would likely incur the same difficulties? I ask because I had hoped to virtualize Mojave in order to run iTunes & to access my 2 TB library including all the metadata (curating), as well as all the internet radio.

Thanks,

Jeremy

LikeLiked by 1 person
- 5
  
  hoakley on August 16, 2020 at 9:53 pm
  
  Funnily enough, I’m in the middle of reviewing Parallels 16.0, and haven’t encountered any such problems with it. So long as you’ve installed the Supplemental Update to 10.15.6, I think they all should work fine now.
  Howard.
  
  LikeLike
6

Jeremy Gurofsky on August 16, 2020 at 10:21 pm

Encouraging . . . I shall try it & report in due course.

Thank you as always!

Jeremy

LikeLiked by 1 person
7

EcleX on August 17, 2020 at 8:29 am

Curiously, these and other macOS issues-flaws-bugs have increased in the past years, right when Apple should have been developing and optimizing macOS for ARM-based (Apple Silicon) Macs. Hopefully, the situation will improve once the transition from Intel x86 is over in two years or earlier!

Besides, such new scenario means that all Apple gadgets (Apple Watch, iPod tourch, iPhone, iPad and Mac) work with the same OS and hardware foundation, which should simplify development tremendously. Actually, that (besides reduced hardware costs) should have been the top reasons for such a switch to ARM Macs. Not the announced increase in battery life or performance, which probably is even lower for the latter when compared to powerful Intel x86 chips. In other words, Intel x86 computers will continue to be more powerful than ARM-based ones. Indeed, x86 is designed more for performance first, with power being more of a secondary consideration. ARM is designed for power efficiency at the cost of performance. ARM systems could throw more cores out to increase performance, but most applications cannot use them in parallel!

LikeLiked by 1 person
- 8
  
  hoakley on August 17, 2020 at 9:27 am
  
  Thank you. I’d like to question those assertions.
  As no one knows how many bugs are extant in macOS, to claim that they have “increased in the past years” appears unfounded. If you could point me to numerical evidence, I’d be very grateful, but the impression that I get is that no one knows. What you are saying is like looking at the babies in a maternity hospital and claiming that the population must be increasing, without counting how many are born and how many people die. How many bugs in Mojave have been fixed in Catalina, for instance? I know of several in PDFKit alone, for instance.
  You then say that “all Apple gadgets … work with the same OS” which isn’t true at all. The macOS kernel and much of what runs above it from extensions upwards is very different from those in iOS or iPadOS. Indeed, the fact that Apple has split iPadOS from iOS indicates divergence rather than convergence on a common OS.
  I suspect that your claim that Intel processors are “more powerful” than ARM-based ones is repeated from Intel marketing literature. It’s frankly rubbish. Apple develops its SoCs for specific hardware applications – and those which you have seen so far haven’t been for desktop computers. ARM processors were originally developed not for low-power devices but for desktop systems, and are currently being used in servers. Even in iOS devices you’ll see some of the cores are optimised for performance rather than low energy use. Have you seen the performance specs of processors in Apple Silicon Macs? If so, I’d love to see them, please.
  Finally, you claim that most apps can’t use cores in parallel. If that were the case, then Intel would also be doomed, as it now relies almost entirely on multi-core processors. In fact, every app which you use is now designed to run on multi-core systems, and the great majority are coded to make best use of many available cores. Even my little utilities are, and macOS itself makes excellent use of all available cores too. The days of the single-core monolithic app are long since past, thankfully.
  Howard.
  
  LikeLike
  - 9
    
    EcleX on August 17, 2020 at 11:24 am
    
    It seems that I did not explain properly. I meant:
    
    – In my experience over the years, macOS has more bugs in recent years. One example is macOS 10.15 Catalina. Apple took years (at least 18 months) to release new Mac versions in the past, whereas now does it every year to keep pace with iOS. Certainly, Mac programmers must start building the new OS before the current one is fixed, thus the problem.
    
    – All Apple gadgets have the same OS X Unix root. Obviously, each one have differences depending on the interface, display area, features, etc.
    
    – We will see how powerful are the ARM-based Macs when released and compared to Intel x86 (and AMD x86). My guess is that the latter will be more powerful, albeit at the cost of higher Thermal Design Power (TDP). Also interesting will be the battery time comparisons, for which I expect Macs to win.
    
    – To cope with power benchmarking, Apple will need to add more cores than x86 architectures. Unfortunately, most applications do not take advantage of all cores if there are many of them. And some tasks cannot be parallelized anyway. You can see it for instance installing MenuMeters and running applications like DiskWarrior. Some steps are parallelized, but not others because they cannot or because the developer has not done that. Imagine a CPU with 64 cores and even more. There are now GPU with thousands of them, but in this case parallelization is usually easier (video rendering, etc).
    
    Last but not least, these are my opinions-predictions based on experience over decades. Of course, I may be wrong and eventually, time will tell. We have exciting times ahead for sure! And thanks for this great site and your amazing work!
    
    LikeLiked by 1 person
    - 10
      
      hoakley on August 17, 2020 at 12:06 pm
      
      Thank you. You made yourself perfectly clear.
      1. On the number of bugs in Catalina, you have no evidence, just a feeling about it. How much do you use Catalina, and since when? I’ve been using it on my production system since Boxing Day, and on my other system since June last year. I wouldn’t hazard a guess at whether it has more or less bugs than Mojave. Indeed, the whole point of this article is not about quantity, but the severity of this particular bug.
      2. “All Apple gadgets have the same OS X Unix root” doesn’t mean much to me, I’m afraid. Actually, in architectural terms it’s more precise to refer to Mach root rather than Unix. And the modern macOS kernel is far removed from its origins. Differences from Mach and between the different Apple OS are far deeper than just the interface. Did you know, for example, that iOS doesn’t use virtual memory in the way that macOS does? That’s a huge difference in terms of memory management.
      3. “My guess is…” confirms that you haven’t seen any performance specifications. On what are you basing that “guess”? Have you even assessed a DTK, for example?
      4. You state that “Apple will need to add more cores”. Why? Is this your evidence-free assumption again that Intel cores are inevitably more powerful that ARM ones could ever be? Yes, some tasks can’t be run in parallel (which applies equally to Intel and ARM processors), but to cite a specialist disk repair tool as a good example is extremely misleading. How about major apps, which most users run most of the time? Adobe, Microsoft and Apple apps do make very good use of multiple cores. Hell, even SilentKnight does! If current software can’t use multiple cores effectively, then that affects all modern processors, not just ARM.
      Howard.
      
      LikeLike
  - 11
    
    EcleX on August 17, 2020 at 3:39 pm
    
    Again, I think that I did not explain myself. Of course that there are differences. They are not identical. On the other hand, ARM needs more cores to reach the same power than x86. That is a fact. And yes, you cannot compress-expand files using parallel processing. Not because it is impossible, but because applications do not do it. Again, I just showed some examples for the sake of the discussion. There are many others. And all in all, we will see. Time will tell. Again, those are my opinions-predictions based on decades of experience.
    
    LikeLiked by 1 person
    - 12
      
      hoakley on August 17, 2020 at 7:08 pm
      
      Thank you. You did explain yourself, and you are continuing to make an assertion without any evidence, which also contradicts the facts about Intel Macs (and any other computer).
      I paid a lot for this iMac Pro, which has eight cores. Are you telling me that I wasted my money, as four would have been just as good, because my software can’t use multiple cores? What difference does it make that the eight or ten or fourteen cores are Intel rather ARM?
      You claim that you can’t compress/expand files using parallel processing, yet that’s exactly what is now used in modern still and video image formats, which are designed to make things like pipelines accelerate the process to the point where it can be done in real time.
      Have you ever heard of the Inmos Transputer? It was the world’s first large-scale manufactured parallel processor, back in the mid 1980s. Well, that was what I cut my programming teeth on, writing code in the parallel language occam, before I switched to the Mac.
      Howard.
      
      LikeLike
  - 13
    
    EcleX on August 17, 2020 at 10:15 pm
    
    Thanks. What I meant is: what are the most powerful x86 vs ARM microprocessors now? As far as I know, they are the former, albeit with hugely higher TDP. Of course, time will tell if that changes in the future and how is it accomplished (for instance, using more ARM cores).
    
    In relation to compression-expansion, I meant applications used by people to do that with files. I have only found pigz at
    A parallel implementation of gzip for modern multi-processor, multi-core machines
    https://zlib.net/pigz
    
    But its compression efficiency is lower than the one of other applications, as shown at
    Parallel lossless compression using GPUs
    
    Click to access S4459-parallel-lossless-compression-using-gpus.pdf
    
    LikeLiked by 1 person
    - 14
      
      hoakley on August 17, 2020 at 10:56 pm
      
      Thank you.
      Apple hasn’t yet released any SoCs with ARM cores for use in macOS systems, has it? Until we know their performance, we’re comparing SoCs developed specifically for low-power mobile devices against those developed for high-end workstations and servers. One feature of RISC processors which Apple hasn’t been able to use until now is the fact that, with their low heat generation, you can clock them at significantly higher speeds than regular CISC processors and you don’t have to devise sophisticated cooling systems for them. I don’t know what clock speeds Apple intends the cores in Apple Silicon desktop systems to be. Do you?
      The gzip algorithm wasn’t designed for implementation in parallel systems. Compression used in modern still and moving images is, and is actually far more commonly encountered, and a greater limitation to performance in computers. I don’t think anyone would seriously suggest using gzip to compress TB of video, would you?
      Anyway, we’re straying far from the issue. Fact is that modern operating systems and apps are designed to run efficiently on multi-core processors. If they weren’t, then both Intel and ARM processors would be ineffective, as they both rely on efficient use of multiple cores, typically now at least six in any decent high-performance computer, and in high-end systems a dozen cores or more. And neither of us knows what the performance of Apple’s SoCs will be, other than the fact that Apple has confidence that within two years it will have been able to replace all its current Intel systems with Apple Silicon ones. That includes high-end MBPs, iMac Pro, and Mac Pro.
      Howard.
      
      LikeLike
  - 15
    
    EcleX on August 18, 2020 at 6:27 am
    
    Thanks. Time will tell. On the other hand, do you know any application (for Mac or other Unix, Windows or Linux) capable of compressing-expanding or compressing-splitting-expanding-joining files in parallel? I mean, a general-purpose one like pigz. For instance, for huge files (3 TB) with 7 milliards (thousand million) short text files inside. That is for bioinformatics genome assembly purposes.
    
    LikeLiked by 1 person
    - 16
      
      hoakley on August 18, 2020 at 6:42 am
      
      That sounds like it’s I/O rather than processor bound. The file system is really going to struggle with 7 billion short text files whatever the processor might try to do.
      As to its compression, no general purpose compression algorithm is going to do well – as with images and audio, you’ll get much better performance from an algorithm designed for the purpose and underlying data.
      Howard.
      
      LikeLike
  - 17
    
    EcleX on August 18, 2020 at 9:58 am
    
    Thanks. Do you know any algorithm to do that?
    
    On the other hand, I thought that the word billion meant a million of millions (10E12) instead of a thousand of millions (milliard; 10E9) in the UK. At least, it was like that in the past. I know that in the USA they use billion for million of millions. Due to that, the word billion is confusing/ambiguous worldwide, since it means 10E9 or 10E12, depending on the country.
    
    LikeLiked by 1 person
    - 18
      
      hoakley on August 18, 2020 at 10:59 pm
      
      No – but it’s not been my specialist field. There’s an excellent account of coding theory, which is the basis of compression and more, in Information Theory, Inference, and Learning Algorithms, by David JC MacKay, ISBN 978 0 521 64298 9.
      Regarding billions, you’re correct in that British English used to use billion to refer to 10^12, and US English to 10^9. However, for some decades now the international standard usage has settled on 10^9, and trillion as 10^12. Milliard exists in English, but it’s pretty well obsolete now.
      Howard.
      
      LikeLike
  - 19
    
    EcleX on August 19, 2020 at 10:37 am
    
    Thanks. Just for the record, we use a standard compression type called DEFLATE which, in turn, uses LZ77 and Huffman code in this bioinformatics sequencing assembly of a huge-giant genome.
    
    LikeLiked by 1 person
    - 20
      
      hoakley on August 19, 2020 at 11:00 am
      
      Thank you. Flate compression is supported in macOS, and is the type most commonly encountered in PDF documents.
      I would have thought in this instance it is far from being optimal. Personally, I would look at coding strategies rather than general-purpose compression. As genomes are constructed using limited vocabularies and contain many common sequences of base pairs, they should be amenable to coding strategies which could save a great deal more storage than regular compression.
      Howard.
      
      LikeLiked by 1 person
  - 21
    
    EcleX on August 19, 2020 at 4:00 pm
    
    Thanks. The problem in our case is that we are using ABySS
    https://www.bcgsc.ca/resources/software/abyss
    to assemble the genome, and it must recognize the compressor used. ABySS is a de novo, parallel, paired-end sequence assembler that is designed for short reads, as shown in the link above.
    
    LikeLiked by 2 people
    - 22
      
      hoakley on August 19, 2020 at 4:12 pm
      
      Ah – you’re in a closed system, then. Sorry, there’s nothing to be done.
      Howard.
      
      LikeLiked by 1 person
23

coxorange on August 17, 2020 at 10:36 am

Hello, it seems I was also affected by this issue. *During* a Time Machine backup my Mac restarted. For this reason I could not know in which condition my backup(s) got, and to be on the safe side I erased the entire backup drive and created a new backup (about 2.5TB). This took the rest of my day…
Do you know wheather after such a restart during backup, the old backups on TM would still have been OK? Thanks.

LikeLiked by 1 person
- 24
  
  hoakley on August 17, 2020 at 12:09 pm
  
  I’m sorry to hear that.
  Yes, TM should have kept that interrupted backup as a ‘backup in progress’. Following the restart, it can then choose whether to complete that backup, or delete it and make another instead. Those shouldn’t affect your existing backups at all.
  Howard.
  
  LikeLike
25

Sean Martell on August 22, 2020 at 11:15 pm

Still struggling with multiple kernel panics daily. I’m at my wits end as this is my main work computer and it’s a struggle to make it a few hours without a sudden restart. I’m running a 2019 16″ and the root cause of the panic events has been from windowserver to coreaudiod to everything in between.

I’m now struggling to decide whether to attempt Big Sur or just take it in for service and hope they discover it’s a failed logic board. Though with the sheer number of people upset with kernel panics lately, I’m not entirely convinced it is my hardware.

LikeLiked by 2 people
- 26
  
  hoakley on August 23, 2020 at 5:49 am
  
  I’m sorry to hear that. I presume that Diagnostics is still returning no hardware issues.
  If that’s the case, I’d perform a clean install of 10.15.6 with the SU applied, and no third-party extensions, and see if that still panics regularly. If it does, it definitely needs to go back to Apple. But it’s essential to rule out third-party extensions and the like first.
  Howard.
  
  LikeLiked by 1 person