Catalina 10.15.6 is prone to kernel panics from a memory leak

There have been recent reports that the macOS Catalina 10.15.6 update can result in kernel panics. This appears to be common among those who run some virtualisation products such as VMWare Fusion and VirtualBox, but I believe this may be a more general problem which could affect any Mac which is left running much of the time.

Many of those using VMWare Fusion 11.5.5 or Virtualbox 6.0.24 who leave their Windows VMs running for long periods have suffered kernel panics, which are discussed in depth here. The conclusion is that this is due to a memory leak in one of the kernel extensions which was updated in the 10.15.6 update, probably Sandbox.kext (com.apple.security.sandbox).

My own iMac Pro was updated to 10.15.6 on 15 July. Since new, it has run High Sierra briefly, then Mojave and now Catalina with hardly a problem (really!), and I can’t recall when it last had a kernel panic. I think over that period of more than 18 months, it has had fewer than five panics in total, and those have mainly been attributable to third-party software which was rapidly removed. I leave this iMac on for very long periods, only restarting every two or three weeks, when I remember, and on this occasion it may not have been restarted since the 10.15.6 update was installed.

Last night, at around 0030, I quit the last open app and put the display onto screensaver for the night. I don’t let the system sleep, and it normally runs an hourly small Time Machine backup, small hourly ChronoSync backups, plus daily Carbon Copy Cloner backups. Otherwise it is left to its own devices while I grab a few hours of sleep.

When performing various routine tasks such as checking UPS status, at 0321 this morning it reported its first serious memory problem:
03:21:09.447981+0100 kernel zone_map_exhaustion: Zone map size 12240662528, capacity 12884901888 [jetsam limit 95%]
03:21:09.448533+0100 kernel zone_map_exhaustion: Largest zone kalloc.48, size 6544393440
03:21:09.449437+0100 kernel kernel zone_map_exhaustion: Nothing to do for the largest zone [kalloc.48]. Waking up memorystatus thread.

Note this involves the zone kalloc.48.

Shortly afterwards, macOS started killing processes to try to recover memory:
03:21:09.480924+0100 kernel 1238233.930 memorystatus: killing_highwater_process pid 415 [lsd] (highwater 3) 107552KB - memorystatus_available_pages: 4176741
03:21:09.481007+0100 com.apple.MemoryMonitor plugin UserEventAgent MemoryMonitor kernel jetsam snapshot note received
03:21:09.481234+0100 osanalyticshelper extending prolongation transaction timer
03:21:09.481275+0100 osanalyticshelper Attempting to write jetsam report
03:21:09.481552+0100 OSAnalytics Process lsd [415] killed by jetsam reason highwater
03:21:09.481606+0100 OSAnalytics Tagging submission policy as alternate for logtype:298

Less than 3 seconds later, memory was nearly exhausted:
03:21:12.036130+0100 kernel zone_map_exhaustion: Zone map size 12240662528, capacity 12884901888 [jetsam limit 95%]
03:21:12.036683+0100 kernel zone_map_exhaustion: Largest zone kalloc.48, size 6544801440
03:21:12.037587+0100 kernel zone_map_exhaustion: Nothing to do for the largest zone [kalloc.48]. Waking up memorystatus thread.

The situation didn’t improve, so another process was sacrificed:
03:21:12.295742+0100 kernel zone_map_exhaustion: Zone map size 12240670720, capacity 12884901888 [jetsam limit 95%]
03:21:12.296302+0100 kernel zone_map_exhaustion: Largest zone kalloc.48, size 6544825920
03:21:12.297206+0100 kernel zone_map_exhaustion: Nothing to do for the largest zone [kalloc.48]. Waking up memorystatus thread.
03:21:12.314565+0100 kernel 1238236.779 memorystatus: killing_highwater_process pid 99257 [appstoreagent] (highwater 3) 52680KB - memorystatus_available_pages: 4221724
03:21:12.314618+0100 com.apple.MemoryMonitor plugin UserEventAgent MemoryMonitor kernel jetsam snapshot note received
03:21:12.314741+0100 osanalyticshelper extending prolongation transaction timer
03:21:12.314761+0100 osanalyticshelper Attempting to write jetsam report
03:21:12.315012+0100 OSAnalytics Process appstoreagent [99257] killed by jetsam reason highwater
03:21:12.315051+0100 OSAnalytics Tagging submission policy as alternate for logtype:298
03:21:12.315403+0100 OSAnalytics Unsupported ips/xattr type NSNull for 'name'
03:21:12.318870+0100 OSAnalytics Saved type '298(298)' report (15 of max 25) at /Library/Logs/DiagnosticReports/JetsamEvent-2020-08-06-032112.251.diag

This continued over the next 24 minutes or so, until processes like CoreServicesUIAgent, cloudphotod and LocationMenu had been killed one by one. The last entries in the log before the iMac shut down were:
03:45:36.683855+0100 kernel 1239701.228 memorystatus: killing_top_process pid 58870 [coreauthd] (zone-map-exhaustion 1) 724KB - memorystatus_available_pages: 4451852
03:45:36.684302+0100 kernel CoreAnalyticsFamily virtual bool CoreAnalyticsUserClient::initWithTask(task_t, void *, UInt32, OSDictionary *)::165:success
03:45:36.684867+0100 kernel zone_map_exhaustion: Zone map size 12884058112, capacity 12884901888 [jetsam limit 95%]
03:45:36.684937+0100 kernel zone_map_exhaustion: Largest zone kalloc.48, size 6586939680
03:45:36.685009+0100 kernel zone_map_exhaustion: Nothing to do for the largest zone [kalloc.48]. Waking up memorystatus thread.

At some time after that, the kernel panicked and the iMac shut down rather than restarted. I then started it up before 0600, and was eventually greeted by the panic report. That gave the reason for the panic as:
panic(cpu 8 caller 0xffffff80017729eb): "zalloc: zone map exhausted while allocating from zone kalloc.12288, likely due to memory leak in zone kalloc.48 (6586956000 total bytes, 137228148 elements allocated)"@/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-6153.141.1/osfmk/kern/zalloc.c:3627

Kernel extensions in the backtrace were com.apple.iokit.IOAcceleratorFamily2 (438.7.3) and com.apple.kext.AMDRadeonX5000 (3.1) with the following notable zones:
kalloc.48 6586956000 4896
kalloc.64 4241453056 5000896

given as current and free size. The backtraced kext suspected of leaking was given as com.apple.filesystems.apfs (1412.141.1) with its dependencies.

My iMac Pro had clearly suffered a kernel panic as a result of a memory leak, which ran zone kalloc.48 out of memory, which it couldn’t recover. This may be unassociated with the bug that’s affecting virtualisation software, but makes me suspect that they result from the same memory leak. In which case, any Mac running 10.15.6 could be so affected.

I hope that Apple’s engineers are onto this case, and that we’ll soon see a Supplemental Update to 10.15.6.