Crashing the whole system due to kernel panic while running lit tests

Hey everyone,
I recently started to experience a Mac freeze and restart while running all tests using lit. Still not able to trace if there is a specific test that causes this issue, if that is system related(recently updated to Ventura) or something else.
I'm running llvm-project/llvm/utils/lit/lit.py -sv --param swift_site_config=build/Ninja-ReleaseAssert/swift-macosx-arm64/test-macosx-arm64/lit.site.cfg swift/test/ and it's happen every time. So I'm now trying to trace if that is due some specific test.

We had someone reporting the similar crash sometime ago Swift causes kernel panic + Incorrect diagnostic while compiling library dependent on PythonKit · Issue #58848 · apple/swift · GitHub
So maybe someone else is experiencing that as well?

Here are the logs from MacOS problem window after restart
panic(cpu 0 caller 0xfffffe002516f2f0): watchdog timeout: no checkins from watchdogd in 93 seconds (467 total checkins since monitoring last enabled)
Debugger message: panic
Memory ID: 0xff
OS release type: User
OS version: 22A380
Kernel version: Darwin Kernel Version 22.1.0: Sun Oct  9 20:14:30 PDT 2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T8103
Fileset Kernelcache UUID: C222B4132B9708E5E0E2E8B8C5896410
Kernel UUID: 0BFE6A5D-118B-3889-AE2B-D34A0117A062
Boot session UUID: 7AA2C8CC-E429-4C81-B9A6-8CBE82288DE5
iBoot version: iBoot-8419.41.10
secure boot?: YES
roots installed: 0
Paniclog version: 14
KernelCache slide: 0x000000001c3bc000
KernelCache base:  0xfffffe00233c0000
Kernel slide:      0x000000001d600000
Kernel text base:  0xfffffe0024604000
Kernel text exec slide: 0x000000001d6e8000
Kernel text exec base:  0xfffffe00246ec000
mach_absolute_time: 0x1aa4a52e4b
Epoch Time:        sec       usec
  Boot    : 0x6361a6da 0x000d89dc
  Sleep   : 0x00000000 0x00000000
  Wake    : 0x00000000 0x00000000
  Calendar: 0x6361b975 0x0009dccc

Zone info:
  Zone map: 0xfffffe1006d50000 - 0xfffffe3006d50000
  . VM    : 0xfffffe1006d50000 - 0xfffffe14d3a1c000
  . RO    : 0xfffffe14d3a1c000 - 0xfffffe166d3b4000
  . GEN0  : 0xfffffe166d3b4000 - 0xfffffe1b3a080000
  . GEN1  : 0xfffffe1b3a080000 - 0xfffffe2006d4c000
  . GEN2  : 0xfffffe2006d4c000 - 0xfffffe24d3a18000
  . GEN3  : 0xfffffe24d3a18000 - 0xfffffe29a06e4000
  . DATA  : 0xfffffe29a06e4000 - 0xfffffe3006d50000
  Metadata: 0xfffffe8fef1c0000 - 0xfffffe8ff71c0000
  Bitmaps : 0xfffffe8ff71c0000 - 0xfffffe8ff9dec000

CORE 0 recently retired instr at 0xfffffe00248a57a0
CORE 1 recently retired instr at 0xfffffe00248a6ea0
CORE 2 recently retired instr at 0xfffffe00248a6ea0
CORE 3 recently retired instr at 0xfffffe00248a6ea0
CORE 4 recently retired instr at 0xfffffe00248a6ea0
CORE 5 recently retired instr at 0xfffffe00248a6ea0
CORE 6 recently retired instr at 0xfffffe00248a6ea0
CORE 7 recently retired instr at 0xfffffe00248a6ea0
TPIDRx_ELy = {1: 0xfffffe2006e7d030  0: 0x0000000000000000  0ro: 0x0000000000000000 }
CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 0 is the one that panicked. Check the full backtrace for details.
CORE 1: PC=0xfffffe002478755c, LR=0xfffffe002478755c, FP=0xfffffe82799a7f00
CORE 2: PC=0xfffffe0024780268, LR=0xfffffe0024780268, FP=0xfffffe8278963e20
CORE 3: PC=0xfffffe002478755c, LR=0xfffffe002478755c, FP=0xfffffe8279b3ff00
CORE 4: PC=0xfffffe0024787478, LR=0xfffffe002478777c, FP=0xfffffe82793eff20
CORE 5: PC=0xfffffe0024780278, LR=0xfffffe0024765b00, FP=0xfffffe8279e63a70
CORE 6: PC=0xfffffe00248947a0, LR=0xfffffe00248a2440, FP=0xfffffe8279fcbe90
CORE 7: PC=0xfffffe00248947bc, LR=0xfffffe00248a2440, FP=0xfffffe8279e03e90
Compressor Info: 27% of compressed pages limit (OK) and 8% of segments limit (OK) with 5 swapfiles and OK swap space
Total cpu_usage: 55600628
Thread task pri cpu_usage
0xfffffe2006e7c838 kernel_task 91 86222
0xfffffe200912e818 Code - Insiders  46 14328
0xfffffe2006e71030 kernel_task 0 1054241
0xfffffe2006e72020 kernel_task 0 969351
0xfffffe2007101030 kernel_task 0 2373568

Panicked task 0xfffffe24d3a3d678: 0 pages, 465 threads: pid 0: kernel_task
Panicked thread: 0xfffffe2006e7d030, backtrace: 0xfffffe8260a6b850, tid: 102
		  lr: 0xfffffe002474a7e8  fp: 0xfffffe8260a6b8c0
		  lr: 0xfffffe002474a590  fp: 0xfffffe8260a6b940
		  lr: 0xfffffe00248ad92c  fp: 0xfffffe8260a6b960
		  lr: 0xfffffe002489f394  fp: 0xfffffe8260a6b9d0
		  lr: 0xfffffe002489cc98  fp: 0xfffffe8260a6ba90
		  lr: 0xfffffe00246f37f8  fp: 0xfffffe8260a6baa0
		  lr: 0xfffffe0024749f1c  fp: 0xfffffe8260a6be50
		  lr: 0xfffffe0024f8ccf8  fp: 0xfffffe8260a6be70
		  lr: 0xfffffe002516f2f0  fp: 0xfffffe8260a6bec0
		  lr: 0xfffffe002516e4c4  fp: 0xfffffe8260a6bf00
		  lr: 0xfffffe002516baf4  fp: 0xfffffe8260a6bf20
		  lr: 0xfffffe00259b6f20  fp: 0xfffffe8260a6bfb0
		  lr: 0xfffffe0024ebe954  fp: 0xfffffe8260a6bfc0
		  lr: 0xfffffe002489ffac  fp: 0xfffffe8260a6bfe0
		  lr: 0xfffffe00246f386c  fp: 0xfffffe8260a6bff0
		  lr: 0xfffffe0024787770  fp: 0xfffffe8279d5bf20
		  lr: 0xfffffe00246fce98  fp: 0x0000000000000000
      Kernel Extensions in backtrace:
         com.apple.driver.AppleInterruptController(1.0d1)[31268190-20B6-3068-886C-0806D7F2A50D]@0xfffffe00259b3fe0->0xfffffe00259b7cf7
            dependency: com.apple.driver.AppleARMPlatform(1.0.2)[11A9713E-6739-3A4C-8571-2D8EAA062278]@0xfffffe002511bff0->0xfffffe002516a55f
         com.apple.driver.AppleARMWatchdogTimer(1.0)[17B239C9-4DBF-3B50-AE07-7FF69198D6D9]@0xfffffe002516a560->0xfffffe002516f317
            dependency: com.apple.driver.AppleARMPlatform(1.0.2)[11A9713E-6739-3A4C-8571-2D8EAA062278]@0xfffffe002511bff0->0xfffffe002516a55f

last started kext at 645021164: com.apple.filesystems.autofs	3.0 (addr 0xfffffe0023b880b0, size 5904)
loaded kexts:
com.apple.filesystems.autofs	3.0
com.apple.driver.AppleTopCaseHIDEventDriver	6000.22
com.apple.driver.CoreKDL	1
com.apple.driver.AppleBiometricServices	1
com.apple.driver.AppleAOPVoiceTrigger	200.4
com.apple.driver.DiskImages.ReadWriteDiskImage	493.0.0
com.apple.driver.DiskImages.UDIFDiskImage	493.0.0
com.apple.driver.DiskImages.RAMBackingStore	493.0.0
com.apple.driver.DiskImages.FileBackingStore	493.0.0
com.apple.driver.BCMWLANFirmware4378.Hashstore	1
com.apple.driver.SEPHibernation	1
com.apple.driver.AppleUSBDeviceNCM	5.0.0
com.apple.driver.AppleThunderboltIP	4.0.3
com.apple.driver.AppleFileSystemDriver	3.0.1
com.apple.nke.l2tp	1.9
com.apple.filesystems.tmpfs	1
com.apple.iokit.AppleBCM5701Ethernet	11.0.0
com.apple.driver.AppleSmartIO2	1
com.apple.driver.ApplePMP	1
com.apple.driver.AppleSmartBatteryManager	161.0.0
com.apple.filesystems.nfs	1
com.apple.filesystems.lifs	1
com.apple.filesystems.apfs	2142.41.2
com.apple.IOTextEncryptionFamily	1.0.0
com.apple.filesystems.hfs.kext	627.40.1
com.apple.security.BootPolicy	1
com.apple.BootCache	40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib	1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless	1.0.0d1
com.apple.driver.ApplePMPFirmware	1
com.apple.AppleEmbeddedSimpleSPINORFlasher	1
com.apple.driver.AppleCS42L83Audio	600.99
com.apple.driver.AppleTAS5770LAmp	600.99
com.apple.driver.AppleSPMIPMU	1.0.1
com.apple.driver.AppleT8020SOCTuner	1
com.apple.driver.AppleT8103CLPCv3	1
com.apple.driver.AudioDMAController-T8103	200.21
com.apple.driver.AppleSerialShim	1
com.apple.driver.AppleSPIMC	1
com.apple.driver.AppleM68Buttons	1.0.0d1
com.apple.AGXG13G	227.2.40
com.apple.driver.AppleAVD	616
com.apple.driver.AppleAVE2	630.29.1
com.apple.driver.AppleJPEGDriver	5.2.6
com.apple.driver.AppleMobileDispH13G-DCP	140.0
com.apple.driver.usb.AppleSynopsysUSB40XHCI	1
com.apple.driver.AppleEventLogHandler	1
com.apple.driver.AppleS5L8960XNCO	1
com.apple.driver.AppleT8103PMGR	1
com.apple.driver.AppleS8000AES	1
com.apple.driver.AppleS5L8920XPWM	1.0.0d1
com.apple.driver.AppleS8000DWI	1.0.0d1
com.apple.driver.AppleInterruptController	1.0.0d1
com.apple.driver.AppleBluetoothModule	1
com.apple.driver.AppleSamsungSerial	1.0.0d1
com.apple.driver.AppleBCMWLANBusInterfacePCIe	1
com.apple.driver.AppleMCDP29XXUpdateSupport	1
com.apple.driver.AppleT8020DART	1
com.apple.driver.AppleS5L8940XI2C	1.0.0d2
com.apple.driver.AppleT8101	1
com.apple.iokit.IOUserEthernet	1.0.1
com.apple.driver.usb.AppleUSBUserHCI	1
com.apple.iokit.IOKitRegistryCompatibility	1
com.apple.iokit.EndpointSecurity	1
com.apple.driver.AppleDiskImages2	198.40.3
com.apple.AppleSystemPolicy	2.0.0
com.apple.nke.applicationfirewall	403
com.apple.kec.InvalidateHmac	1
com.apple.kec.Compression	1
com.apple.kec.AppleEncryptedArchive	1
com.apple.driver.driverkit.serial	6.0.0
com.apple.driver.AppleHIDKeyboard	231
com.apple.driver.AppleMultitouchDriver	6400.44
com.apple.driver.AppleInputDeviceSupport	6400.40
com.apple.driver.AppleHSBluetoothDriver	6000.22
com.apple.driver.IOBluetoothHIDDriver	9.0.0
com.apple.iokit.IOAVBFamily	1100.24
com.apple.plugin.IOgPTPPlugin	1110.13
com.apple.driver.AppleMesaSEPDriver	100.99
com.apple.iokit.IOBiometricFamily	1
com.apple.driver.usb.IOUSBHostHIDDevice	1.2
com.apple.driver.usb.cdc	5.0.0
com.apple.driver.AppleUSBAudio	500.2
com.apple.iokit.IOAudioFamily	400.3
com.apple.vecLib.kext	1.2.0
com.apple.driver.AppleAOPAudio	200.45
com.apple.driver.DiskImages.KernelBacked	493.0.0
com.apple.driver.AppleSEPHDCPManager	1.0.1
com.apple.driver.AppleTrustedAccessory	1
com.apple.iokit.AppleSEPGenericTransfer	1
com.apple.driver.AppleXsanScheme	3
com.apple.driver.usb.networking	5.0.0
com.apple.nke.ppp	1.9
com.apple.driver.AppleThunderboltPCIDownAdapter	4.1.1
com.apple.driver.AppleThunderboltUSBDownAdapter	1.0.4
com.apple.driver.AppleThunderboltDPInAdapter	8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily	8.5.1
com.apple.driver.usb.AppleEmbeddedUSBXHCIPCI	1
com.apple.driver.usb.AppleUSBXHCIPCI	1.2
com.apple.driver.AppleDCPDPTXProxy	1.0.0
com.apple.driver.DCPDPFamilyProxy	1
com.apple.driver.AppleConvergedIPCOLYBTControl	1
com.apple.driver.AppleConvergedPCI	1
com.apple.driver.AppleBluetoothDebug	1
com.apple.driver.AppleBTM	1.0.1
com.apple.driver.AppleBSDKextStarter	3
com.apple.kext.triggers	1.0
com.apple.driver.IOHIDPowerSource	1
com.apple.filesystems.hfs.encodings.kext	1
com.apple.driver.AppleSyntheticGameController	10.2.9
com.apple.driver.AppleDiagnosticDataAccessReadOnly	1.0.0
com.apple.driver.AppleCSEmbeddedAudio	600.99
com.apple.driver.AppleHPM	3.4.4
com.apple.AGXFirmwareKextG13GRTBuddy	227.2.40
com.apple.AGXFirmwareKextRTBuddy64	227.2.40
com.apple.driver.AppleEmbeddedAudio	600.99
com.apple.iokit.AppleARMIISAudio	200.8
com.apple.iokit.IONVMeFamily	2.1.0
com.apple.driver.AppleNANDConfigAccess	1.0.0
com.apple.driver.AppleSPU	1
com.apple.driver.AppleStockholmControl	1.0.0
com.apple.driver.AppleDialogPMU	1.0.1
com.apple.driver.ApplePassthroughPPM	3.0
com.apple.driver.AppleMCA2-T8103	701.14
com.apple.driver.AppleEmbeddedAudioLibs	200.5
com.apple.iokit.IOGPUFamily	65.0.22
com.apple.driver.AppleH11ANEInterface	6.101.1
com.apple.driver.DCPAVFamilyProxy	1
com.apple.iokit.IOMobileGraphicsFamily-DCP	343.0.0
com.apple.driver.AppleDCP	1
com.apple.driver.AppleFirmwareKit	1
com.apple.driver.AppleFirmwareUpdateKext	1
com.apple.iokit.IOMobileGraphicsFamily	343.0.0
com.apple.driver.AppleSPMI	1.0.1
com.apple.driver.AppleUSBXDCIARM	1.0
com.apple.driver.AppleUSBXDCI	1.0
com.apple.iokit.IOUSBDeviceFamily	2.0.0
com.apple.driver.usb.AppleSynopsysUSBXHCI	1
com.apple.driver.usb.AppleUSBXHCI	1.2
com.apple.driver.AppleEmbeddedUSBHost	1
com.apple.driver.usb.AppleUSBHub	1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice	1.2
com.apple.driver.AppleT8103TypeCPhy	1
com.apple.driver.AppleSART	1
com.apple.driver.ApplePMGR	1
com.apple.driver.AppleARMWatchdogTimer	1
com.apple.driver.AppleDisplayCrossbar	1.0.0
com.apple.driver.AppleTypeCPhy	1
com.apple.driver.AppleThunderboltNHI	7.2.81
com.apple.driver.AppleT8103PCIeC	1
com.apple.iokit.IOThunderboltFamily	9.3.3
com.apple.driver.ApplePIODMA	1
com.apple.driver.AppleT8103PCIe	1
com.apple.driver.AppleMultiFunctionManager	1
com.apple.driver.AppleBluetoothDebugService	1
com.apple.driver.AppleBCMWLANCore	1.0.0
com.apple.iokit.IO80211Family	1200.13.0
com.apple.driver.IOImageLoader	1.0.0
com.apple.driver.AppleOLYHAL	1
com.apple.driver.corecapture	1.0.4
com.apple.driver.AppleEmbeddedPCIE	1
com.apple.iokit.IODisplayPortFamily	1.0.0
com.apple.driver.AppleGPIOICController	1.0.2
com.apple.driver.AppleFireStormErrorHandler	1
com.apple.driver.AppleMobileApNonce	1
com.apple.driver.usb.AppleUSBHostPacketFilter	1.0
com.apple.iokit.IOTimeSyncFamily	1110.13
com.apple.driver.DiskImages	493.0.0
com.apple.iokit.IOGraphicsFamily	597
com.apple.iokit.IOBluetoothSerialManager	9.0.0
com.apple.iokit.IOBluetoothHostControllerUSBTransport	9.0.0
com.apple.iokit.IOBluetoothHostControllerUARTTransport	9.0.0
com.apple.iokit.IOBluetoothHostControllerTransport	9.0.0
com.apple.driver.IOBluetoothHostControllerPCIeTransport	9.0.0
com.apple.iokit.IOBluetoothFamily	9.0.0
com.apple.iokit.CSRBluetoothHostControllerUSBTransport	9.0.0
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport	9.0.0
com.apple.driver.AppleSSE	1.0
com.apple.driver.AppleSEPKeyStore	2
com.apple.driver.AppleUSBTDM	547
com.apple.iokit.IOUSBMassStorageDriver	232
com.apple.iokit.IOPCIFamily	2.9
com.apple.iokit.IOSCSIBlockCommandsDevice	476
com.apple.iokit.IOSCSIArchitectureModelFamily	476
com.apple.driver.AppleRSMChannel	1
com.apple.iokit.IORSMFamily	1
com.apple.driver.AppleIPAppender	1.0
com.apple.driver.AppleFDEKeyStore	28.30
com.apple.driver.AppleEffaceableStorage	1.0
com.apple.driver.AppleCredentialManager	1.0
com.apple.driver.KernelRelayHost	1
com.apple.iokit.IOUSBHostFamily	1.2
com.apple.driver.AppleUSBHostMergeProperties	1.2
com.apple.driver.usb.AppleUSBCommon	1.0
com.apple.driver.AppleSMC	3.1.9
com.apple.driver.RTBuddy	1.0.0
com.apple.driver.AppleEmbeddedTempSensor	1.0.0
com.apple.driver.AppleARMPMU	1.0
com.apple.iokit.IOAccessoryManager	1.0.0
com.apple.driver.AppleOnboardSerial	1.0
com.apple.iokit.IOSkywalkFamily	1.0
com.apple.driver.mDNSOffloadUserClient	1.0.1b8
com.apple.iokit.IONetworkingFamily	3.4
com.apple.iokit.IOSerialFamily	11
com.apple.driver.AppleSEPManager	1.0.1
com.apple.driver.AppleA7IOP	1.0.2
com.apple.driver.IOSlaveProcessor	1
com.apple.driver.AppleBiometricSensor	2
com.apple.iokit.IOHIDFamily	2.0.0
com.apple.AUC	1.0
com.apple.iokit.IOAVFamily	1.0.0
com.apple.iokit.IOHDCPFamily	1.0.0
com.apple.iokit.IOCECFamily	1
com.apple.iokit.IOAudio2Family	1.0
com.apple.driver.AppleIISController	200.2
com.apple.driver.AppleAudioClockLibs	200.5
com.apple.driver.AppleM2ScalerCSCDriver	265.0.0
com.apple.iokit.IOSurface	334.0.1
com.apple.driver.IODARTFamily	1
com.apple.driver.FairPlayIOKit	68.19.0
com.apple.driver.AppleARMPlatform	1.0.2
com.apple.iokit.IOSlowAdaptiveClockingFamily	1.0.0
com.apple.iokit.IOReportFamily	47
com.apple.security.quarantine	4
com.apple.security.sandbox	300.0
com.apple.iokit.IOStorageFamily	2.1
com.apple.kext.AppleMatch	1.0.0d1
com.apple.driver.AppleMobileFileIntegrity	1.0.5
com.apple.iokit.CoreAnalyticsFamily	1
com.apple.security.AppleImage4	5.0.0
com.apple.kext.CoreTrust	1
com.apple.iokit.IOCryptoAcceleratorFamily	1.0.1
com.apple.kec.pthread	1
com.apple.kec.Libm	1
com.apple.kec.corecrypto	12.0



** Stackshot Succeeded ** Bytes Traced 313084 (Uncompressed 842752) **
2 Likes

This looks like the tests are simply overwhelming the system, preventing watchdogd from getting enough CPU time to reply to messages. How much RAM do you have? Maybe you’re hitting a VM pageout death spiral.

1 Like

Thanks for the response @ksluder
I currently with 8GB, which was fine up until recently (my guess is since installed VenturaOS, but don't remember if I had a successful run since). I can try monitor CPU usage and memory while running to see what may be happening there.

Found that the tests causing the crash are swift/test/Sanitizers/tsan/ when running all tests in this folder(llvm-project/llvm/utils/lit/lit.py -sv --param swift_site_config=build/Ninja-ReleaseAssert/swift-macosx-arm64/test-macosx-arm64/lit.site.cfg swift/test/Sanitizers/tsan) the crash happens. Interestingly running each test individually seems to be fine and also running then all in a single worker (-j 1) is ok as well. Some of then take some time to execute which makes me think that they are actually blocking so when running on all 8 cores that may be the cause of the issue (all workers blocked at once)?
It still weird that I don't see anyone else reporting this issue recently so that also points to something in my system maybe? but in that case this go outside of scope of my limited knowledge...

2 Likes

Not much progress here, just end using lit.py --filter-out to not run any sanitizers test for now ...

It still weird that I don't see anyone else reporting this issue recently so that also points to something in my system maybe?

Also encounter this issue on M1 MacBook Pro VenturaOS(13.0.1) 16GB ram.
--filter-out the sanitizers test will do the trick!
Thanks for your help in investigating this issue!

1 Like

Since it appeared that was only me that was having this problem(maybe was something on my system) was unsure worth it an issue for that. But since you have encounter that as well I just reported as [Compiler] Crash the whole machine when running sanitizers/tsan tests · Issue #62128 · apple/swift · GitHub. Thanks @hank121314!

Also seeing this behavior, M1 Pro 16gb on Ventura 13.1, running the tests via lit.py or directly with cmake