Recovering from Mac Kernel panics with a little help from Osquery

5 min readMay 16, 2022

Recently one of our build machines ended up in a boot loop which always resulted in a kernel panic. Whilst all the online help nearly always results in a re-build of the system. How do you debug and recover from a kernel panic during boot or at runtime?

Now there will be some nuances depending of the Apple hardware you’re running. In these examples we’re running an Intel Mac. For the new M1 Macs or similar systems Google the cosponsoring commands if they don’t work. We have provided links in most cases.

Recovering from a panic a boot time

If you end up in a continual loop where the system crashes upon each restart before getting to the login screen. You can get a more detailed or verbose debug message by holding down the following keys during the system start:

⌘ + v
(Command + v)

When the system restarts instead of the normal Apple boot screen with the logo and progress bar you’ll see something like this:

Eventually, the system will cause a kernel panic and cause a kernel crash dump to appear on the screen. Along with a trace of where the problem arose. This will provide you with the necessary technical information to troubleshoot the issue. This can be very technically verbose. However, some clues may lie in the actions that were taking place before the crash occurred.

Mac recovery mode

If it looks like the issue maybe with software or a recent configuration change that you wish to undo. You can boot your mac into its recovery mode and access the hard drive directly.

⌘ + r
(Command + r)

You’ll see the Apple logo, like you would during the normal boot process. However, it’ll take a little longer and you’ll be greeted with the Mac recovery screen.

To access your hard drive. You’ll first need to mount it using the disk utility.

Once disk utility has loaded. Click on the drive from the left hand side menu and click mount. You may need to enter your Filevault (disk encryption) password.

Exit the disk utility app and then under utilities menu from top navigation bar you can bring up the terminal. All your files will be available from the:

cd /Volumes/OSX/ (Or your drive name)

If you’re looking for logs or Kernel dumps please see the section below on debugging panics post boot. As there is a chance during the boot process that the panic or logs just simply aren’t able to be written to disk.

Diagnostic mode

If it looks like it might be a hardware fault. You can restart your mac again. This time using the following keyboard combination:

⌘ + d
(Command + d)

Devices with built in networking will be initially greeted with the networking option. Which, the first time I saw it thought it might be the network boot option. Instead this is so you can send any diagnostic information back to Apple (if you so choose). The test will take a few minutes to complete.

Apples KB is available here. Once the test has finished, anything other than an ADP000 (indicating no faults) — you can look up the error codes here.

NVRAM / PRAM reset

NVRAM (nonvolatile random-access memory) is a small amount of memory that your Mac uses to store certain settings and access them quickly. PRAM (Parameter RAM) stores similar information.

Settings that can be stored in NVRAM could include settings like; sound volume, display resolution, startup disk selection, time zone and recent kernel panic information. The settings stored in NVRAM depend on your Mac and the devices you’re using with your Mac.

These stored settings could be altering the way your Mac boots and or behaves. Resetting it can resolve a panic. As before, during the system start hold down these keys.

⌘ + ⌥ + p + r 
(Command + Option + p + r)

The screen will flash black after the Apple logo and restart once more.

SMC Reset

The System Management Controller (SMC). Controls how your Mac manages power, thermals, battery charging, video mode switching, sleep and wake, hibernation, and LED indicators. Any issues affecting the above maybe fixed with a reset of the SMC. Otherwise, leave it alone. More information can be found here.

ctrl + ⌥ + shift
(Control + Option + Shift)

Debugging a panic post boot time

If you can still boot into OSX. The Console application will have direct access to all your system logs. As well as the ability to parse some of the .asl and .diag file formats.

Other locations and commands for potential crash information:

cd ~/Library/Logs/DiagnosticReports
cd /Library/Logs/DiagnosticReports
cat /var/log/system.log
sudo dmesg

More information on Apple kernel panics

https://support.apple.com/en-us/TS3742

The kernel_panic table in Osquery

Osquery has a helpful table for remotely checking if a system has had any recent kernel panics. Useful if uses have been reporting bad system behavior.

SELECT * FROM kernel_panic;

Under the hood. This simply checks the /Library/Logs/DiagnosticReports folder for any files with the .panic extension. It will also partly parse the dump and provide you with the follow information:

.mode line
SELECT * FROM kernel_panics ORDER BY time DESC LIMIT 1;
            path = /Library/Logs/DiagnosticReports/Kernel_2015-06-10-104120_Example.panic
            time = Wed Jun 10 10:41:19 2015
       registers = CR0:0x0000000080010033 CR2:0x0000004300000007 CR3:0x0000000168ac8126 CR4:0x00000000001627e0 RAX:0x0000004300000006 RBX:0x0000000000000066 RCX:0xffffff7f9c654c06 RDX:0xffffff8039263480 RSP:0xffffff822b67b990 RBP:0xffffff822b67b9e0 RSI:0x000000008020690c RDI:0xffffff805aed4a08 R8:0x0000000000000000 R9:0x00000000000001b0 R10:0x00000000000001b8 R11:0x0000000000000202 R12:0xffffff8056d66008 R13:0x0000000000000002 R14:0xffffff8056d66400 R15:0x0000000000000066 RFL:0x0000000000010282 RIP:0xffffff7f9c654c9b CS:0x0000000000000008 SS:0x0000000000000010
 frame_backtrace = 0xffffff822b67b640 : 0xffffff801ad2bda1
module_backtrace = foo.tun(1.0)[424F8631-7A29-4E02-B0C9-2442FB894E25]@0xffffff7f9c654000->0xffffff7f9c659fff
    dependencies = 
            name = ifconfig
      os_version = 14D136
  kernel_version = Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64
    system_model = MacBookPro11,1 (Mac-189A3D4F975D5FFC)
          uptime = 162729462029316
     last_loaded = com.apple.driver.CoreStorageFsck 471.20.7 (addr 0xffffff7f9cebc000, size 102400)
   last_unloaded = com.apple.driver.AppleUSBCDC 4.3.3b1 (addr 0xffffff7f9c759000, size 16384)

There are other files in this directory which can be listed like so:

SELECT * FROM file WHERE path LIKE '/Library/Logs/DiagnosticReports/%';

This directory contains other diagnostic and crash reports from other applications.

The crash dump itself, if required can also be retrieved directly using the Osquery Carves table. This does require a remote service like Zercurity that can store and process Osquery carves.

SELECT * FROM carves WHERE path IN (SELECT path FROM kernel_panics ORDER BY time DESC LIMIT 1) AND carve = 1;

Once the query has run you’ll be able to download the file archive containing the crash dump. You can find out more about file retrieval in Osquery here:

File retrieval with Osquery using carves on Zercurity

With remote server support in Osquery you can remotely grab files from systems using the carves table within Osquery as…

zercurity.medium.com

Its all over!

We hope you found this helpful. Please feel free to get in touch if you have any questions.