What We Worked On
This is the story of a workstation that froze so completely that Philip had to reboot it with magic sysrq. The machine is a Pop!_OS box with 124 GiB of RAM, five NVIDIA cards reserved for compute, and a desktop that runs on the small RDNA2 iGPU inside his Ryzen CPU. Philip asked one question: why did it crash? The session answered that question as far as the evidence allowed, and then hardened the machine against the most likely cause.
The previous boot ended at 11:34:08. The journal stops in the middle of routine database inserts from his SeeSharpSwap service, with no panic, no oops, and no OOM kill anywhere in the boot. The EFI pstore was empty, so the kernel never got far enough to record what killed it. The SMART data on every disk was clean. The only warnings in the final minutes came from two places. The kernel reported xfs_end_io hogging the CPU at 11:32:56, which means heavy disk I/O. Plasmashell reported a stream of GPU shared-image errors at 11:33:05, which means the compositor’s GPU state was failing. Earlier that same morning, at 01:56, the amdgpu driver had logged Fence fallback timer expired on ring comp_1.1.0. A compute ring on the iGPU had already stopped responding once that day.
Magic sysrq still responded during the freeze. That means the kernel was alive but the display and the I/O path were wedged, which is the classic shape of an amdgpu hang on an APU.
The picture that emerges is consistent: the desktop runs on the one GPU in the machine with a history of hangs, and the freeze happened while the system was under heavy memory and I/O pressure.
The Mitigations
Philip chose to keep the desktop on the iGPU, so the fix was to make the iGPU harder to kill. Pop!_OS manages kernel parameters with kernelstub, and three parameters went in:
Each one addresses a specific failure mode.
| Parameter | What it does |
|---|---|
| amdgpu.gpu_recovery=1 | Forces the driver to reset the GPU on a hang instead of deadlocking the machine. A future hang should cost a few seconds of frozen screen, not a reboot. |
| amdgpu.dcdebugmask=0x12 | Disables Panel Self-Refresh (0x10) and memory stutter mode (0x2). Both are known hang triggers on DCN 3.1.5, the display block inside Raphael. |
| amdgpu.sg_display=0 | Forces every scanout buffer into the contiguous VRAM carve-out instead of scattered system RAM. Scatter-gather scanout competes with the rest of the machine for memory bandwidth, and it loses exactly when the machine is busiest. |
The kernel package also got its latest available rebuild, and the firmware and Mesa were already current, so there was nothing further to update.
All three parameters are read-only at runtime (0444 in /sys/module/amdgpu/parameters/), so nothing takes effect until the next reboot. Reloading the module would tear down the desktop session and risks the same hang it is meant to prevent, so a clean reboot is strictly better.
One more finding rounded out the session. The iGPU carve-out turned out to be the default 512 MiB with 492 MiB already in use, on a 4K monitor. With sg_display=0, every framebuffer must fit in that carve-out. The recommendation is to raise the UMA Frame Buffer Size to 4 GB in the UEFI, which costs nothing on a 124 GiB machine and can happen in the same reboot that activates the kernel parameters.
What Went Well
The diagnosis moved by elimination and landed somewhere defensible. Memory, panic, and disk failure were each ruled out with direct evidence before the iGPU took the blame. The mitigations map one-to-one onto the observed failure: recovery for the hang itself, display-core flags for the known DCN 3.1.5 triggers, and contiguous scanout for the memory-pressure correlation. The carve-out check happened before the reboot rather than after it; a 96% full carve-out plus sg_display=0 would have traded one failure mode for another.
What Didn’t Go Well
The freeze left no direct evidence. The journal simply stops, and even the sysrq keystrokes never reached the disk. The diagnosis is strong but circumstantial. None of the mitigations could be applied to the running kernel, so the machine stays exposed until Philip reboots. Verification is also pending: until the next heavy-load session passes without a GPU reset begin line in the journal, the fix is a hypothesis with good aim.
The journal ended mid-sentence, and the absence of evidence became the evidence. — session retrospective
Takeaways
-
1A silent journal is itself a finding
When logging stops mid-write with no panic and no OOM, the kernel died holding its locks. On an APU desktop, the amdgpu driver is the first suspect.
-
2Check where the monitor is actually plugged in
Five NVIDIA cards sat idle for display purposes while the whole desktop rode on the weakest GPU in the machine. The physical cable is part of the diagnosis.
-
3sg_display=0 and a small carve-out do not mix
Forcing scanout into VRAM only helps if the VRAM exists. Measure
mem_info_vram_totalagainstmem_info_vram_usedbefore flipping the switch. -
4Magic sysrq responding is diagnostic information
It separates “kernel is alive but wedged” from “kernel is gone,” and each points to a different class of cause.
The Watercolor
The sheet begins almost white, because that is what the evidence looked like: a journal that ends mid-sentence, a pstore with nothing in it, a page with no pigment where the crash should be. I would paint the absence first. A wash of the palest grey-violet covers the upper half, and where the freeze happened I would leave the paper bare, hard-edged, a rectangle of nothing that the eye keeps returning to.
The clues get small, deliberate strokes around that emptiness. A thin ochre line for the XFS I/O grinding at 11:32. A scatter of broken rose-colored marks for the compositor errors at 11:33. And low in the corner, from one in the morning, a single dry-brush drag of rust where a compute ring stopped answering; the stroke that, in hindsight, was the underdrawing for everything above it.
The lower half of the painting is where the confidence comes in. Three bands of cerulean, laid wet and even, one for each kernel parameter, anchored by a wide foundation stroke of warm gold for the 4 GB carve-out still to come. The composition points forward: every line in the lower half leans toward the right edge of the sheet, toward a reboot that has not happened yet. I would title it The Page Was Blank on Purpose and note, in pencil along the margin, that painting what is missing is still painting what happened.