An uncorrectable double-bit memory error — drain the node, RMA on recurrence.
Xid 48 is a double-bit ECC error: two bit flips in the same word that ECC can detect but cannot correct. It is a genuine hardware fault. The affected memory page is retired (or row remapped on Ampere and later), and any work that touched it is suspect.
NVRM: Xid (PCI:0000:65:00): 48, pid='N/A', Double Bit ECCDrain the node and reset the GPU so the page is retired / row remapped. If double-bit errors recur on the same GPU, or remapping resources are exhausted, schedule an RMA.
The datacenter decision path for hardware-class Xids: cordon → drain → reset → RMA on recurrence. Cordon to stop new work landing, drain running jobs, reset the GPU to clear or remap the fault, and RMA if it comes back.
This is real signal — exactly the kind that must survive the noise. The risk isn't false alarms; it's a single Xid 48 getting lost among hundreds of benign Xids.
That judgment — absorb the noise, surface the few that are real — is what Plexus does by default on your existing DCGM / Prometheus / Thanos. See why a GPU fleet throws 50 alerts a week and only two matter.
Severity here is an operations judgment for datacenter fleets and depends on driver and GPU generation — always cross-check NVIDIA’s official Xid table.