This weekend, I was excited to deploy my first Ryzen 3000-powered workstation in my home office. Unfortunately, a microcode bug—originally discovered in July but still floating around in large numbers in the wild—wrecked my good time. I eventually got my Ryzen 3700X system working, and it's definitely fast. But unfortunately, it's still bugged, and there's no easy way to fix it.
Not long after the product launch, AMD Ryzen 3000 customers started noticing problems with their shiny new CPUs. Windows users couldn't successfully launch Destiny 2 (due to a power-management bug, unrelated to the one sidelining my system), and Linux users in many cases couldn't even get their system to boot. Jason Evangelho covered the initial discovery and report of the bug at Forbes back in July, and an AMD representative provided him with a statement by email:
AMD has identified the root cause and implemented a BIOS fix for an issue impacting the ability to run certain Linux distributions and Destiny 2 on Ryzen 3000 processors. We have distributed an updated BIOS to our motherboard partners, and we expect consumers to have access to the new BIOS over the coming days.
This sounds happy and upbeat, but the reality isn't quite so simple. When there's a bug in the CPU microcode, you're at the mercy of your motherboard vendor to release a new system BIOS that will update it for you—you can't just go to some download link at AMD and apply a fix yourself.
AMD responded to the bug in July. As far as I can tell, AMD did so only by direct email response; there's no press release about it—and the company's response made it sound like everything would be fixed in a week or two.
I have the unfortunate duty of reporting to you, three months later, that it is not.
Whats an RDRAND?
The microcode bug in question is a faulty response to the RDRAND instruction. Modern x86_64 CPUs—beginning with Intel's Broadwell and AMD's Zen architectures—are supposed to have high-quality onboard random number generators (RNGs), which use thermal "noise" to very rapidly offer high-entropy pseudorandom numbers to anybody with kernel-level access who wants it. RDRAND is, in turn, the instruction that provides these random numbers.
All of this is supposed to be fairly failsafe. There's a CPUID function call that checks for the availability of RDRAND, and there's also a "carry bit" in the return value from a call to RDRAND that's supposed to let the calling application know if the CPU's RNG was unable to generate a sufficiently random number due to lack of entropy. Unfortunately, unpatched Ryzen 3000 says "yes" to the CPUID 01H call, sets the carry bit indicating it has successfully created the most artisanal, organic high-quality random number possible… and gives you a 0xFFFFFFFF for the "random" number, every single time.
In a sufficiently broad dataset, 20 consecutive 0xFFFFFFFF returns might be considered a valid "random" grouping. This is not a sufficiently broad dataset. Jim Salter You shouldn't rely on /dev/hwrng to tell you if you're NOT vulnerable to the AMD microcode bug, because /dev/hwrng might be getting its data from somewhere else. In my case, it's getting its data from RDRAND, and it's pretty obvious that data's no good. Jim Salter
Obvious RDRAND bug impacts
When theRDRAND bug in Ryzen 3000 first surfaced back in June, Linux users widely reported that their entire Ryzen 3000-powered systems wouldn't boot. The failure to boot was due to systemd's use of RDRAND—and it wasn't systemd's first clash with AMD and a buggy random-number generator, unfortunately.
A much earlier bug in older CPUs caused some AMD systems to stop generating properly "random" numbers after resuming from suspend. The new bug caused Ryzen 3000 users to never get any proper random numbers at all. Both problems caused lockups in Linux operating systems using systemd, so in May systemd committed a patch that falls back to using alternate RNG sources if systemd receives the characteristic 0xFFFFFFFF back from the RNG. (This kinda sucks, because 0xFFFFFFFF is technically a perfectly valid random number—the implication here is that, after a sufficient length of time, systemd will decide any system has a buggy RNG when it eventually receives the "bad" number, even if it has never seen that number before.)
Systemd's patch is ugly, but it certainly works well enough to allow systems to boot. Unfortunately, it doesn't fix the actual problem, which is that the CPU's random number generator is no more "random" than a two-headed penny. On my own system, I spent my entire weekend chasing phantom problems, first suspecting the system's brand-new RX 590 graphics card and (necessarily) updating distro and kernel versions before haring off from there.
—PU#n stuck for 22s errors, which would rapidly lock the whole system up. This call trace from /var/log/syslog didn't tell me the actual problem, but it was the first clue.Jim Salter
Eventually, after many false trails and much swearing, coffee, and less-respectable beverages, I actually read the call trace from my frequent CPU lockups—and "WireGuard" was right there, in every one of them. As it turns out, WireGuard relies on RDRAND (when available) to generate new session IDs. The session IDs need to be unique, and WireGuard wants them not to be simple consecutive integers, so it pulls a pseudorandom value from RDRAND, compares it against its existing session ID list to make sure there's no collision, then assigns it to the session.
Read that last part again carefully—it makes sure there's no collision first. If an existing session has the same ID as the new number, WireGuard asks RDRAND for another "random" number, checks it for uniqueness, and so on. Since RDRAND on my system—and any non-microcode-updated Ryzen 3000 system—always returned 0xFFFFFFFF no matter what, that means infinite loop. Infinite loops in kernel code are bad; they introduce you to the value of the hardware reset button in a hurry.
I want to be very clear here, this is not a WireGuard bug! WireGuard correctly checks to see if RDRAND is available, fetches a value if it is, and correctly checks to see if the carry bit is set. Then it indicates that, not only is there a value, it's a properly random one. Nevertheless, it's one that will lock up affected systems hard.
A modern system needs high-quality pseudo-random numbers for lots of tasks, and the security implications of "random" meaning "always return0xFFFFFFFF" are difficult to predict. One obvious candidate is Address Space Layout Randomization (ASLR). Both Windows and Linux use RDRAND as at least part of the randomness used to make sure code is never loaded in the same order twice, which mitigates against stack-smashing attacks.
Fixing the problem—or at least recognizing it
As AMD's representatives told reporters back in July, the real fix comes from applying BIOS updates to your motherboard and hoping that the BIOS update also includes the microcode patch for the CPU itself. When I checked my own BIOS using the dmidecode utility, I saw a date of August 12, 2019. But when I looked at Asus&Read More – Source
