Bit-Flips Are Crashing Your Devices and It's Not a Software Bug
You’ve got a fleet of 50,000 sensors in the field. Every week, about 50 of them reboot. No crash log. No stack trace. No pattern in firmware version, batch number, or deployment site. Your engineering team has spent 6 weeks on this. They’ve reviewed every line of code, stress-tested the power supply, swapped out capacitors, and re-examined every interrupt handler. QA closed the ticket as “no repro” because they can’t trigger it on the bench.
The code is fine. The hardware isn’t broken. A particle smaller than an atom is flying through your SRAM and flipping a bit.
These are called single-event upsets (SEUs), commonly known as bit-flips. They’re not exotic. They’re not a “space problem.” They’re a physics problem that affects every embedded system ever built, at some calculable probability. Most embedded teams have zero defenses against them.
What Actually Happens When a Bit Flips
Every SRAM cell in your microcontroller is a tiny latch, usually built from 6 transistors wired as two cross-coupled inverters. One side holds a 1, the other holds a 0. The feedback loop keeps it stable, until something deposits enough charge on one side to overwhelm the feedback.
That “something” is usually a charged particle: a secondary neutron from a cosmic ray hitting the atmosphere, an alpha particle emitted by trace uranium or thorium in the chip’s packaging material, or a thermal neutron wandering through your board. The particle rips through the silicon, leaving a trail of electron-hole pairs. If enough charge collects at the right node, the latch flips.
SRAM Cell (6-Transistor Latch)
┌──────────────────────────────┐
│ │
│ Stable State: stores "1" │
│ ┌───┐ ┌───┐ │
│ │ 1 │◄───────►│ 0 │ │
│ └───┘ cross- └───┘ │
│ coupled │
│ inverters │
└──────────────────────────────┘
│
▼ Ionizing particle strike
┌──────────────────────────────┐
│ Corrupted: now stores "0" │
│ ┌───┐ ┌───┐ │
│ │ 0 │◄───────►│ 1 │ │
│ └───┘ └───┘ │
└──────────────────────────────┘
No physical damage. No error flag.
The system just reads wrong data.This is a soft error: transient, non-destructive, and completely invisible. The cell isn’t damaged; it just holds the wrong value now. Compare this to a hard error, where the silicon is physically broken. Hard errors are easy to find. Soft errors are ghosts.
Here’s the trend that makes this worse every year: as process nodes shrink, each transistor stores less charge. A 28nm SRAM cell holds less charge than a 90nm cell, which means less energy is needed to flip it. The soft error rate per bit has been climbing with each generation. Smaller, cheaper, more power-efficient MCUs are also, bit for bit, more vulnerable.
Why Your Debugger Will Never Find This
Standard debugging assumes deterministic failure. You set a breakpoint, reproduce the bug, inspect state. But a bit-flip is non-deterministic, transient, and leaves no trace after a reset. The corrupted bit gets overwritten or the whole RAM is re-initialized on boot.
Printf won’t catch it. JTAG won’t catch it. Your logic analyzer won’t catch it. The evidence is destroyed the moment the system resets.
The failure mode can be anything:
- A flipped bit in a pointer causes a hard fault
- A flipped bit in a loop counter creates an infinite loop
- A flipped bit in a peripheral config register makes your UART run at the wrong baud rate
- A flipped bit in a state machine variable skips a critical safety check
This is exactly why teams misattribute the problem. They chase firmware bugs for months, blame EMI, suspect the power supply, or (my personal favorite) chalk it up to “cosmic rays” as a joke without realizing they’re right.
Environments That Make It Worse
The background neutron flux at sea level is roughly 20 neutrons per cm² per hour. That sets your baseline soft error rate (SER) for embedded systems sitting on a desk or in a warehouse. Move that device to a different environment and the numbers shift by orders of magnitude.
Environment | Relative Neutron Flux | Risk Level
───────────────────┼───────────────────────┼───────────
Sea level | 1x (baseline) | Low/device
Denver (1,600m) | ~3-4x | Moderate
Commercial flight | ~300x | High
Low Earth Orbit | ~1,000x+ | Extreme
───────────────────┼───────────────────────┼───────────
Note: Fleet scale multiplies any per-device rate.Altitude is the biggest single factor. Cosmic ray secondary neutron flux increases roughly 300x from sea level to 35,000 feet. If you’re building anything for aviation, high-altitude base stations, or mountain-deployed sensors, your per-device SER is dramatically higher than what you tested for in your lab in San Jose.
Industrial and medical environments introduce a different vector: proximity to radiation sources. Even concrete walls and ceramic components can contain trace radioactive isotopes that emit alpha particles.
Automotive systems face wide temperature swings. High temperatures reduce the noise margin in SRAM cells, lowering the energy threshold for a flip. A device that’s fine at 25°C might be significantly more vulnerable at 85°C.
Then there’s pure scale. FIT, or failures in 10⁹ device-hours, is the standard unit for soft error rates. Say your MCU has 256 Kbit of SRAM with a soft error rate of 1,000 FIT per Mbit. That works out to about 256 FIT for the SRAM block. In practical terms, that’s roughly 1 soft error per device every 450 years. Sounds fine, right? Deploy 100,000 of those devices and you’re looking at about 220 bit-flip events across your fleet every year. Some will land in unused memory and do nothing. Some will corrupt a critical variable and cause a mysterious reboot. At fleet scale, “astronomically unlikely” starts happening every Tuesday.
Catching the Invisible
You don’t need ECC hardware to start detecting bit-flips. Several architecture-agnostic techniques work on practically any MCU. You can’t prevent every flip, but you can make your system notice when one happens.
Software CRC/checksums on critical data. Store a CRC alongside every configuration block, calibration table, or state machine structure. Recompute and verify before each read. If the checksum fails, you know something changed. The overhead is tiny (a few microseconds per check) and the coverage is enormous.
Redundant variable storage. The classic triple modular redundancy (TMR) pattern: store critical variables 3 times in different memory locations, majority-vote on every read. If one copy is corrupted, the other two outvote it. This works for anything that fits in a few bytes: mode flags, safety-critical counters, threshold values.
Stack canaries and memory guards. You probably already use these for buffer overflow detection. They catch bit-flip corruption too. A known sentinel value at the top of your stack or around critical data structures gives you an early alarm.
Periodic memory scrubbing. Walk through static or rarely-changing memory regions on a timer. Compare against known-good values or stored checksums. Log any discrepancy. This is especially useful for catching bit-flips in data that sits in RAM for hours or days, like device configuration or accumulated sensor readings.
Watchdog patterns beyond simple timers. A basic watchdog only tells you the MCU is alive. A windowed watchdog tells you it’s responding at the right cadence. A sequence-checking watchdog goes further: the firmware must feed specific values in a specific order, proving it’s actually executing the correct code path rather than just spinning. This catches corruption in program flow, not just total lockups.
ECC peripherals where available. Some MCU families include hardware ECC on Flash, SRAM, or both. The TI Hercules (TMS570) family, Infineon AURIX TC3xx, and certain STM32 parts offer this. If you’re selecting an MCU for a new design, check the datasheet for ECC support. It’s one of the most effective, lowest-effort mitigations available, and it’s often free if you pick the right part. If you’re integrating firmware on devices that use Bluetooth to relay sensor data, the Hubble device SDK documentation covers how to structure your firmware alongside these kinds of reliability patterns.
Hardening Without Over-Engineering
Mitigation is a spectrum. Treating every $3 consumer widget like an avionics system is wasteful. Treating a medical infusion pump like a consumer widget is dangerous. Match your defenses to your system’s criticality.
Criticality Tier | Examples | Recommended Mitigations
────────────────────┼────────────────────────┼──────────────────────────
Tier 1: Annoyance | Consumer gadget, | Watchdog, CRC on config,
| smart plug | graceful reboot strategy
────────────────────┼────────────────────────┼──────────────────────────
Tier 2: Costly | Industrial sensor, | TMR on critical vars,
| fleet telematics | memory scrubbing, ECC MCU
────────────────────┼────────────────────────┼──────────────────────────
Tier 3: Safety | Medical device, | ECC MCU mandatory, lockstep
| automotive ADAS, | cores, full FMEA on memory,
| avionics | rad-tolerant componentsMost teams should start at Tier 1. A well-designed watchdog and CRC strategy costs almost nothing in code size, execution time, or bill of materials. It won’t catch every bit-flip, but it’ll catch a huge percentage of the ones that matter: the flips that corrupt configuration, break control flow, or wedge a peripheral.
The underlying mindset is graceful degradation: assume corruption will happen and design the system to detect it and recover cleanly. A device that catches a checksum failure, logs it, and reboots into a known-good state will always beat a device that silently runs with corrupted data for hours.
A few more practical wins: keep critical constants in Flash (which is inherently more resistant to bit-flips than SRAM because of its charge storage mechanism). Minimize the time stale data sits in RAM. Keep critical execution paths short so there’s less window for corruption to affect a decision. If your devices report telemetry, set up a webhook endpoint to flag anomalous reboot patterns at the fleet level; catching the statistical signal across thousands of devices is often easier than diagnosing a single unit.
For readers who want to go deeper on SER measurement methodology, the JEDEC JESD89B standard defines how soft error rates are measured and reported for semiconductor devices. It’s dry reading, but it gives you the vocabulary to have informed conversations with your silicon vendor.
What to Do This Sprint
The problem isn’t that bit-flips happen. Physics doesn’t care about your product timeline. The problem is that most embedded teams don’t design for them at all. They treat memory as perfectly reliable, ship the product, and then spend months chasing ghost bugs when the field data says otherwise.
Check your MCU’s datasheet for ECC support on Flash and SRAM. Add CRC verification to your 3 most critical data structures. Audit your watchdog: is it actually checking execution correctness, or just confirming the main loop is spinning? Those 3 things together probably take a senior engineer 2 to 3 days. They’ll eliminate the majority of bit-flip-induced failures your fleet will ever see.
Remember those 50 devices rebooting every week? The fix probably isn’t in your code. It’s in the code you haven’t written yet: the code that expects a bit to flip, catches it, and keeps going.
Hubble Network connects to your devices directly from satellite—even low-power sensors with tiny antennas—so when a bit-flip triggers a reboot in the field, you don’t lose visibility. See how it works →