How SpaceX Ships Firmware to Millions of Terminals — and What Your BLE Fleet Can Steal From Their Rollout Playbook

Staged firmware rollouts to millions of connected devices without breaking the fleet

Starlink has millions of dishes bolted to roofs across 100+ countries, and SpaceX pushes firmware to them without a technician ever climbing a ladder. What’s striking isn’t the scale. It’s how rarely you hear about a bad update bricking someone’s dish.

SpaceX has said little about the internals, and I won’t pretend to know them. But you can infer how it works from how any sane operation at that scale runs: updates go out in stages, telemetry decides whether the next batch ships, and there’s a way back when something goes wrong. Reputable reporting on Starlink software updates lines up with this. Nothing exotic, just discipline applied hard.

You don’t have a satellite budget. But the rollout discipline is 100% portable to your BLE fleet, whether you’re running 500 sensors or 5 million trackers.

Here’s the playbook, translated for embedded engineers about to scale past the “works on my desk” phase and (rightly) nervous about the 2am brick-the-whole-fleet scenario.

The Core Pattern: Blast Radius Control

A staged firmware rollout limits how many devices can fail at once. You don’t optimize for speed. You optimize for the size of the crater when something breaks, because eventually something will.

The pattern is a set of expanding rings. Ship to a tiny slice, watch it, then widen. Between each ring sits a gate: a decision point where telemetry either says “keep going” or “stop everything.”

FIRMWARE ROLLOUT: BLAST RADIUS CONTROL

  Canary        Ring 1       Ring 2        Full Fleet
  ~0.5%    -->   ~5%    -->   ~25%    -->    100%
  [....]        [########]   [############] [############]
    |             |             |              |
   GATE          GATE          GATE          done
 (24-48h)      (auto-check)  (auto-check)
    |             |             |
    +-- fail? HALT + ROLLBACK --+

This is the same idea Google’s SRE teams describe as progressive rollout with canary analysis, and it’s what tools like Mender, Balena, and Memfault build into their fleet products. If you’re already on Nordic’s nRF or ESP-IDF OTA, you know the single-device mechanics. Staging is the layer that sits on top once you have more than one device to worry about.

Stage 1: The Canary Cohort

Pick 0.1 to 1% of your fleet and ship to them first. For 100,000 devices that’s 100 to 1,000 units. Small enough that a total loss is annoying, not fatal.

Don’t pick them at random from one warehouse. Spread the canary across hardware revisions, firmware baselines, and physical environments. A v1 board in a cold outdoor enclosure behaves differently from a v3 board on someone’s desk, and you want both in the first wave. Bugs love the corners you forgot about.

Define success before you ship, not after. Write down the numbers you’ll accept. If you decide the threshold after seeing the data, you’ll rationalize a bad build through. Reasonable starting anchors (tune these to your fleet):

  • Crash rate under 0.5%
  • Update success rate above 98%
  • Reconnect regression under 2% versus the prior firmware
  • Battery drain delta under 5%

Then watch the canary for 24 to 48 hours. BLE devices sleep, wake, and report on their own schedules, so you need real elapsed time to see a representative sample check in. An hour tells you almost nothing.

If the canary holds, you’ve earned the right to widen. If it doesn’t, you’ve contained the damage to a rounding error.

Stage 2: Progressive Cohorts & Telemetry Gates

After the canary, expand in rings: 5%, then 25%, then 100%. Each jump is gated by automated checks against the same metrics you defined up front.

The gate should be code, not a person eyeballing a dashboard at midnight. A human is fine for the go/no-go on the canary, where the sample is small and the stakes are high. But once you’re auto-promoting thousands of devices per ring, you want the pipeline to halt itself the moment a threshold trips.

        +---------------------------+
        |  Promote to next cohort?  |
        +---------------------------+
                     |
     crash rate < X? ---- NO ---> HALT
                     | YES
   reconnect rate ok? --- NO ---> HALT
                     | YES
   battery delta < Y? ---- NO --> HALT
                     | YES
                  PROMOTE

Four gate conditions cover most of what kills a fleet:

  • Crash rate. Watchdog resets and fault handlers firing. The clearest signal something’s broken.
  • Reconnect rate. If devices start dropping their link and reconnecting more than baseline, your new firmware probably broke the radio stack or timing. This one’s easy to miss because devices still “work,” just badly.
  • Battery drain delta. A firmware bug that leaves the radio on too long won’t crash anything. It’ll just quietly halve your battery life and you’ll find out in six months. Watch current draw or voltage slope against the prior version.
  • Update success rate. What fraction of targeted devices actually took the update and booted the new image? A sagging number here often means a transfer or connectivity problem, not a firmware bug.

Make the halt automatic and loud. When a ring trips a gate, the pipeline stops promoting, pages whoever’s on call, and holds the fleet at its current split until a human clears it. The cost of a false halt is a delayed rollout. The cost of a missed one is the whole fleet.

Wider rings need less soak time than the canary because you’ve already burned down the obvious bugs. A few hours per ring is often enough once the metrics are stable, though this depends entirely on how often your devices report.

Stage 3: OTA Rollback, Designed In

Here’s the part people skip and regret: you can’t bolt rollback on after you’ve shipped. It has to be in the firmware architecture from day one.

The standard mechanism is A/B (dual-bank) partitioning. You carry two firmware slots. One’s active, one’s staged. The new image writes to the inactive bank, and only after it boots cleanly and passes a self-check do you mark it permanent.

| Slot   | Version | State    |
|--------|---------|----------|
| Bank A | v2.3    | ACTIVE   |
| Bank B | v2.4    | STAGED   |
  -> boot v2.4; watchdog fails -> revert to Bank A

If v2.4 boots and hangs, or the watchdog doesn’t get its “I’m healthy” pat within a timeout, the bootloader reverts to Bank A on the next reset. The device heals itself without anyone touching it. Nordic’s nRF and ESP-IDF both support this pattern natively, so you’re not building it from scratch.

Two things worth separating. Roll forward means pushing a fix as a new version. Roll back means reverting devices to the last known-good image. You want both. Roll forward is your default once a device is reachable and stable. Roll back is your safety net for devices that got a bad image and can’t be trusted to accept a new one over the air.

Version pinning matters too. When you halt a rollout, you want to freeze the mixed fleet exactly where it is. Some devices sit on v2.3, some on v2.4, and you don’t want stragglers to keep pulling the bad build. Your pipeline should treat “which version should this device be on” as an explicit, per-cohort setting, not a global “latest wins.”

Skip this stage on a large-scale rollout and your first bad image becomes a field recall.

The BLE Reality Check

BLE adds constraints that change the details.

Intermittent connectivity. Starlink dishes are powered and mostly online. Your BLE devices sleep to save battery, wake briefly, and may not be reachable for hours or days. Your rollout timeline stretches to match. A ring isn’t “done” when you’ve sent the image; it’s done when enough devices have actually checked in, taken it, and reported back. Plan gate windows around your devices’ duty cycle, not your patience.

Small MTU and slow transfer. BLE’s default ATT MTU is tiny (23 bytes, often negotiated up to 247). Even with data-length extension, moving a firmware image over GATT is slow and fragile compared to a broadband link. Larger images mean longer connections, more chances to drop mid-transfer, and more battery burned per update. This is a real reason to keep images small and to lean on the update-success-rate gate.

Tight power budgets. Every retry, every failed transfer, every extra minute of radio-on costs coulombs you can’t get back on a coin cell. The battery-delta gate carries real weight here, not a nice-to-have.

And the constraint that trips up everyone scaling a distributed fleet: reach. A device you can’t talk to can’t be updated and can’t be observed. Your whole staged pipeline assumes you can reach the devices in each ring. For a fleet spread across cities, countries, or shipping containers, that normally means deploying and maintaining your own gateway infrastructure everywhere your devices go.

That’s the step Hubble’s gateway network removes. Devices reach Hubble’s satellites directly over BLE (7 in orbit today, with the constellation expanding), backed by terrestrial gateways, so your fleet gets global reach without you shipping and maintaining hardware in every region. For the mechanics of getting firmware and data across that link, the BLE OTA and device integration guides walk through it, and the terrestrial network docs cover how the reach layer works.

Your Rollout Checklist

  • Define success metrics and thresholds before shipping (crash < 0.5%, update success > 98%, reconnect regression < 2%, battery delta < 5% as starting points)
  • Build A/B dual-bank partitioning into firmware from day one
  • Add a watchdog-triggered revert to last known-good
  • Pick a canary of 0.1 to 1%, spanning hardware revs and environments
  • Soak the canary 24 to 48 hours (or longer for slow duty cycles)
  • Expand in rings: 5% to 25% to 100%
  • Automate telemetry gates between rings, with auto-halt on breach
  • Support version pinning so a halt freezes the fleet in place
  • Size gate windows to your devices’ sleep/report cadence, not clock time
  • Confirm you can reach and observe every device before you rely on any of it

Start With Reach, Then Stage Everything

The teams that ship firmware to huge fleets without headlines aren’t faster than you. They’re stricter about blast radius. Stage everything, gate on real telemetry, and design rollback in before you need it.

The prerequisite under all of it is reach: your rollout only works on devices you can actually talk to. If your fleet is distributed and you’d rather not deploy gateways everywhere, take a look at Hubble’s global BLE network and sign up free at hubble.com to see whether direct satellite BLE reach fits your rollout.

Hubble Network gives you direct satellite reach to every BLE device in your fleet—no gateway deployment required. See how it works →