Zephyr Thread Priorities and Scheduling: Avoiding Priority Inversion in BLE Firmware

Debugging thread scheduling and priority inversion in Zephyr RTOS firmware for BLE devices

Your BLE peripheral disconnects every few minutes. No crash. No error log. Just a quiet supervision timeout and a dropped link. You’ve checked your antenna, your connection parameters, your PHY settings. Everything looks fine. It’s your thread priorities.

Bluetooth has hard real-time deadlines baked into the protocol. Miss a connection event, and the link layer doesn’t complain; it just starts counting toward a supervision timeout. Stack up a few misses, and the connection drops. The root cause is a scheduling problem in your application code.

Here’s how to find it, fix it, and design your Zephyr thread priority layout so it doesn’t happen again.

Why BLE Firmware Is Especially Vulnerable

Zephyr’s native BLE stack runs host-layer processing in real kernel threads. Those threads, bt_rx and bt_tx, share a scheduler with your application threads. They compete for CPU time on the same core using the same priority system.

The BLE controller enforces hard timing. The T_IFS (inter-frame space) is 150 microseconds. Connection events have strict windows. If host-layer processing stalls because your app thread is holding a resource it needs, the controller can’t do its job.

Picture this: your sensor-reading thread grabs a mutex protecting a shared data buffer. A medium-priority logging thread preempts it. Meanwhile, bt_rx wakes up to process an incoming HCI event, needs that same mutex, and blocks. No HCI processing means no connection event response. The link supervisor counts a miss. A few more, and you get disconnected with reason 0x08. No crash, no assert, just silence.

That’s priority inversion. And it’s the default outcome if you don’t design your priority map with the BLE stack’s internal threads in mind.

Zephyr Scheduling: The 30-Second Refresher

You know this stuff, but the details matter here.

Priority Scale (Zephyr Default Config)
───────────────────────────────────────
 -16  ← Highest cooperative (ISR-like)
  ...
  -1  ← Lowest cooperative
   0  ← Highest preemptive
  ...
  14  ← Lowest preemptive (typical main)
───────────────────────────────────────
  Cooperative: never preempted by scheduler
  Preemptive:  can be preempted by higher-prio

Key points for this discussion:

Lower numerical value = higher priority. Priority -8 beats priority 0, which beats priority 5.

Cooperative threads (negative priorities) run until they explicitly yield or block. The scheduler won’t yank the CPU away from them. Preemptive threads can be interrupted by any higher-priority thread becoming runnable.

Time-slicing only applies among preemptive threads at the same priority level. It does nothing for cooperative threads.

CONFIG_NUM_COOP_PRIORITIES and CONFIG_NUM_PREEMPT_PRIORITIES set the available range. The defaults (16 and 16) are usually fine, but check yours.

Mapping the Native BLE Stack’s Internal Threads

This is the part that catches people. Zephyr’s BLE stack quietly spawns threads at boot, and they have specific priorities you need to know about:

Thread         Default Prio  Type          Kconfig Override
─────────────────────────────────────────────────────────
BT Controller  -14 (typ.)    Cooperative   CONFIG_BT_CTLR_*_PRIO
bt_rx          -8            Cooperative   CONFIG_BT_RX_PRIO
bt_tx          -7 (typ.)    Cooperative   CONFIG_BT_TX_PRIO (if exists)
sysworkq        0            Preemptive    CONFIG_SYSTEM_WORKQUEUE_PRIORITY
─────────────────────────────────────────────────────────

The controller threads sit at the top of the priority stack because they handle link-layer timing. Don’t touch these.

bt_rx handles all incoming HCI event processing on the host side: parsing events, processing L2CAP, triggering GATT callbacks. At priority -8, it’s cooperative, meaning it runs without preemption whenever it’s ready.

bt_tx handles outbound HCI command serialization.

The system workqueue (k_sys_work_q) is where some BLE callbacks execute, especially those deferred via k_work_submit(). It runs at preemptive priority 0 by default.

⚠️ These defaults vary across Zephyr versions. Always check Kconfig.bt in your specific version’s subsys/bluetooth/ directory. The priorities listed here are representative, not guaranteed.

These BLE threads are cooperative because they must not be preempted during time-critical host processing. A preemption mid-parse could mean a missed response window. Cooperative scheduling guarantees they run to completion (or until they voluntarily block).

How Priority Inversion Actually Happens

Let’s trace the exact sequence. If you’re building a BLE peripheral that reads sensor data and sends GATT notifications, you probably have something like this:

Time →
─────────────────────────────────────────────────
App (P2)    ██████[holds mutex]░░░░░░░██[release]
Log (P1)    ·······████████████████████··········
bt_rx(-8)   ···············XX[BLOCKED]X██ runs ██
─────────────────────────────────────────────────
            ↑ App gets mutex  ↑ bt_rx needs it  ↑ resolved

Step by step:

  1. App thread (priority 2, preemptive) acquires a k_mutex protecting a shared sensor data buffer. It starts formatting data.
  2. Logging thread (priority 1, preemptive) becomes runnable. Since priority 1 beats priority 2, it preempts the app thread. It starts writing to flash or UART. This takes a while.
  3. bt_rx (priority -8, cooperative) wakes up to process an incoming HCI event. It needs the same mutex to read the sensor buffer for a GATT notification. It calls k_mutex_lock() and blocks.
  4. bt_rx is now stuck. It can’t process HCI events. The logging thread keeps running because it doesn’t need the mutex. The app thread can’t run because the logging thread has higher preemptive priority.

bt_rx being cooperative doesn’t save it here. Cooperative scheduling means it won’t be preempted, but k_mutex_lock() is a voluntary block. The thread is waiting, not running. The scheduler won’t give it the mutex; only the app thread can release it, and the app thread is stuck behind the logging thread.

This is textbook priority inversion: a medium-priority thread indirectly blocks the highest-priority thread in the system.

Fix 1: Enable Priority Inheritance (It’s Already There)

Zephyr’s k_mutex implements priority inheritance by default. When bt_rx blocks on a mutex held by the app thread, the app thread temporarily inherits bt_rx’s priority (-8). This lets it preempt the logging thread, finish its critical section, release the mutex, and unblock bt_rx.

But this only works if you’re using k_mutex. Here’s the trap:

⚠️ If you’re using k_sem_give()/k_sem_take() as a binary lock, you get zero priority inheritance. Semaphores have no concept of ownership in Zephyr, so the kernel can’t boost the holder’s priority. Always use k_mutex for mutual exclusion.

The correct pattern:

static K_MUTEX_DEFINE(sensor_buf_mutex);

/* In your app thread: */
k_mutex_lock(&sensor_buf_mutex, K_FOREVER);
/* ... update shared buffer ... */
k_mutex_unlock(&sensor_buf_mutex);

/* In your BLE notification callback: */
k_mutex_lock(&sensor_buf_mutex, K_MSEC(5));
/* ... read shared buffer ... */
k_mutex_unlock(&sensor_buf_mutex);

Use a timeout on the BLE side. K_FOREVER in a cooperative BLE thread is asking for trouble if something goes wrong.

Fix 2: Design Your Priority Map Intentionally

Priority inheritance is a safety net. A well-designed priority map is the real fix. Here’s a layout I’d recommend for a typical BLE peripheral (sensor product):

Recommended Priority Map: BLE Peripheral
──────────────────────────────────────────────────
Prio   Thread                 Notes
──────────────────────────────────────────────────
-14    BT Controller          Don't touch
 -8    bt_rx                  Don't touch
 -7    bt_tx                  Don't touch
 -2    Sensor ISR workqueue   Dedicated k_work_q
 -1    Critical app logic     Cooperative, short-lived
  0    System workqueue       BLE callbacks land here
  2    Application main       General app logic
  5    Display / UI thread    Non-critical
 10    Logging thread         Lowest priority
──────────────────────────────────────────────────

Three principles:

Never place app threads at or above BLE stack priorities unless you’ve deeply analyzed the consequence. Your sensor-reading thread at priority -9 will preempt bt_rx every time it wakes up.

Keep critical section hold times tiny at every priority level. If you hold a mutex for 2 ms at priority 2, that’s 2 ms that bt_rx could be blocked even with priority inheritance active.

Use dedicated workqueues for high-priority app work. The system workqueue is shared with BLE callbacks, so dumping your app work there creates contention that leads to exactly the kind of scheduling confusion you’re trying to avoid. Create your own:

#define SENSOR_WQ_STACK_SIZE 1024
#define SENSOR_WQ_PRIORITY   -2

K_THREAD_STACK_DEFINE(sensor_wq_stack, SENSOR_WQ_STACK_SIZE);
static struct k_work_q sensor_work_q;

/* In your init function: */
k_work_queue_start(&sensor_work_q, sensor_wq_stack,
                   K_THREAD_STACK_SIZEOF(sensor_wq_stack),
                   SENSOR_WQ_PRIORITY, NULL);

Submit time-sensitive sensor work to sensor_work_q instead of k_sys_work_q. This gives you explicit control over priority without polluting the system workqueue.

If you’re building on Zephyr with BLE, the Zephyr RTOS quick-start guide for terrestrial devices covers the initial project scaffolding, and the Hubble Zephyr reference app shows a working priority layout you can use as a starting point.

Fix 3: Eliminate Shared State Entirely

The cleanest fix for priority inversion is removing the inversion vector altogether. If BLE threads and app threads never share a mutex, inversion can’t happen.

The pattern: BLE callbacks write into a k_msgq or k_pipe. App threads read from the queue at their own pace. No mutex, no contention, no inversion.

K_MSGQ_DEFINE(sensor_msgq, sizeof(struct sensor_reading), 8, 4);

/* In your BLE notification prep (called from BLE context): */
k_msgq_put(&sensor_msgq, &reading, K_NO_WAIT);

The app thread pulls from the queue whenever it’s ready. BLE threads never block. Default to this pattern. Only reach for shared mutexes when message-passing genuinely can’t express your data flow.

Diagnosing Priority Inversion

Before you rearchitect anything, confirm the problem. Here’s a triage checklist:

  1. CONFIG_THREAD_ANALYZER=y and CONFIG_THREAD_ANALYZER_AUTO=y. These dump thread states, priorities, and stack usage periodically. Look for your BLE threads showing “pending” or “blocked” state while a lower-priority thread shows “running.”

  2. CONFIG_SCHED_THREAD_USAGE=y gives you per-thread CPU consumption numbers. A mid-priority thread eating 80%+ of cycles is your prime suspect for blocking everything below and starving everything above.

  3. CONFIG_BT_DEBUG_LOG=y (or the equivalent in your Zephyr version’s logging config). Watch for “supervision timeout” or disconnect reason 0x08, the calling cards of missed connection events.

  4. CONFIG_TRACING=y with Segger SystemView or CTF output is the gold standard. You get a timeline of every context switch, and you can literally see the moment bt_rx blocks, the mid-priority thread runs, and the inversion plays out.

  5. Grep for k_sem_take used as mutual exclusion. Any binary semaphore protecting shared data should be converted to k_mutex before you investigate further. It’s the single most common cause.

For ongoing monitoring, the Hubble device SDK documentation covers how telemetry and diagnostic data can be structured alongside your BLE payload, which is useful for catching these issues in deployed devices.

The Three-Part Fix

Priority inversion in BLE firmware doesn’t crash; it disconnects. Three things prevent it:

  • Use k_mutex (never bare semaphores) for any shared resource.
  • Design your priority map with full awareness of the BLE stack’s internal threads at -14, -8, and -7.
  • Prefer message-passing over shared state wherever the architecture allows.

When a mystery disconnect shows up, enable the thread analyzer and tracing before you start guessing. The data will show you exactly where the inversion is happening, and you’ll fix it in an afternoon instead of fighting ghosts for a week.

Hubble Network enables direct satellite connectivity for BLE devices—no gateways, no extra infrastructure. See how it works →