How to Build Production-Ready IoT Devices with Zephyr: Project Structure, Debugging, and Field Deployment Patterns
Your Zephyr blinky works. It compiled, it flashed, the LED toggles. Ship it to 10,000 units in the field with over-the-air updates, crash diagnostics, three board variants, and a four-person firmware team committing to the same repo.
That gap between sample and product isn’t about features. Zephyr already has the drivers, the BLE stack, the power management. The gap is structural: how you organize the project, how you debug when things break at 3 AM, and whether your build system helps or haunts you when production deadlines hit. Most teams that abandon Zephyr don’t leave because of missing functionality. They leave because they never invested in understanding the build system, and the resulting mess became untenable around month three.
This article gives you the opinionated blueprint to skip that pain. We’ll use the nRF52840-DK as our reference platform, but every pattern here generalizes to any Zephyr-supported SoC.
Why Your Workspace Layout Is a Load-Bearing Decision
Zephyr offers two workspace topologies. Use T2 (application-centric). Full stop. In T2, your application repo is the west manifest repository. Zephyr, MCUboot, and HAL modules are dependencies you pin to exact commits in west.yml. This gives you reproducible builds, clean CI pipelines, and the ability to update Zephyr on your schedule, not the other way around.
Here’s the layout you should start with:
my-iot-product/ ← Your repo (manifest repo)
├── west.yml ← Pins Zephyr, MCUboot, HAL versions
├── CMakeLists.txt
├── prj.conf ← Base Kconfig (common to all boards)
├── app.overlay ← Base devicetree overlay (if any)
├── boards/
│ ├── nrf52840dk_nrf52840.conf ← Board-specific Kconfig
│ ├── nrf52840dk_nrf52840.overlay ← Board-specific DT overlay
│ └── custom_board/ ← Out-of-tree board definition
│ ├── custom_board_defconfig
│ ├── custom_board.dts
│ └── custom_board.yaml
├── drivers/ ← Out-of-tree drivers
│ └── sensor/my_sensor/
├── dts/bindings/ ← Custom devicetree bindings
├── src/
│ ├── main.c
│ ├── ble/
│ │ ├── CMakeLists.txt
│ │ ├── ble_service.c
│ │ └── ble_service.h
│ ├── sensors/
│ │ ├── CMakeLists.txt
│ │ └── sensor_mgr.c
│ └── transport/
│ ├── CMakeLists.txt
│ └── mqtt_client.c
├── include/ ← Public app headers
├── tests/ ← ztest unit tests
├── scripts/ ← Build/flash/sign helpers
└── child_image/
└── mcuboot.conf ← MCUboot overlay configsTwo things to notice. First, src/ is split by subsystem, each with its own CMakeLists.txt. This isn’t cosmetic. It enforces modular compilation and makes it trivial for one engineer to own src/ble/ while another works in src/sensors/ without merge conflicts in a monolithic source list. Second, the boards/ directory is doing critical work that many teams discover too late.
Kconfig and Devicetree Discipline That Scales
Here’s the anti-pattern: a single prj.conf with 200 lines, half of them wrapped in comments like “# only for nrf52840” and “# enable for debug builds.” This is the Zephyr equivalent of #ifdef spaghetti, and it will break you when you add a second board variant.
The pattern: base prj.conf carries only configuration common to every board and build type. Board-specific settings go in boards/<board_name>.conf. Zephyr’s build system picks these up automatically, no extra CMake plumbing needed.
# prj.conf — base, all boards
CONFIG_BT=y
CONFIG_BT_PERIPHERAL=y
CONFIG_LOG=y
CONFIG_SENSOR=y# boards/nrf52840dk_nrf52840.conf — board-specific
CONFIG_BT_CTLR=y
CONFIG_BOARD_ENABLE_DCDC=y
CONFIG_GPIO_AS_PINRESET=yFor build-type variants (debug vs. release), use CMake cache variables or separate .conf fragments passed via OVERLAY_CONFIG:
west build -b nrf52840dk_nrf52840 -- -DOVERLAY_CONFIG="debug.conf"The same principle applies to devicetree overlays. Your app.overlay carries base overlay changes; boards/nrf52840dk_nrf52840.overlay carries board-specific pin mappings or peripheral configurations. When you eventually spin a custom PCB, you add a board definition under boards/custom_board/ with its own .dts and _defconfig. Nothing in your application code changes. That’s the payoff.
Picking the Right Debug Tool at the Right Time
Debugging Zephyr on the nRF52 is well-supported, but the ecosystem offers enough tools that knowing which one to reach for matters more than knowing they exist.
┌─────────────────────┬──────────────────┬───────────────────┬──────────────────┐
│ Scenario │ Primary Tool │ Secondary Tool │ Key Config │
├─────────────────────┼──────────────────┼───────────────────┼──────────────────┤
│ Hard fault / crash │ GDB + J-Link │ addr2line │ CONFIG_DEBUG_INFO│
│ Thread deadlock │ GDB thread-aware │ Zephyr shell │ CONFIG_DEBUG_ │
│ │ │ (thread analyze) │ THREAD_INFO │
│ Timing / perf issue │ Logic analyzer │ Logging (deferred)│ CONFIG_LOG_MODE_ │
│ │ + GPIO toggles │ │ DEFERRED │
│ BLE stack issue │ RTT logging │ nRF Sniffer │ CONFIG_USE_ │
│ │ │ │ SEGGER_RTT │
│ Field failure │ Coredump to flash│ Post-mortem GDB │ CONFIG_DEBUG_ │
│ (post-mortem) │ │ │ COREDUMP │
└─────────────────────┴──────────────────┴───────────────────┴──────────────────┘On-Target GDB: Your First Instinct Should Be Right
west debug launches GDB connected to your target via J-Link. west attach connects without resetting. Use this when you need to inspect a running system that’s already in a bad state.
The critical enabler most teams miss: thread-aware debugging. With CONFIG_DEBUG_THREAD_INFO=y, GDB can enumerate all Zephyr threads, show their states, and let you switch context between them. When you’ve got eight threads and a hard fault, this is the difference between a ten-minute fix and a two-day investigation. SEGGER Ozone and VS Code with the Cortex-Debug extension both support this, if you prefer a GUI.
Shell and Logging as Runtime Debug Tools
Zephyr’s shell subsystem (CONFIG_SHELL=y) gives you a command-line interface over UART or RTT. The kernel threads command shows thread stack usage, states, and priorities, live, on target. During development, enable the shell over RTT (no UART wiring needed). In production builds, disable it entirely or gate it behind a GPIO jumper.
For the logging subsystem, one rule matters above all others: use deferred logging mode in any timing-sensitive code path. Deferred mode queues log messages to a processing thread, so your ISR or real-time loop doesn’t stall waiting for UART transmission. Only switch to immediate mode (CONFIG_LOG_MODE_IMMEDIATE=y) when you’re actively chasing a crash where the system resets before the log thread flushes.
Per-module log levels let you crank src/ble/ to LOG_LEVEL_DBG while keeping src/sensors/ at LOG_LEVEL_WRN. This is configured in Kconfig, not littered through source files.
Reading Fault Output and Post-Mortem Analysis
When Zephyr hits a hard fault, the built-in fault handler dumps registers, the faulting instruction address, and the active thread to your console. Grab the PC value and run:
arm-none-eabi-addr2line -e build/zephyr/zephyr.elf 0x0001a3c4This gives you the exact source file and line number. With CONFIG_DEBUG_INFO=y (which you should always enable in development builds), the ELF contains full debug symbols.
For field failures where you can’t be connected, Zephyr’s coredump subsystem (CONFIG_DEBUG_COREDUMP=y) can write a core dump to flash. Retrieve it on the next connection and analyze it offline with GDB. This is remarkably powerful for devices that fail intermittently in environments you can’t replicate on your bench.
nRF52 Pitfalls That Waste Days
Flash region collisions. If you’re using Nordic’s MPSL for the BLE controller (which you likely are via Zephyr’s HCI driver), be aware that it reserves specific flash and RAM regions. Adding MCUboot to the mix carves up flash further. Get your memory map right in your linker overlay early. Discovering a collision at month four is costly.
west flash --erase is destructive. On nRF52, this erases the entire flash, including the MBR and any bootloader. Use west flash without --erase as your default, or use --erase only intentionally during initial provisioning.
Premature power optimization. Enabling CONFIG_PM=y before your application logic is stable will mask race conditions. Idle threads that enter sleep states change timing enough to hide (or create) concurrency bugs. Get your system solid first, then enable PM and retest.
Field Deployment Patterns That Follow From Good Structure
The project structure from earlier isn’t just for clean code. It directly enables production deployment. This topic warrants its own deep-dive article, but here’s the skeleton.
MCUboot and OTA Updates
MCUboot integrates as a child image in Zephyr’s build system. Your child_image/mcuboot.conf overrides MCUboot’s Kconfig for your specific flash layout and key configuration. Building with MCUboot enabled produces a signed application image. The west sign command handles image signing against your key material.
For the nRF52840, DFU over BLE using mcumgr is the standard OTA path. The mcumgr library runs as a Zephyr subsystem in your application, accepting signed image chunks over a BLE SMP service. Because your project already separates board overlays and Kconfig fragments, adding MCUboot doesn’t require restructuring anything.
Watchdog and Graceful Recovery
Enable CONFIG_WATCHDOG=y in your production prj.conf. Configure CONFIG_ASSERT to trigger a controlled reset rather than a hang. The production pattern: on assertion failure, write the fault context (PC, thread ID, assert file/line) to retained RAM (memory that survives a reset on nRF52), then trigger a system reset. On the next boot, check retained RAM, and if fault data exists, queue it for upload on the next server connection.
This turns every field crash from a mystery into a diagnosable event.
Power and Connectivity Lifecycle
Zephyr’s PM subsystem and connection manager APIs are mature enough for production, but they require careful integration with your application’s state machine. We’ll cover fleet-scale telemetry, power budgeting, and connection lifecycle management in a dedicated article. These topics deserve more depth than a sketch can provide.
Your Production Readiness Checklist
Before you ship, walk through this:
┌───┬──────────────────────────────────────────┬─────────────────────┐
│ # │ Decision / Action │ Phase │
├───┼──────────────────────────────────────────┼─────────────────────┤
│ 1 │ T2 (app-centric) west workspace │ Project Setup │
│ 2 │ Modular src/ with per-subsystem CMake │ Project Setup │
│ 3 │ Base prj.conf + per-board .conf fragments │ Project Setup │
│ 4 │ Board-specific .overlay files in boards/ │ Project Setup │
│ 5 │ Thread-aware GDB configured │ Debug Setup │
│ 6 │ RTT logging + shell enabled (dev builds) │ Debug Setup │
│ 7 │ Deferred logging for timing-sensitive code │ Debug Setup │
│ 8 │ MCUboot integrated as child image │ Deployment Prep │
│ 9 │ Signed image workflow (west sign) │ Deployment Prep │
│ 10│ Watchdog + assert-to-retained-RAM pattern │ Deployment Prep │
│ 11│ CI pipeline: west build + twister tests │ Deployment Prep │
│ 12│ PM subsystem validated under load │ Pre-Production │
└───┴──────────────────────────────────────────┴─────────────────────┘Building on This Foundation
Zephyr’s production readiness is real, but the learning curve is front-loaded in the build system: west, CMake, Kconfig, Devicetree. Teams that invest in understanding these four pieces during the first two weeks save months of thrashing later. Teams that try to treat Zephyr like a vendor SDK and fight the build system never stop paying for it.
The patterns here, T2 workspace, modular Kconfig, thread-aware debugging, MCUboot integration, aren’t optional extras for teams that ship products. They’re the foundation that makes everything else (OTA, fleet management, multi-variant builds, CI/CD) possible without heroics.
Start by cloning your current prototype into the directory structure above. Get west build producing clean builds for your board. Add one boards/*.conf fragment. Run west debug and verify thread-aware debugging works. That’s your first afternoon. Everything after that is building on solid ground.
Hubble Network enables Bluetooth IoT devices to transmit data directly to satellites—no gateways, no terrestrial infrastructure required. See how it works →