Backend Implementation

Ingesting device packet data at scale requires designing a robust pipeline from Hubble's cloud to your backend.

---

Device Mapping

Every device that you register with Hubble Network will have a unique device.device_id as the primary UUID identifier. You may choose to add identifiers for device mapping in your organization:

• Use Update Device to populate device.name - for example, with your proprietary Device ID. Note that uniqueness is not enforced by Hubble.
• You may also append Tags in key:value pairs for your devices - for example, if you have a category of devices each with a unique ID.

---

Packet Ingestion

Establish a connection to your data transmitted by Hubble Network, then process packets based on what matters for your business.

Set Up your Data Pipeline

There are two primary methods for consuming packet data on Hubble's cloud: API request and webhook subscription. Learn more about the advantages of each, and how to use them in combination, in our Get Device Packet Data guide.

Tactics for Testing Packet Ingestion:

• API: Use the Retrieve Organization Packets API with the optional device.id parameter to return packet data for a specific test device
• Webhook: Test receiving an array of packet data via webhook by setting a small max_batch_size initially, then ramp batch size to the maximum as you scale the number of devices broadcasting on Hubble Network

Migrating to a New Webhook Endpoint

Hubble supports up to two (2) configured webhooks at once to facilitate zero-downtime transitions. This enables safe migration from an old endpoint to a new one with minimal risk or packet loss.

Follow these steps to seamlessly between migrate webhook endpoints:

1. Deploy your new endpoint: ensure that the new HTTPS webhook URL is able to accept and process incoming events in the same format as your current implementation.
2. Add the new endpoint as a secondary Hubble Network webhook: whether using the Organization Metadata API or the developer tools in your organization's dashboard, add the new webhook without removing the existing one. Both endpoints will receive identical webhook events during the migration.
3. Monitor and validate the new endpoint: Verify that the new endpoint is processing payloads correctly. You can compare logs from both endpoints to ensure consistency in received payloads and timing.
4. Transition any dependent systems: If downstream systems rely on webhook processing, begin routing them through the new endpoint once it's validated.
5. Remove the old endpoint: After the migration is complete, we recommend deregistering the old endpoint in advance of a future migration.

Process Packet Data

You may parse packet data objects to map data to your backend and run integrated workflows. Learn more about the data objects returned in device packets in our Parsing Packets guide.

• Reference deviceId, deviceName or tag when mapping devices on your backend.
• Retrieve timestamp and GPS coordinates along with accuracy metadata from the scanning device that recognized your device.
• Decode the custom payload (base 64 encoded) if you transmit additional data from the device, like sensor readings.
• Note the network field if your integration workflows differ by network.

---

Filtering Data Packets

By design, all data packets transmitted from your registered devices through the network are made available on the Hubble cloud. Hubble does not perform any immediate filtering or aggregation of raw captures. Our priority is to provide you with the complete data set.

Packet Uniqueness

During normal operation, every packet delivered by Hubble is unique. (Note that because of our "at least one" delivery policy, duplicate packets may be sent through webhook subscription.) However, you may expect to ingest nearly identical packets in some cases. For example, multiple scanners in the same area may record the same transmission from a single device, resulting in multiple captures of the same data packet with slightly varied location and/or timestamp data.

Filtering Multiple Packets

A packet filtering strategy may be applicable, depending on your product requirements.

• What is the most important unit of information for your use case? Is it the unique device transmission (packet), or the occurrence of an event at a location and time?
• How critical is it to capture every single raw transmission?
• What are the potential consequences of processing identical data in your downstream systems?

Below are some components of potential filtering strategies based on the characteristics of the data.

Component	Strategy	Implementation Detail	Use Case
By Device Packet	Identify redundant captures if packet counter and sequence number combination is the same.	Every device payload will have a unique combination of the rollover counter and sequence_number, which increments with each transmitted packet and resets upon counter rollover. These values are generated by the Hubble SDK to denote the latest data. You may also decide to include a bespoke counter in the packet payload that you define.	Prioritize latest device packet data over location updates. Ensure that you process each unique data transmission from the device only once.
By Location	Identify redundant captures if location data is essentially the same within a time threshold.	Only keep one data point if multiple packets are returned in the same time window with nearly identical GPS location. Choose your location sensitivity based on significant digits returned with latitude, longitude and altitude.	Prioritize location events over device packet contents. Ensure that you update location data only when a meaningful location change has occurred.
By Timestamp	Sample packet or location data captured within a defined time threshold.	Define a short window, such as 60 seconds. If multiple packets are received in this window for the same device, choose a rule to determine which record(s) you will process. If prioritizing unique packets, use device.timestamp; if prioritizing location updates, use location.timestamp. Sample among >1 packets based on first/last received, location accuracy or RSSI, proprietary data returned in the payload, etc.	Helpful if mitigating near-simultaneous readings from multiple scanners with slight variation in timestamp, both when prioritizing packet data or location data.

Filtering Individual Packets

For a single packet, you may see a time difference between the two timestamps returned in the packet data: location.timestamp and device.timestamp. Learn more about the data objects returned in device packets in our Parsing Packets guide.

Scenario	Technical Context	Root Cause
Typical: location.timestamp < device.timestamp	Standard mobile scan: Mobile gateways like a smart phone scanning for Hubble refresh its location periodically (e.g., every 30 seconds). When the gateway detects a Hubble device (device.timestamp), it attaches GPS and time from its most recent location lock (location.timestamp).	The gateway had a valid, slightly older location lock cached when the device was detected.
Delayed Sync: location.timestamp > device.timestamp	Backlogged transmission: If the mobile gateway detects a device while offline (no cellular or WiFi connection) the packet is queued until a network connection is reestablished.	Upon regaining connection, the gateway acquires a new location lock before forwarding the queued packets to the Hubble backend.

Recommendation for filtering individual packets:

• If you primarily care about location, a large difference in timestamps might suggest discarding that device detection.
• If you primarily care about custom payload reported by the beaconing device (e.g. telemetry or sensor readings) you should use device.timestamp, or include time in the custom payload.