Solution Architecture: IoT Device Data and Telemetry Ingestion

[janhalen]

Implement an standardised ingestion pipeline to integrate proprietary data transfers from select IoT devices with the FIWARE ecosystem, supplementing the limitations of the standard FiWare IoT Agent.

---

Architectural pattern recommended by @janhalen ⬇️

Adapter Pattern:

Seek to avoid the overhead and maintenance chores of a custom-coded "Front Broker" by reusing existing Cloud Native patterns and components.

By adopting this pattern, we 'swim with the current' of the Kubernetes ecosystem, favoring the composition of existing components over 'swimming against the tide' trying to recreate logic already provided by platform standards.

This shifts the workload from application-level engineering to infrastructure-level composition and ensures that developers can focus on delivering high-value business logic rather than recreating and maintaining infrastructure-level primitives—such as backpressure, retries, circuit breaking, observability, and protocol parsing. Historically, reinventing and maintaining in-house versions of infrastructure primitives has shown to incur significant, long-term technical debt."

---

This issue originates in this discussion item by @andreasDroid in #19

[janhalen]

The research from @andreasDroid and @DanielDroids (read more in this issue for full context: #19 ) identifies a set of challenges that needs to be adressed.

To adress these concerns I recommend that we implement a stateless, configuration-driven ingestion pipeline.

TCP to HTTP bridging

Translation from the TCP protocol of our current gps-devices into a standard HTTP protocol format that plugs effortlessly into the FiWare ecosystem. By moving this component to the edge in a small purpose-built containerized workload, the ingestion layer can be auto-scaled based on device packet frequency without needing to scale the heavier IoT Agent or Context Broker.
So we are looking for an existing Cloud Native abiding data adaptor / stream processing component to help us here, avoiding custom procedural logic. It should be declarative, config driven and play well with the k8s ecosystem.

Handling Nested Objects:

Since a generic iotagent-json seems to be unable to parse nested structures, we need a pre-processor component to flatten complex binary-derived data into flat key-value pairs required by the IoT Agent. This is a standard challenge in datapipeline design and data streaming, so we should look into a purpose built data adaptor to do stream processing that is k8s native and as above is driven by declarative configs avoiding procedural custom code.

Device Authentication

To prevent unknown devices from hitting internal services, the IMEI check should be enforced at the edge. For this a Policy Enforcement Point (PEP) functionality in the data adaptor is needed, validating device identity against a central device auth service. Using exisiting components for this would point to research into using our central KeyCloak instance for device auth based on standardised methods and protocols. And researching a PEP component that can be applied at the edge as an enforcement point. Ideally this PEP should be a part of the streamprocessor component to avoid sprawl and deployment complexity.

The multiprotocol challenge

While some devices support MQTT, our goal should be a Unified Southbound. By using an data adapter layer that can handle both Raw TCP, MQTT as inputs and normalize them before they hit the IoT Agent, we avoid maintaining a custom IoT Agent for every protocol variation.

The genric IoT Agent role

By offloading the raw data transformation to the infrastructure layer, the IoT Agent functions as a pure semantic mapper. This ensures that the Orion context broker receive rich, metadata-annotated NGSI-LD entities without the need to invest in custom procedural code in the ingestion path.

To illustrate an unopinionated solution landscape before we dive in to technical implementation details I have created this diagram:

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#e8f5e9', 'primaryBorderColor': '#81c784', 'lineColor': '#2e7d32', 'tertiaryColor': '#f1f8e9' }}}%%
flowchart TD
    subgraph External [Physical Layer]
        Device([Iot Device])
    end

    subgraph NetworkLayer [Layer 1: Network Infrastructure]
        Gateway[Network Gateway]
        direction LR
        N1[<b>Concern: Connectivity</b><br/>L4 TCP Termination<br/>DoS Protection<br/>Traffic Routing]
    end

    subgraph LogicLayer [Layer 2: Payload Interpretation]
        Adapter[Data Adapter]
        Identity[(Identity Provider)]
        direction LR
        L1[<b>Concern: Data Shape</b><br/>Hex Decoding<br/>Payload Flattening<br/>IMEI Auth Check]
    end

    subgraph SemanticLayer [Layer 3: Semantic Component]
        Manager[IoT Agent]
        direction LR
        S1[<b>Concern: Meaning</b><br/>NGSI-LD Mapping<br/>Metadata Enrichment<br/>Multi-tenancy]
    end

    subgraph StateLayer [FiWare Ecosystem]
        Broker(Context Broker)
    end

    %% Flow
    Device -- "Raw Proprietary Bytes" --> Gateway
    Gateway -- "TCP/MQTT Stream" --> Adapter
    Adapter <--> |"Validate Hardware ID"| Identity
    Adapter -- "Normalized Flat JSON" --> Manager
    Manager -- "Enriched NGSI-LD Entity" --> Broker

    %% Styling
    style Gateway fill:#c8e6c9,stroke:#66bb6a
    style Adapter fill:#c8e6c9,stroke:#66bb6a
    style Manager fill:#a5d6a7,stroke:#4caf50
    style Identity fill:#f1f8e9,stroke:#9ccc65
    style Broker fill:#81c784,stroke:#2e7d32,color:#fff

Loading

[janhalen]

I have a vendor meeting cancelled today, so I'm dedication the time to learning more about potentially using the existing KeyCloak service to pre-register devices (can be done via keycloak UI, custom UI + Keycloak API or declaratively and config driven that are automatd or via manual commits or a custom "git UI" if done manually) This would save a lot of development time, the tradeoffs might be using the clunky KeyCloak UI, but i do prefer declarative device configs anyway. In that way imagine having all device registrations live in a version controlled repo.. sounds robust and perfect for automaton to me!

[andreasDroid]

@janhalen , thanks for diving into this.
Regarding the diagram, I'm concerned about the extra components introduced.
The data adapter, should in our case, still be intelligent enough to parse different IoT signals, hence it does a lot of the heavy parsing, due to the differences in device protocol and payloads.

It points to a validation in the IDP, that would also happen in the IoT agent, since it's required that devices are provisionend and mapped before the IoT picks up the signal.

This will yield more moving parts, when and if the "Data Adapter" is already sketched up to handle both TCP and MQTT, validation and parsing. Essentially making the IoT agent a redundant step, especially when there's no broker between the agent and the adapter.

Excuse if I'm interpreting this wrong, but I'm not sure this will simplify the architecture. Could you clarify what the IoT Agent provides that the Data Adapter couldn't handle directly?
Specifically:

• If the Adapter already parses protocol-specific data into flat JSON
• validates the device against the IDP

What transformation is left for the IoT Agent beyond formatting for NGSI-LD?

[janhalen]

The data adapter / streaming processor handles raw data handling converting it to sane JSON and is flexible and extensible when new devices types and protocols are added. A new device type would be a few lines of code in a declarative (and version controllable) template language.

I am not done researching for a good fit, but it seem like RedHat and Fiware are collaborating on Smart City projects using such a small simple stream-processor.. For an upstream open source component Bento seems very promising. It has a small scalable footprint and rather simple expression language (even i can understand it from looking at it briefly 😄 ). Its fast, purposebuild and written in Go.

Some other processors on the radar that popped up was Vector and MiniNifi.. but havent had the time to dive very deep into them, bento seemed as the most obvious tool by looking at the docs and coming from the Redhat/Fiware collab projects that describe streaming processors in exactly that way.. I have some time-slots next week to continue looking into upstream alternatives..

The iOtAgent handles the "business domain" concerning the specific FiWare NGSI-LD mapping and enriching making it simple and easy to understand for all contributors. But as it is not designed with cloudnative and k8s deployments in mind (the design is a bit old), a lot of custom infrastructure concerns would have to be baked in, making a rather mixed piece of busines logic, data handling, logging and security all in one codebase. That would be an anti-pattern from todays cloud native perspective.

The potential risk scenario list is long, but some cases that spring to mind is:

1. Mass simultaneous reconnection attempts following a network disruption, causing "thundering herd" surges that can overwhelm processing resources.
2. Persistent TCP session saturation where the volume of active device connections exhausts the available file descriptors on the host.
3. Internal network compromise if there is a vulnerability in a protocol parser that is exposed directly to the public internet.
4. Cascading ingestion failure if latency in the Context Broker causes a blocking state in the gateway's ability to accept new device packets.
5. Permanent data loss during service maintenance where a container restart discard active TCP streams.
6. Operational fragmentation when a heterogeneous fleet of devices requires multiple, independently managed agent instances for different firmware versions.
7. Downstream "Data Poisoning" where malformed or corrupted binary packets are persisted into the FiWare database because of a lack of e.g. pre-validation logic.
8. Hardware-level connection timeouts caused by the gateway's inability to acknowledge packets fast enough during periods of high system load.

Offcourse this (and all i forgot) can be handled in procedural code inside the iotAgent, but do we really want to maintain all that ourselves? Furthermore, should it be handled inside the same code as the business logic for creating the NGSI objects? I'm recommending we avoid doing that, and instead find a purpose built upstream component (if it indeed exists, still researching that). From my chair the ideal scenario would be, that the only maintenance needed would be to handle the device protocol and data conversion to an open standard format (a flattened JSON delivered via HTTP) written in a declarative expression language.

Regarding the client auth.. if it is simple to add to the iotagent it could be done there sure, its a question of "refusing devices at the door" or accepting all data into the "business layer" and then procedurally filtering them out or "quarantining" them after the data has entered the system.

So basicly we are "Seperating Concerns" and are following the "Do on thing and do it well" UNIX mantra.

I feel confident that we will make the right choice, fueled by these very relevant discussions..

[andreasDroid]

We for sure want to follow a path where we need to maintain as litte code as possible, but the number of environment components can potentially explode if the sheer goal is to re-use existing components to accomplish a relatively simple goal. For us it's equally important that the system is manageable and simple to its purpose.
I'm not too keen on the idea of having KeyCloak as device registry.

Going a bit deeper, adopting Bento this is how it could look - if possible.

• Group and device provisioning on IoT-agent. Southbound handling multi-tenancy
• Some basic auth or security connection from devices. We need to research how this can be done in a scalable and doable way for both maintainers and users
• Multiple gateway layers in practice to handle different types, L4 and L7
• Bento with device specific mapping. For Ruptela it seems additional Go logic is needed to parse hex data
• Basic IoT json agent, quick research shows ability to handle ~1000 request/second before crashing

Adopting a new IoT device would require:

• Ensuring JSON payload
• Template for provisioning group and device in IoT agent
• Configuring mapping in Bento

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#e8f5e9', 'primaryBorderColor': '#81c784', 'lineColor': '#2e7d32'}}}%%
flowchart TD
    subgraph Devices ["Physical Devices"]
        Ruptela([Ruptela GPS<br/>MQTT-native])
        Teltonika([Teltonika GPS<br/>TCP + JSON])
        FutureHTTP([Future Device<br/>HTTP])
        FutureMQTT([Future Device<br/>MQTT])
    end

    subgraph Gateway ["Gateway Layer"]
        direction TB
        Mosquitto[Mosquitto<br/>MQTT Broker<br/>:8883]
        TCPListener[TCP Listener<br/>:5000]
        HTTPListener[HTTP Endpoint<br/>:8080]
        
        G1[<b>Concern:</b><br/>Protocol Termination<br/>TLS/Connection Management<br/>Basic Rate Limiting]
    end

    subgraph DataLayer ["Data Layer (Bento)"]
        direction TB
        BentoInput[Bento<br/>Input]
        BentoPipeline[Bento Pipeline]
        
        D1[<b>Concern:</b><br/>Flatten Nested JSON<br/>Decode Hex Values<br/>Normalize to Flat JSON<br/>Route by Device Type]
    end

    subgraph SemanticLayer ["Semantic Layer"]
        direction TB
        IoTAgent[IoT Agent<br/>JSON]
        
        S1[<b>Concern:</b><br/>Device Validation<br/>NGSI-LD Mapping<br/>Entity Provisioning<br/>Multi-tenancy]
    end

    subgraph ContextBroker ["FIWARE Ecosystem"]
        Orion[(Orion LD<br/>Context Broker)]
    end

    %% Device to Gateway flows
    Ruptela -- "MQTT<br/>TLS" --> Mosquitto
    Teltonika -- "TCP<br/>JSON" --> TCPListener
    FutureHTTP -- "HTTPS<br/>JSON" --> HTTPListener
    FutureMQTT -- "MQTT<br/>TLS" --> Mosquitto

    %% Gateway to Data Layer
    Mosquitto -- "Subscribe" --> BentoInput
    TCPListener -- "Stream" --> BentoInput
    HTTPListener -- "Request" --> BentoInput
    
    BentoInput --> BentoPipeline

    %% Data Layer to Semantic Layer
    BentoPipeline -- "Flat JSON<br/>HTTP POST" --> IoTAgent

    %% Semantic Layer to Context Broker
    IoTAgent -- "NGSI-LD<br/>Entity" --> Orion

    %% Styling
    style Ruptela fill:#a5d6a7,stroke:#4caf50
    style Teltonika fill:#a5d6a7,stroke:#4caf50
    style FutureHTTP fill:#c8e6c9,stroke:#81c784,stroke-dasharray: 5 5
    style FutureMQTT fill:#c8e6c9,stroke:#81c784,stroke-dasharray: 5 5
    style Mosquitto fill:#bbdefb,stroke:#64b5f6
    style TCPListener fill:#bbdefb,stroke:#64b5f6
    style HTTPListener fill:#bbdefb,stroke:#64b5f6
    style BentoInput fill:#fff9c4,stroke:#ffd54f
    style BentoPipeline fill:#fff9c4,stroke:#ffd54f
    style IoTAgent fill:#ffccbc,stroke:#ff8a65

    style Orion fill:#81c784,stroke:#2e7d32,color:#fff

    

Loading

[janhalen]

I'll be happy to invest a couple of hours here before the weekend sets in to explore if any of the challenges you mention can be handled if we add the Bento stream processor.

Luckily it seems with a quick flick through the docs that both TCP + MQTT and HTTP receivers are built in to Bento .. nice..

MQTT support
TCP
HTTP

It looks like it can be enabled with a few of lines of yaml. (i hope 😉 )

With the bloblang decoding and mapping to a simple flat JSON that the iOtAgent understands should be possible in declarative code too. I will try and explore IF the HEX conversion and mapping challenge you are describing is at all possible in this neat little expression language.
I found that the fork of the project has a nice webbased playground i can mess around in 😄

Routing and dispatching behavoir could be controlled with metadata and handling the separation of the LOGIN and DATA types of packages from say a ruptela device should be a simple match expression

I'll give it a try and get back before the chilled IPA i have ready in the fridge for friday is cracked open.... I promise 😃

[janhalen]

Ok.. I ran out of time and used an LLM to summarize the Ruptela v.1.84 protocol packet byte format docs for me to create a mock test payload.. Not sure if that was a wise choice 😉 - but it got some example code clobbered together in a pretty short time.. the bloblang expression language feels pretty intuitive and functional, which i personally like.

---

# =========================================================
# INPUT SIMULATION (Playground Only - https://warpstreamlabs.github.io/bento/playground/index.html)
# =========================================================
let packet = content().decode("hex")

# =========================================================
# 2. RUPTELA PROTOCOL CONSTANTS
# =========================================================
meta ruptela_32bit_sign_bit = 2.pow(31)
meta ruptela_32bit_range    = 2.pow(32)
meta ruptela_gps_precision  = 10.pow(6)
meta ruptela_volt_precision = 10.pow(3)

# =========================================================
# 3. RUPTELA TYPE INTERPRETER
# =========================================================
map ruptela_hex_interpreter {
  # 1. Input Normalization
  # Place raw bytes into a JSON object so we can attach a 'signed' instruction if the Ruptela negative bit is set.
  let params = match {
    this.type() == "object" => this,
    _                       => { "val": this, "signed": false }
  }

  # 2. Ruptela binary to integer conversion
  # Converts the ruptela raw binary slice into a standard 64-bit integer for math operations without overflow risks.
  let raw_int = ("0x" + $params.val.encode("hex")).uint64()

  # 3. Signed negative integer interpretation
  # Checks if the Ruptella sign bit is set signifying that the input is an int32 that can contain negative values.
  root = match {
    $params.signed && $raw_int >= meta("ruptela_32bit_sign_bit").number() => $raw_int - meta("ruptela_32bit_range").number(),
    _                                                                     => $raw_int
  }
}

# =========================================================
# 4. DATATYPE NORMALIZATION
# =========================================================
# Mappers to apply standard data types and scaling to the binary data
map to_coords   { root = { "val": this, "signed": true }.apply("ruptela_hex_interpreter") / meta("ruptela_gps_precision").number() }
map to_unsigned { root = this.apply("ruptela_hex_interpreter") }
map to_datetime { root = this.apply("ruptela_hex_interpreter").ts_strftime("%Y-%m-%d %H:%M:%S", "UTC") }
map to_volts    { root = this.apply("ruptela_hex_interpreter") / meta("ruptela_volt_precision").number() }
map to_boolean  { root = this.apply("ruptela_hex_interpreter") > 0 }

# =========================================================
# 5. DATA FIELD ASSIGNMENT (Execution)
# =========================================================
# Extracts data chunks from the binary data based on 
# the Ruptela protocol definitions and assigns them to named fields.
root.device_id   = $packet.slice(2, 11).encode("hex")
root.timestamp   = $packet.slice(12, 16).apply("to_datetime")
root.lat         = $packet.slice(17, 21).apply("to_coords")
root.lng         = $packet.slice(21, 25).apply("to_coords")
root.speed       = $packet.slice(30, 32).apply("to_unsigned")
root.ignition_on = $packet.slice(36, 37).apply("to_boolean")
root.ext_battery = $packet.slice(39, 41).apply("to_volts")
root.int_battery = $packet.slice(42, 44).apply("to_volts")

---

28 lines of declarative expressions to parse the binary into a flat JSON with normalised datatypes.
(excluding comments and test data setup)

Injectiong a mock binary data packet input of

`002C11223344556677880301698FC2F00103589B70009B0E30000A005A09003C0003010101021D30D41E0F3CABCD

it outputs this when running in the warpstreamlabs playground:

{
  "device_id": "112233445566778803",
  "ext_battery": 12.5,
  "ignition_on": true,
  "int_battery": 3.9,
  "lat": 56.138608,
  "lng": 10.161712,
  "speed": 60,
  "timestamp": "2026-02-14 00:33:52"
}

Tried to mock a vehicle travelling 60kmt somewhere in Aarhus

[andreasDroid]

Alright, that's cool @janhalen.

When you say “ruptela packet byte format”
Just to be clear, the example you posted is parsing a raw binary Ruptela packet. That’s not the nested JSON payload we’re currently receiving over MQTT.

Next step on our side: we’ll verify whether we can configure Ruptela to send the same raw packet format directly over TCP to our listener, and if so, whether we can reliably decode it with this approach.

If that works, it would simplify the architecture with no public MQTT endpoint needed for Ruptela and fewer gateway concerns.

[janhalen]

Updated the comment above with the docs I could find (on scribd)

If we can get the data in JSON ist EVEN better and a LOT less code needed BUT, the tradeoff offcouse is maintaining a complete queue system instead of a single go-binary like Bento/Benthos. I prefer the latter, but lets continue the research to get the best fit. I havent dug into the MQTT support in Bento/Benthos but that might be a possible hybrif approach to recieve JSON directly in Bento..

If I can finde some teltonika docs somewhere, I can try writing a similar bento mapper example for those devices, that is if i can find a timeslot before I leave for FOSDEM 2026 later next week.

[andreasDroid]

Alright, thanks. We'll take a look at that.

Well, that's kinda the whole reason this MQTT stuff kicked off, because we wanted readable json from the Ruptela.
As you mention, I think we need to weigh if it's worth it to have an additional entrypoint, if we can handle all of it through parsing in bento.

We already have some successful parsing with Teltonika through TCP, so this should also be doable in bento. Will update this issue on Monday, when I've had opportunity to discuss this with @DanielDroids.

[DanielDroids]

I have taken a real-world drive to collect actual production data from both GPS devices, so we are no longer relying on mock data that is always in a “perfect” state or limited to a single record.

The purpose of this is twofold:

To clearly demonstrate the practical differences between the two devices

To ensure that we can extract real, usable data regardless of which implementation path we choose

Below is a snapshot of the raw data received from each device during the same drive.

Ruptela Plug5

The Plug5 sends binary, multi-record payloads over TCP. A single transmission can contain multiple concatenated records, making parsing and acknowledgment non-trivial:
b'\x03\xda\x00\x03\x10Z\x8f\xc1\xc5\xa1D\x01\rix\xcd\x08\x00\x00\x00\x07\x80q\xeb!-S\xed\x00\x00P\x14\x00\x00\x00\x00\x00\x07\x04\x00\xfb\x01\x00\x1c\x01\x00\xad\x00\x01\xa2\x01\x04\x00\x1d\x00\x00\x00\x1e\x0eQ\x00\x16\x00\x03\x00\x17\x00\x01\x03\x00A\x00\x00\x11\x9a\x00\x96\x00\x00\\xfe\x04\xd1\x00\x00\x00\x00\x00ix\xcd\r\x00\x00\x00\x07\x80q\xeb!-S\xed\x00\x00P\x14\x00\x00\x00\x00\x00\x07\x04\x00\xfb\x01\x00\x1c\x01\x00\xad\x00\x01\xa2\x01\x04\x00\x1d\x00\x00\x00\x1e\x0eQ\x00\x16\x00\x03\x00\x17\x00\x01\x03\x00A\x00\x00\x11\x9a\x00\x96\x00\x00\\xfe\x04\xd1\x00\x00\x00\x00\x00ix\xcd\x13\x00\x00\x00\x07\x80q\xeb!-S\xed\x00\x00P\x14\x00\x00\x00\x00\x00\x07\x04\x00\xfb\x01\x00\x1c\x01\x00\xad\x00\x01\xa2\x01\x04\x00\x1d\x00\x00\x00\x1e\x0ea\x00\x16\x00\x03\x00\x17\x00\x01\x03\x00A\x00\x00\x11\x9a\x00\x96\x00\x00\\xfe\x04\xd1\x00\x00\x00\x00\x00ix\xcd\x18\x00\x00\x00\x07\x80q\xeb!-S\xed\x00\x00P\x14\x00\x00\x00\x00\x00\x07\x04\x00\xfb\x01\x00\x1c\x01\x00\xad\x01\x01\xa2\x01\x04\x00\x1d\x00\x00\x00\x1e\x0ea\x00\x16\x00\x02\x00\x17\x00\x02\x03\x00A\x00\x00\x11\x9a\x00\x96\x00\x00\\xfe\x04\xd1\x00\x00\x00\x00\x00ix\xcd\x1e\x00\x00\x00\x07\x80q\xeb!-S\xed\x00\x00P\x14\x00\x00\x00\x00\x00\x07\x04\x00\xfb\x01\x00\x1c\x01\x00\xad\x01\x01\xa2\x01\x04\x00\x1d\x00\x00\x00\x1e\x0ea\x00\x16\x00\x02\x00\x17\x00\x02\x03\x00A\x00\x00\x11\x9a\x00\x96\x00\x00\\xfe\x04\xd1\x00\x00\x00\x00\x00ix\xcd#\x00\x00\x00\x07\x80q\xeb!-S\xed\x00\x00P\x14\x00\x00\x00\x00\x00\x07\x04\x00\xfb\x01\x00\x1c\x01\x00\xad\x01\x01\xa2\x01\x04\x00\x1d\x00\x00\x00\x1e\x0ea\x00\x16\x00\x02\x00\x17\x00\x02\x03\x00A\x00\x00\x11\x9a\x00\x96\x00\x00\\xfe\x04\xd1\x00\x00\x00\x00\x00ix\xcd)\x00\x00\x00\x07\x80q\xeb!-S\xed\x00\x00P\x14\x00\x00\x00\x00\x00\x07\x04\x00\xfb\x01\x00\x1c\x01\x00\xad\x01\x01\xa2\x01\x04\x00\x1d\x00\x00\x00\x1e\x0e_\x00\x16\x00\x02\x00\x17\x00\x02\x03\x00A\x00\x00\x11\x9a\x00\x96\x00\x00\\xfe\x04\xd1\x00\x00\x00\x00\x00ix\xcd.\x00\x00\x00\x07\x80q\xeb!-S\xed\x00\x00P\x14\x00\x00\x00\x00\x00\x07\x04\x00\xfb\x01\x00\x1c\x01\x00\xad\x01\x01\xa2\x01\x04\x00\x1d\x00\x00\x00\x1e\x0e_\x00\x16\x00\x02\x00\x17\x00\x01\x03\x00A\x00\x00\x11\x9a\x00\x96\x00\x00\\xfe\x04\xd1\x00\x00\x00\x00\x00ix\xcd4\x00\x00\x00\x07\x80q\xeb!-S\xed\x00\x00P\x14\x00\x00\x00\x00\x00\x07\x04\x00\xfb\x01\x00\x1c\x01\x00\xad\x01\x01\xa2\x01\x04\x00\x1d\x00\x00\x00\x1e\x0eT\x00\x16\x00\x03\x00\x17\x00\x03\x03\x00A\x00\x00\x11\x9a\x00\x96\x00\x00\\xfe\x04\xd1\x00\x00\x00\x00\x00ix\xcd9\x00\x00\x00\x07\x80q\xeb!-S\xed\x00\x00P\x14\x00\x00\x00\x00\x00\x07\x04\x00\xfb\x01\x00\x1c\x01\x00\xad\x01\x01\xa2\x01\x04\x00\x1d\x00\x00\x00\x1e\x0eZ\x00\x16\x00\x05\x00\x17\x00\x03\x03\x00A\x00\x00\x11\x9a\x00\x96\x00\x00\\xfe\x04\xd1\x00\x00\x00\x00\x00ix\xcd?\x00\x00\x00\x07\x80q\xeb!-S\xed\x00\x00P\x14\x00\x00\x00\x00\x00\x07\x04\x00\xfb\x01\x00\x1c\x01\x00\xad\x01\x01\xa2\x01\x04\x00\x1d\x00\x00\x00\x1e\x0e\\x00\x16\x00\x03\x00\x17\x00\x03\x03\x00A\x00\x00\x11\x9a\x00\x96\x00\x00\\xfe\x04\xd1\x00\x00\x00\x00\x00ix\xcdD\x00\x00\x00\x07\x80q\xeb!-S\xed\x00\x00P\x14\x00\x00\x00\x00\x00\x07\x04\x00\xfb\x01\x00\x1c\x01\x00\xad\x01\x01\xa2\x01\x04\x00\x1d\x00\x00\x00\x1e\x0eK\x00\x16\x00\x04\x00\x17\x00\x02\x03\x00A\x00\x00\x11\x9a\x00\x96\x00\x00\\xfe\x04\xd1\x00\x00\x00\x00\x00ix\xcdJ\x00\x00\x00\x07\x80q\xeb!-S\xed\x00\x00P\x14\x00\x00\x00\x00\x00\x07\x04\x00\xfb\x01\x00\x1c\x01\x00\xad\x01\x01\xa2\x01\x04\x00\x1d\x00\x00\x00\x1e\x0eJ\x00\x16\x00\x03\x00\x17\x00\x02\x03\x00A\x00\x00\x11\x9a\x00\x96\x00\x00\\xfe\x04\xd1\x00\x00\x00\x00\x00\xf02'

This represents real-world traffic, not an idealized single-record example.

Teltonika

The Teltonika device sends JSON-formatted AVL data, one record at a time, which is immediately human-readable and easier to validate:
Connected by ('127.0.0.1', 41595)
Received: b'\x00\x0f864636060329170'
IMEI received -> sending ACK
Received: b'{"state":{"reported":{"ts":1769522869000,"pr":0,"latlng":"55.662523,12.585605","alt":0,"ang":152,"sat":17,"sp":10,"evt":0,"239":1,"240":1,"21":5,"200":0,"69":1,"181":12,"182":6,"66":14247,"67":0,"68":0,"241":23801,"16":564776}}}'
AVL data received -> sending record ACK
Received: b'{"state":{"reported":{"ts":1769522870000,"pr":0,"latlng":"55.662510,12.585630","alt":0,"ang":127,"sat":17,"sp":9,"evt":0,"239":1,"240":1,"21":5,"200":0,"69":1,"181":9,"182":6,"66":14262,"67":0,"68":0,"241":23801,"16":564776}}}'
AVL data received -> sending record ACK
Received: b'{"state":{"reported":{"ts":1769522871000,"pr":0,"latlng":"55.662503,12.585657","alt":0,"ang":104,"sat":17,"sp":8,"evt":0,"239":1,"240":1,"21":5,"200":0,"69":1,"181":12,"182":6,"66":14245,"67":0,"68":0,"241":23801,"16":564780}}}'
AVL data received -> sending record ACK

Each message corresponds to a single record and is acknowledged individually.

Going forward, this real-world data should be used as the reference when validating parsing logic, acknowledgments, and architectural decisions. This avoids false positives that can occur when testing only against synthetic or overly simplified input.

Using real traffic ensures that any conclusions we draw are grounded in conditions that actually occur in production.

[andreasDroid]

Here is updated payload from the Ruptela and Teltonika respectively through mqtt:

Ruptela sends to topic ruptela/862406075073953/data
{"ts":1769594217,"trigger":8,"prio":0,"imei":"862406075073953","ext":0,"pos":{"lat":556613783,"lon":125840983,"alt":77,"dir":19250,"spd":46,"sat":15,"hdop":7},"data":{"251":"1","28":"1","173":"1","418":"1","29":"3755","30":"E87","22":"4D6","23":"3","65":"1276","150":"5CFE","1233":"0"}}

Teltonika sends to topic teltonika/864636060329170/data
{"state":{"reported":{"ts":1769610760000,"pr":0,"latlng":"0.000000,0.000000","alt":0,"ang":0,"sat":0,"sp":0,"evt":240,"239":1,"240":0,"21":3,"200":0,"69":2,"181":0,"182":0,"66":20597,"67":0,"68":0,"241":23801,"16":563688}}}

I imagine a flow like this, without any concerns right now towards the device authorisation at the edge:

flowchart TB
    subgraph Registration ["Device Registration (One-time)"]
        Frontend[Frontend/Admin]
        Frontend -->|"1. POST /iot/services<br/>Fiware-Service: {municipality}<br/>apikey: {device_type}"| IoTAgent1[IoT Agent]
        Frontend -->|"2. POST /iot/devices<br/>Fiware-Service: {municipality}<br/>device_id: {imei}"| IoTAgent1
    end

    subgraph DataFlow ["Data Flow (Continuous)"]
        GPS[GPS Device]
        Mosquitto[Mosquitto MQTT Broker]
        Bento[Bento Stream Processor]
        IoTAgent2[IoT Agent]
        Orion[(Orion-LD Context Broker)]
        
        GPS -->|"MQTT publish<br/>topic: {device_type}/{imei}/data<br/>payload: device-specific JSON"| Mosquitto
        Mosquitto -->|"Subscribe: ruptela/+/data<br/>Subscribe: teltonika/+/data"| Bento
        Bento -->|"1. Extract IMEI from topic<br/>2. Normalize payload<br/>3. POST /iot/json?i={imei}&k={device_type}"| IoTAgent2
        IoTAgent2 -->|"Lookup device → tenant<br/>Forward with Fiware-Service header"| Orion
    end

    style Frontend fill:#e1f5fe
    style GPS fill:#c8e6c9
    style Mosquitto fill:#bbdefb
    style Bento fill:#fff9c4
    style IoTAgent1 fill:#ffccbc
    style IoTAgent2 fill:#ffccbc
    style Orion fill:#c8e6c9

Loading

sequenceDiagram
    participant Admin as Frontend/Admin
    participant IoTA as IoT Agent
    participant GPS as GPS Device
    participant MQTT as Mosquitto
    participant Bento as Bento
    participant Orion as Orion-LD

    Note over Admin,IoTA: Device Registration (one-time per device)
    
    Admin->>IoTA: POST /iot/services<br/>Fiware-Service: naestved<br/>{"apikey": "ruptela", "resource": "/iot/json"}
    IoTA-->>Admin: 201 Created
    
    Admin->>IoTA: POST /iot/devices<br/>Fiware-Service: naestved<br/>{"device_id": "123456789012345", ...}
    IoTA-->>Admin: 201 Created

    Note over GPS,Orion: Data Flow (continuous)
    
    GPS->>MQTT: PUBLISH ruptela/123456789012345/data<br/>{"lat": 55.123, "lon": 12.456, "spd": 45}
    MQTT->>Bento: Message on ruptela/+/data
    
    Note over Bento: Extract IMEI from topic<br/>Normalize payload
    
    Bento->>IoTA: POST /iot/json?i=123456789012345&k=ruptela<br/>{"lat": 55.123, "lon": 12.456, "spd": 45}
    
    Note over IoTA: Lookup: device_id → tenant<br/>Found: naestved
    
    IoTA->>Orion: PATCH /ngsi-ld/v1/entities/urn:ngsi-ld:GPSTracker:123456789012345<br/>Fiware-Service: naestved
    Orion-->>IoTA: 204 No Content
    IoTA-->>Bento: 200 OK

Loading

[janhalen]

Thanks @DanielDroids for some valid real world data snippets. I agree that that is the right way to move along!
I'm leaving for FOSDEM 2026 (maybe next year you will be able to attend too?) so im a bit out of timeslots for the gps-connector, but I skimmed your ideas and it touches on important points for moving into some tests and eventually a PoC. Catching potential gaps in our knowledge and discussing and planning thoroughly is the recipe for success in production readiness! I have reserved some time next week for going a bit deeper, so i hope you have the patience to continue this dialogue then.. Adressing @andreasDroid s valid concerns of the number of moving parts will be part of diving deeper into the potential architecture. Talk to you again next week!

[janhalen]

@andreasDroid, @DanielDroids, @Lyneborg: As we discussed today:

IF we set the requirements for devices to be

• Support at least TLS version 1.3
• Support delivering data in (flat) JSON
• Support data delivery via the MQTT protocol (should also be versioned)

Then a lot og the more hairy conversion stuff would be eliminated... Lets discuss if we want to do that..

This might even help us reduce the number of moving parts, potentially looking at NanoMQ as a replacement for Mosquitto and Bento.. But lets test if the NanoMQ rule engine can deliver what we need first. I have to admit it looks a bit basic from skimming the docs: https://nanomq.io/docs/en/latest/rule/config-file.html, at least compared to bento/benthos

Furthermore, a secure entrypoint (like Cillium) into the cluster in front of the queue should be discussed..

Thanks for a fast, effective meeting!