Skip to content

MTM-64962 Rework MQTT Service overview for GA #3852

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: mqtt-service-ga-integration
Choose a base branch
from
Open

MTM-64962 Rework MQTT Service overview for GA #3852

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
99e1dd7
MTM-64962 initial rework of the introduction
scmi-c8y Oct 24, 2025
31aedfe
MTM-64962 commit work in progress
scmi-c8y Oct 24, 2025
17169ca
MTM-64962 new architecture diagram
scmi-c8y Oct 24, 2025
2c75520
MTM-64962 first draft for review
scmi-c8y Oct 27, 2025
341832f
Apply suggestions from code review
scmi-c8y Oct 28, 2025
5ecd905
MTM-64962 add a disclaimer to the preview block
scmi-c8y Oct 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 64 additions & 30 deletions content/device-integration/mqtt-service-bundle/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,43 +4,77 @@ layout: redirect
title: Overview
---

### Architecture {#architecture}
The MQTT Service works together with the Messaging Service to quickly and securely integrate MQTT devices with the {{< product-c8y-iot >}} platform.
Devices that already understand the {{< product-c8y-iot >}} domain model can use the [Core MQTT](/device-integration/mqtt/) protocols (SmartREST and JSON-over-MQTT) to communicate directly with {{< product-c8y-iot >}}.
Alternatively, devices can send and receive messages with arbitrary payloads on arbitrary MQTT topics.
For these so-called _generic_ devices, a _mapping_ must be provided by the tenant to convert between the device message format and the {{< product-c8y-iot >}} domain model.
Copy link
Contributor

@rthrippleton rthrippleton Nov 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might just be be, but seeing "a mapping must be provided" conjures up an image of configuration, especially if I've been thinking of Core MQTT. Not quite sure how, but I want to see "you must write CODE you poor soul" before I get that impression.

A mapping can be implemented by a microservice running inside the platform, or by an external client application.

_IoT device integration_ is the main intended use case for the MQTT Service.
The design of the service is optimised for this use case, which has some highly asymmetric properties:
* A large number (up to tens of millions) of simultaneously connected devices publishing messages into the IoT platform
Copy link
Contributor

@rthrippleton rthrippleton Nov 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"IoT platform" is odd wording - surely "Cumulocity platform" as we've been using everywhere else in the doc?

* A large number (up to tens of millions) of unique MQTT topics
* A high aggregate throughput (up to millions per second) of unique messages published into the IoT platform
* A small number of high-throughput message consumers within the IoT platform
* Individual devices have a smaller message throughput (up to hundreds per second)
* Smaller aggregate message throughput from the IoT platform to devices (up to thousands per second)
* No direct peer-to-peer device communication
* Endpoint security for widely distributed devices can be challenging
Copy link
Contributor

@rthrippleton rthrippleton Nov 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really know what this means, can you clarify? At first glance it looks like "our security is hard to configure".


Other, more symmetric _server-to-server_ or _application integration_ use cases may exceed the [limits](/service-terms/quotas/#mqtt-service) enforced by the MQTT Service.
For optimal performance, these use cases should be implemented using a more traditional publish/subscribe architecture with direct connections to the Messaging Service.

### Key features {#key-features}

The MQTT Service works together with the Messaging Service to provide a framework for highly customizable and flexible MQTT message processing solutions.
The diagram below illustrates how a message flows, starting from the device, through the Messaging Service,
then to a user-provided microservice where it is converted to the {{< product-c8y-iot >}} JSON format and
delivered to {{< product-c8y-iot >}} using the standard REST API.
#### Protocols and clients {#protocols-clients}

![MQTT Service send](/images/mqtt-service/mqtt-service-send.svg)
| | |
|---------------------------------------------|----------------------------------------------------------------------------------------------------------------------|
| Connection protocols | TCP only. |
| MQTT protocol versions | 3.1, 3.11 and 5.0. See [MQTT protocol implementation](#implementation) for more details. |
| Generic MQTT device protocols | MQTT devices can publish and subscribe arbitrary payloads on arbitrary MQTT topics. |
| {{< product-c8y-iot >}} Core MQTT protocols | SmartREST 1.0, SmartREST 2.0, JSON-over-MQTT are supported.<sup>(1)</sup> |
Copy link
Contributor

@rthrippleton rthrippleton Nov 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the (1) footnote is really burying an important lede here - don't get my hopes up when "are in experimental support" is just as easy.

| Apache Pulsar | Microservices and external clients [connect directly to the Messaging Service](#pulsar-client) to map between device protocols and the {{< product-c8y-iot >}} domain model.<sup>(2)</sup> |
| Thin Edge | Out of the box support in [Thin Edge](https://thin-edge.io/) for generic and Core MQTT protocols on the same device. |

All MQTT messages published to the MQTT Service are forwarded to the Messaging Service, where they are persisted, waiting to be consumed.
A custom microservice or [Streaming Analytics app](/streaming-analytics/epl-apps/#using-cumulocity-mqtt-service) that understands the topic and payload structure can consume the MQTT messages, and then translate and push them into {{< product-c8y-iot >}}.
Notes:
1. Core MQTT protocol support is in preview and should be considered experimental.
2. Client applications are also responsible for registering devices as {{< product-c8y-iot >}} Managed Objects if required.
<br><br>

Similarly, a custom microservice or Streaming Analytics app can send messages to devices, as shown in the diagram below.
In this case, the user-provided microservice receives messages from {{< product-c8y-iot >}} through a Notifications 2.0 subscription.
These messages are mapped to the payload structure used by the MQTT devices, then published to MQTT topics.
#### Security and isolation {#security-isolation}

![MQTT Service push](/images/mqtt-service/mqtt-service-push.svg)
| | |
|--------------------|-------------------------------------------------------------------------------------------------------------------------------------|
| Multi tenancy | MQTT devices from multiple tenants connect to the same endpoint. Each tenant's connections, topics and messages are fully isolated. |
| Device isolation | MQTT devices within a tenant cannot communicate directly with each other. Each device effectively has its own private topic space. |
| Client access | Microservices and external applications connecting to the Messaging Service have access to all topics and messages within a tenant. |
| Bi-directional TLS | Certificate trust anchors are managed within {{< product-c8y-iot >}}. Certificates are tightly bound to individual MQTT devices. |

As with MQTT messages published by devices, messages published from a microservice will be forwarded to the Messaging Service, where they can be consumed by MQTT devices subscribed to the relevant topics.
#### Performance and scaling {#performance-scaling}

| | |
|--------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| Horizontal scaling | The MQTT Service can be scaled independently from the {{< product-c8y-iot >}} core. |
| Benchmark results | TBC |
| Limits and quotas | Per-tenant and per-client [limits and quotas](/service-terms/quotas/#mqtt-service) ensure service stability and prevent "noisy neighbour" problems. |


### Architecture {#architecture}

Custom microservices may use the [Java client](/device-integration/mqtt-service#java-client) to publish to or consume from MQTT topics. They can use the [Microservice SDK](/microservice-sdk/java) to push data into {{< product-c8y-iot >}}.
The diagram below illustrates the MQTT Service data flows within a tenant.

### MQTT Service compared to Core MQTT {#mqtt-service-vs-cumulocity-iot-mqtt}
<p align="center" width="100%">
<img width="80%" src="/images/mqtt-service/mqtt-service-architecture.svg" alt="MQTT Service architecture">
</p>

The table below presents a basic comparison between the {{< product-c8y-iot >}} Core MQTT functionality and that of the MQTT Service.
All messages published by MQTT devices are forwarded to the Messaging Service, where they are persisted until they are consumed.
{{< product-c8y-iot >}} domain model messages published to the MQTT topics used by the Core MQTT protocols are consumed directly by the {{< product-c8y-iot >}} core.
Messages published to other MQTT topics are consumed by microservices and/or external clients that are responsible for mapping the messages to the {{< product-c8y-iot >}} domain model.
Similarly, the {{< product-c8y-iot >}} core and clients can publish messages to the Messaging Service that will be consumed by the MQTT Service and forwarded to devices.

| | Core MQTT | MQTT Service |
|:-----------------------------|:--------------------------------------------------------|:------------------------------------------------------------------------------------|
| QoS | 0, 1, 2 | 0, 1 |
| Clean session | Starting with clean session is recommended | Starting with clean session is required |
| Retained flag | Not supported | Not supported |
| Last will | Supported | Supported |
| MQTT 5.0 features | Not supported | MQTT 5.0 clients can connect. Partial support for MQTT 5.0 features |
| Authentication | Basic and TLS device certificates | Basic and TLS device certificates |
| Scalability | Horizontal | Currently a single instance. Horizontal scaling will be available in the GA release |
| Topic format | Determined by the SmartREST 2.0 protocol | Unrestricted. SmartREST topic names are reserved and cannot currently be used |
| Payload | Determined by the SmartREST 2.0 protocol | Unrestricted. The maximum message size is 128 KiB including all headers |
| Extensibility | Limited by SmartREST 2.0 custom templates | Streaming Analytics apps or custom mapping microservices can support arbitrary MQTT-based protocols |
| Message processors/consumers | Built-in message processor for each SmartREST 2.0 topic | Streaming Analytics apps or custom mapping microservices can support multiple processors for a topic |
| JSON via MQTT | Limited feature set | Streaming Analytics apps or custom mapping microservices can support arbitrary JSON payloads |
_Device isolation_ means there is no interaction between topics with the same name used by different clients.
Effectively, every device has its own private topic space that can only be accessed by that device.
This can be seen in the diagram where _device 1_ and _device N_ are both publishing and subscribing on _topic A_.
Because devices are isolated, _device 1_ cannot see any of the messages published by _device N_, and vice-versa.
However, a microservice or external application client has full access to the topics used by all devices, and can forward messages between clients if required.
40 changes: 16 additions & 24 deletions content/device-integration/mqtt-service.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,32 +12,24 @@ That is, it is not yet generally available and may be subject to change in the f
{{< /c8y-admon-preview >}}

{{< c8y-admon-req >}}
To work with the MQTT Service, the following requirements must be met:
* The {{< product-c8y-iot >}} Messaging Service must be deployed in your {{< product-c8y-iot >}} environment.
* The {{< product-c8y-iot >}} MQTT Service must be deployed in your {{< product-c8y-iot >}} environment.
* Your tenant must be subscribed to the mqtt-service microservice.
This may have been done automatically, depending on how your {{< product-c8y-iot >}} environment was configured.
To check the subscription, open the Administration application and navigate to **Ecosystem** > **Microservices**.
If you do not see the mqtt-service microservice listed, contact [product support](/additional-resources/contacting-support/) (for public environments) or your {{< product-c8y-iot >}} administrator (for dedicated environments) to request the subscription for your tenant.
To use the MQTT Service your tenant must be subscribed to the `Mqtt-service` microservice.
This may have been done automatically, depending on how your {{< product-c8y-iot >}} environment was configured.
To check the subscription, open the Administration application and navigate to **Ecosystem** > **Microservices**.
If you do not see the `Mqtt-service` microservice listed, contact [product support](/additional-resources/contacting-support/) (for public environments) or your {{< product-c8y-iot >}} administrator (for dedicated environments) to request the subscription for your tenant.
{{< /c8y-admon-req >}}

The MQTT Service is a new MQTT endpoint implementation for {{< product-c8y-iot >}} that provides the following benefits:
The MQTT Service provides a single, unified endpoint for integrating MQTT devices with the {{< product-c8y-iot >}} platform.
It is not a general-purpose MQTT broker.
It is optimized for scenarios where there are a large number of connected devices, and a high aggregate throughput of messages from devices into the platform.

* Sending and receiving arbitrary payloads on any MQTT topic.
Note that the topics used by the {{< product-c8y-iot >}} [Core MQTT](/device-integration/mqtt/) implementation currently cannot be used with the MQTT Service.
* User-provided microservices can send and receive messages on MQTT topics, and map messages to and from the {{< product-c8y-iot >}} data model.
The typical use case for such a microservice is to map between MQTT device payloads, and the {{< product-c8y-iot >}} REST and Notifications 2.0 APIs.
* Multi-tenancy support.
A single endpoint serves multiple tenants and tenants are completely isolated from each other.
* Bi-directional TLS support.
All MQTT traffic is encrypted and clients can authenticate using X.509 certificates.
This documentation is aimed at developers who want to integrate MQTT devices with {{< product-c8y-iot >}}, or to build clients that communications with their devices.
It does not describe the basics of MQTT communication.
If you are unfamiliar with MQTT, there are numerous introductions available, starting with the <a href="https://mqtt.org/mqtt-specification/" target="_blank">MQTT website</a>.

The MQTT Service does not replace the existing [Core MQTT](/device-integration/mqtt/) capability of {{< product-c8y-iot >}} that supports sending device data already in the {{< product-c8y-iot >}} domain model directly into the platform.
The new capability provided by the MQTT Service allows for easier integration of MQTT devices that cannot use the {{< product-c8y-iot >}} domain model.
It also supports more flexible communication patterns between devices, applications, and the {{< product-c8y-iot >}} platform, controlled by user-provided microservices.
The documentation is structured as follows:
* [Overview](#overview) presents the key features and high-level architecture of the service.
* [MQTT protocol implementation](#implementation) discusses the supported MQTT protocol versions and features.
* [Connecting MQTT devices](#connecting-devices) shows how to integrate devices with the service, using either the Core MQTT protocols or generic MQTT messaging.
* [Connecting microservice and applications](#pulsar-client) explains how to develop clients that communicate with generic MQTT devices.
Comment on lines +33 to +36
Copy link
Contributor

@rthrippleton rthrippleton Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Begs the question - because we're having to put in a navigation 'guide' here, doesn't that mean our section headings could be better? Not everybody reads doc linearly, so if somebody can locate the relevant section just from the tree on the left, that'd be better.

My modest and hastily written suggestions:

  • Key features and high-level architecture
  • Supported MQTT versions and features
  • Integrating MQTT devices with the service
  • Developing clients to communicate with MQTT devices


**Device isolation** is the default behaviour for the MQTT Service. Each MQTT client has its own private topic space and cannot directly receive messages from other clients, enhancing security and isolation between devices.

This documentation does not describe the basics of MQTT communication.
If you are unfamiliar with MQTT, we recommend you to consult one of the numerous introductions on the internet.
Some references can be found on the <a href="https://mqtt.org/mqtt-specification/" target="_blank">MQTT website</a>.
Finally, see [service quotas](/service-terms/quotas/#mqtt-service) for details of the limits and quotas enforced by the MQTT Service.
Loading