[GSoC Project Proposal]: Add MQTT support to ERDDAP™

[ChrisJohnNOAA]

Project Description

Add support for MQTT to ERDDAP™.
MQTT is a message protocol which was in particular designed to be lightweight and used by remote/IoT devices.

The project can be done in stages, with the first stage to implement the ability for ERDDAP™ to be a MQTT client. This would include connecting to an existing broker and building datasets based on events from them.

The next stage would be integrating an MQTT broker into ERDDAP™.

For both MQTT client and MQTT broker there are a variety of libraries available, so this project is mostly about integrating the selected library(ies) into ERDDAP™ and properly handling the provided data.

Expected Outcomes

MQTT support in ERDDAP™ that allows datasets to be defined based on subscribing to MQTT events. This will enable new datasets to be included in ERDDAP™ as well as enable near real-time data to be made available. MQTT Broker support would simplify the process of a new project being able to collect data from MQTT enabled sensors.

Skills Required

Java

Additional Background/Issues

ERDDAP/erddap#37

Mentor(s)

Chris John (@ChrisJohnNOAA)

Mentor Contact Email(s)

[email protected]

Expected Project Size

350 hours

Project Difficulty

Intermediate

[div-dev123]

Hey @ChrisJohnNOAA
I am excited about the opportunity to work on adding MQTT support to ERDDAP. I have a strong background in Java and experience working with data integration, networking protocols, and real-time data processing.I understand that this project involves enabling ERDDAP to act as an MQTT client, allowing it to subscribe to events from an existing broker and build datasets based on incoming data. The next step would be integrating an MQTT broker directly into ERDDAP, making it easier to collect data from IoT sensors. This project aligns well with my skills, and I look forward to contributing !

[ChrisJohnNOAA]

@div-dev123 I generally recommend doing a small issue in ERDDAP. You can see the issues listed here: https://github.com/ERDDAP/erddap/issues

The goal is for me to get a better sense of your abilities and for you to get a sense of what working in the ERDDAP code would be like.

One issue that would introduce you to code of one of our current dataset types is this one: ERDDAP/erddap#38
Though feel free to look through the list and find another issue that interests you.

You can use Jetty for local development: https://github.com/ERDDAP/erddap/tree/main/development/jetty
It also would likely be helpful to at least read through our programmer contributor guide:https://erddap.github.io/docs/contributing/programmer-guide

[div-dev123]

Okay thanks a lot ! I'll look through it!

[ayushsingh01042003]

Hi @ChrisJohnNOAA had some questions about this:

• Description mentions ERDDAP being able to integrate both as client and broker libraries. I understand adding the client so ERDDAP can subscribe to a MQTT broker but why does ERDDAP need to integrate broker libraries and act as a broker when it can just subscribe to an existing one? Is it going to be data transfer between two ERDDAP servers or ERDDAP subscribing to a broker on which data through IOT stuff is published or both? This is a little confusing
• What exactly is meant when we say it should be cross platform? what exactly should we keep in mind so that we keep it cross platform?

[ChrisJohnNOAA]

Hi @ayushsingh01042003 , answers inline below.

Hi @ChrisJohnNOAA had some questions about this:

• Description mentions ERDDAP being able to integrate both as client and broker libraries. I understand adding the client so ERDDAP can subscribe to a MQTT broker but why does ERDDAP need to integrate broker libraries and act as a broker when it can just subscribe to an existing one? Is it going to be data transfer between two ERDDAP servers or ERDDAP subscribing to a broker on which data through IOT stuff is published or both? This is a little confusing

The main reason to have ERDDAP able to be a broker is to enable creating datasets from remote sensors without the need for running another server. Somebody with an existing MQTT setup would already have a broker, but ERDDAP as a broker would enable somebody to get started collecting remote sensing data (from an MQTT client) without needing to start running yet another server. For a user trying to create datasets from MQTT sensors, ERDDAP being a broker would allow them to have one server that is responsible for collecting the data and turning it into a dataset,

I think there are possibilities for other features to be built on top of this integration. For example being able to subscribe to dataset changes as MQTT messages (instead of emails or webhooks) which could enable new integrations.

• What exactly is meant when we say it should be cross platform? what exactly should we keep in mind so that we keep it cross platform?

I believe the intent of cross-platform there is working on all (common) operating systems (Windows, Linux, MacOS).

[ayushsingh01042003]

Some more question:

• So mqtt broker libraries to allow ERDDAP to collect data directly from sensors eliminating the need for a broker server in the middle and client libraries to allow ERDDAP to connect to an existing broker and subscribe to it for data. Did i get that correct?

• Should ERDDAP also be able to publish data to a broker?

• The discussion mentioned using EDDTableFromHttpGet as a way to fetch the real time data. So do you think we should build on it for real time data fetching or create a new dataset type like EDDTableFromMQTT for this? Any thoughts?

• Any opinions about using libraries like moquette that provide both client and broker APIs over using different libraries for both client and broker features?

• Should we worry about things like implementing auth mechanisms for both the scenarios client and broker scenarios?

Thanks!

[ChrisJohnNOAA]

@ayushsingh01042003 answers inline below.

Some more question:

• So mqtt broker libraries to allow ERDDAP to collect data directly from sensors eliminating the need for a broker server in the middle and client libraries to allow ERDDAP to connect to an existing broker and subscribe to it for data. Did i get that correct?

Yes.

• Should ERDDAP also be able to publish data to a broker?

In the long term, yes. That said I think it's a stretch goal for the GSoC project. The priorities for the GSoC project are 1) being able to subscribe to events from a broker to contribute to a dataset and 2) being able to act as a broker to directly connect to sensors. Publishing dataset updates as MQTT events would be a great feature, however I think the way ERDDAP currently updates datasets that will be a significant undertaking and not just because of the MQTT side of things.

• The discussion mentioned using EDDTableFromHttpGet as a way to fetch the real time data. So do you think we should build on it for real time data fetching or create a new dataset type like EDDTableFromMQTT for this? Any thoughts?

I don't know at this time. I can see how they might be similar, but it's also possible that to take full advantage of MQTT there will be changes needed that necessitate a new dataset type.

• Any opinions about using libraries like moquette that provide both client and broker APIs over using different libraries for both client and broker features?

I have not done an evaluation of potential libraries yet. If there is one that is open source with a compatible license and has a good implementation of both broker and client that sounds good.

• Should we worry about things like implementing auth mechanisms for both the scenarios client and broker scenarios?

Yes, auth should be implemented.

Thanks!