Support REST Api for sending configs and commands to devices (Device Communication API)

[gdimitropoulos-sotec]

This is another issue in order to qualify Hono as a replacement for Google IoT Core customers. In Google Iot Core clients can send commands and configs to a device through a REST API. Therefore we would like to add another REST API component to Hono as a replacement, named “Hono Device Communication API”. The new component will cover only communication via the MQTT adapter and will not communicate directly with the devices, but only with the command router.

Workflow

Client sends HTTP command/config requests to the Hono Device Communication API, api sends commands/configs to the command router which forwards them to the device.

Config Data

Config data does not need a specific schema and could be any binary data that a device understands. Configuration data will be stored with config version history.
Config Request
To update device configurations, the client should send a post request to the API and define fields versionToUpdate and binaryData in the request body.

POST https://API_URL/v1/device/config/{tenant-id}/{device-id}

Request body

Schema fields:

{
  "versionToUpdate":  string (int64 format),
  "binaryData": string (bytes format)
}

versionToUpdate field

Every device config has a version id (greater than 0). By updating device configs clients can update a specific config version or if the versionToUpdate field is set to 0, then the current version of configs will be always updated. Config update will fail if requested version does not exist.

binaryData

Any binary data that a specific device can understand.

Response body

If successful, api will return an DeviceConfig.

For more information see Google's IoT API documentation.

DeviceConfig DB

DeviceConfig objects will be saved in a Database so that a client can request them via the API. Database will be also used by MQTT adapter to make sure that all the desired configs have been delivered to the devices and also that all the devices have acknowledged their configs.

Keeping DeviceConfig sync

MQTT adapter after a configuration version was successfully updated, will publish the updated config version with retain set to true and so every device after a new start will always receive the updated current configurations to use.

[sophokles73]

I have taken a look at your component diagram and I have a few questions/comments.
In general, the Command Router utilizes the messaging infrastructure (Kafka) for two purposes: a) receiving the commands from the applications and b) forwarding the command to the protocol adapter instance that the (target) device is connected to.

My understanding is that you do not want to use Kafka for submitting the command messages from applications to the Command Router. However, you will still need the messaging infrastructure in order to route a command to the (MQTT) protocol adapter instance. For that purpose, the protocol adapter instances maintain a (protocol adapter instance scoped) consumer to the messaging infrastructure and the Command Router puts commands to these topics. I can imagine using Pub/Sub for this purpose as well in order to get Kafka out of the picture altogether.

However, the Command Router has no API endpoint for submitting commands from applications. So I wonder how you envision your new API component to forward commands from applications to the Command Router?

[gdimitropoulos-sotec]

However, the Command Router has no API endpoint for submitting commands from applications. So I wonder how you envision your new API component to forward commands from applications to the Command Router?

To make our new component similar to the existing Hono components, API will communicate with the Router via AMQP (internal communication).

[sophokles73]

The Command Router (currently) does not provide an AMQP 1.0 based API endpoint for submitting commands for devices. Instead, it opens a consumer to the messaging infrastructure. If you implement such an AMQP 1.0 (server) endpoint in the Command Router that accepts command messages, then your REST API component could connect to that endpoint and submit commands for devices. However, the Command Router will still need to use messaging infrastructure in order to route the commands to the appropriate protocol adapter instance. So, either you keep the Kafka messaging infrastructure in place for this, or you adapt the Command Router (and protocol adapters) to support using Google Pub/Sub for routing the commands. However, in the latter case, you would probably better let the Command Router consume the commands from the applications via the Pub/Sub infra as well, instead of wiring the REST API component to the Command Router directly ...

[julian-sotec]

If you implement such an AMQP 1.0 (server) endpoint in the Command Router that accepts command messages, then your REST API component could connect to that endpoint and submit commands for devices

That is exacly what we are planning to do. And we are aware that we still need to use the messaging infrastructure (kafka) to deliver these commands to the adapters. We are planning submit another issue soon where we plan to add an option (a thrid option beside kafka and amqp) for the messaging infrastructure to be Google Cloud Pub Sub. However this (for now) is not planned to be part of this issue. Hope this helps to clarify which option we prefere

[sophokles73]

@julian-sotec I see. However, as pointed out earlier, if you plan to fully replace Kafka with Pub/Sub, then I do not see much value in implementing the AMQP 1.0 endpoint in the Command Router. Instead, you would (/should) allow the Command Router to consume commands from Pub/Sub and let your REST API component publish commands to Pub/Sub. Analogously to how the system currently works for Kafka and AMQP 1.0 based messaging infrastructure ...

[julian-sotec]

FMPOV the value would be that client applications do not necessarily need a pub/sub / kafka client anymore. Instead you could use standard rest apis (like with device tenant registration) to control / interact with your devices and data.

[sophokles73]

Sure, but that would still be the case with your new REST API component, wouldn't it? Of course, the REST API component would need to be implemented to publish commands from applications to the messaging infrastructure (Pub/Sub) instead of connecting to the Command Router directly ...

[julian-sotec]

When this new REST API component does not interact with to the command router directly - wouldn't it be more like an (hono external) Application providing a Rest API for Command / Configs using kafka / pubsub clients? We were discussing this and AFAWK a compnent like this would/should not be part of hono - because one could argument that such application are customer specific and part of the solution tier only (refering to the architecture picture)

[sophokles73]

Even better then, right? In that case you wouldn't even need to contribute it to Hono :-) However, some years ago, when we decided to add support for Kafka, we actually had discussed putting an API endpoint in front of the messaging infrastructure as well. If adding the REST API component is necessary to mimic the standard Google IoT Core functionality for the applications, then we could also decide to add it to the Hono code base or the hono-contrib repository. In any case, the question of where to contribute it should not determine the architectural decision that we need to make here.

[julian-sotec]

Makes sense to me. So from an architecural point of view this issue will cover these components:

• We will implement a communication API component defined as stated above
• We will implement a pub/sub publisher that publishes commands / configs to pub/sub messaging infrastructure only for now
• Adjustments to the command router to support the pub/sub messaging infrastructure

What we will not cover in this issue:

• Adjustments for the communication API that implements integration for kafka messaging infrastrucure (Also another optional issue later)

Decision wheter this will be contributed for core hono or hono-contrib to be done when PR is opened

[sophokles73]

sounds good to me 👍

[julian-sotec]

@sophokles73 we have moved this issue to hono-extras repository - thus this issue can be closed here. See Support REST Api for sending configs and commands to devices
Who could help us with the PR 19 there?

[sophokles73]

Great! Maybe @calohmn can jump in here, he is the expert when it comes to command & control.