MQTT.js vNext Architecture Feedback

[YoDaMa]

Hey all, @YoDaMa here. I'm creating this discussion and migrating some of my writing from the existing vNext PR that I created to the Github Discussions since this seems like a much more apt medium for community engagement.

As part of the ongoing effort to revive / rearchitect this project, I've been working on creating an easily digestible UML-ish diagram of the existing architecture, as well as a new architecture. The goal is these will make it easier for community members with not so much experience in the repository or time to dive deep into all the code to contribute. Architecture can sometimes be hard to see when you're reading through the codebase and looking at all the weird red-herrings of the existing design.

Here is a class diagram of the existing MQTT.js Library's code, from the perspective of a User Application:

The existing library's code is structured around what I'll call a Connection Factory. If you look in lib/connect/index.js you will see the main entry point of the client generation is through this file, where connection is chosen based on the protocol of the parsed URL given during the call to connect (mqtt.connect(<myUrl>)).

We can model the connection factory like so:

I've left out a few of the existing "ConnectionStream implementations" as I'll call them. You can see in the existing repo that there are:

• ali.js
• tcp.js
• tls.js
• ws.js
• wx.js

Of all of these, two are Alibaba specific implementations, to my knowledge: wx.js and ali.js. I am confused as to why a company specific set of factories has been added to this repository, and don't see a compelling reason to keep them unless there's some proxy support that is useful for all users of the MQTT.js library, regardless of service they're connecting to. (I work for Microsoft, but it doesn't make sense to build Microsoft specific connection layers into this library either.)

With all that being covered, it's now a community dialogue about where we go. How do people like the existing API? Any changes people would like to see? And how this library can be written in a way so it is more testable, and ultimately more maintainable by those without historical 'tribal' knowledge of how different pieces of it's functionality came to be?

I have anticipated that I will be the main code writer on this rewrite project, which is fine. But I don't want to make drastic changes that everyone doesn't know about or dislikes. So your feedback and engagement is crucial.

[BertKleewein]

This is only scratching the surface, but I have a few ideas here:

1. I don't immediately see an easy way to "inject a ConnectionStream" or "Make an MqttClient from a ConnectionStream". It seems like ali.js and wx.js might not have been necessary to add to this repo if there was a way for the user to instantiate their ConnectionStream and make a client. This would also help with:

   • testing. You could create a TLS stream and wrap it with a recorder component that you could then play back with a different platbayk stream. You could also have handcrafted streams for testing.

   • stream customization. I don't know if node supports different TLS implementations, but Python has two different SSL libraries and there are advantages to picking "the uncommon one". I notice that tls.js passes an options struct into the Tls initializer, but I'm not sure if this gives us the right level of customization. I'm thinking about customers who want fine-grain control over TLS. e.g. Only accept certain TLS versions, cyphers, key lengths, etc.

2. client.js packs too much functionality into the MqttClient. This could benefit from being broken up. A few ideas:

   • Converting between bytestream and packetstream feels like it belongs in its own object. client.js should only talk with a packetstream. Also, this packet stream should be injectable, so you can record packets, play them back, inject handcrafted packet streams, etc.

   • I'm not 100% sure how this would fit in, but connection lifetime feels like it belongs in it's own object. "what to do when the client disconnects" is a sufficiently complex and nuanced issue that it should be separated out. Likewise with the entire connection state machine. Moving between "disconnected->connecting->connected->reauthorizing->disconnecting->disconnected" states needs to be centralized. Again, make these objects injectable and independently testable.

   • Subscription management feels like it could be pulled out. So does QOS code and code that matches MIDs for PUBACK and SUBACKs and calls the correct "on_puback" code. Again, independently testable. I want to be able to inject packets into a QoS-handler object and make sure QoS 0,1,& 2 are done correctly without any other components in the way.

3. I don't remember if MQTT5 supports multiplexing. If it does, that will affect things (hopefully only at the bytestream level).

4. I might even go so far as to make separate objects for each of:

   • connect->connack
   • maybe something separate for reauthentication
   • publish->puback w/ pubcomp+pubrel
   • subscribe->suback
   • pingreq->pingack
   • disconnect packets

   Maybe with an object who's sole purpose is to take packets off the packetstream and decide which object to send them to (connack packets go the connect->connack object, etc).

[dsonet]

wx and ali are for specific mobile runtime environment.

[seriousme]

While creating Opifex (a MQTT server in Typescript on Deno, only the start of a client now) I did exactly that:

1. take a socket conn
2. turn in into an MQTT conn
3. run a for-await-of loop on top of the MQTT conn which takes packets of the stream and feeds them to specific handlers

In my test scripts I just pass a dummy socket conn, e.g: see mqttConn.test.ts

Hope this helps :-)

[BertKleewein]

And, for what it's worth, I've been neck-deep in rhea and there might be something to be learned from looking at their architecture. I don't think mqttjs needs something quite as complex, but the way they break up transport, terminus, session, container, connection, endpoint, and frames is interesting.

[YoDaMa]

There have been some discussions around the use of options within the client instantiation process, as well as the design of a ConnectionFactory which creates an abstract interface for potentially allowing users to pass in their own custom connection layer via the options object.

The ConnectionFactory interface would look like this:

interface ConnectionFactory {
  getConnectionStream: function (options: ConnectOptions): Duplex
}

and this would then be passed into via options like:

const options = {
  brokerUrl: 'myurl.com'
  ...
  customConnectionFactory: {
    getConnectionStream(options: ConnectOptions): Duplex {
      ...
      return myCustomStream
    }
  }
}

[oceanlvr]

I'm a heavy user of mqtt.js, and there's a lot of attention on the next step in the design of mqtt.js. For the previous mqtt.js its data flow pattern is really complicated, and I think refactoring is really necessary.

• Packetstream is more appropriate to use rxjs to abstract data flow.
• State machines such as XState describe the current state of the mqtt client.
• Suitable lifecycle abstraction, allowing users to register custom plugins in the corresponding cycle

[YoDaMa]

thanks for sharing feedback @oceanlvr ^.^ follow along on the vNext_dev branch to see how our development progresses. Will be good to have your eyes on our development work so we can bring up the best possible package.

[BertKleewein]

@oceanlvr - thanks for this. I have some followup questions.

1. I don't know much about packetstream and rxjs. Why do you say it/them are more appropriate? I'm not fully versed, but from what I've seen, the stream objects in Node are somewhat old & crufty and filled with historical baggage.

2. I've used state machines (specifically machina) with some success, but there are traps here along with the benefits. Plus, supply chain attacks are making me more leery of taking additional dependencies. What specifically about XState makes you like it for this scenario? I'm guessing it has something to do with your third suggestion. What sort of life cycle/states/transitions do you think might be useful? How would you use these in your code?

[oceanlvr]

Thanks for your reply, I reviewed the previous questions, probably a bit presumptuous.

1. For 1. The background is that I need to build a desktop application.js mqtt, but I found that when my connection is too large, the node subscription callback will always take up a lots of memory, I need to manually batch process these received messages, and since mqtt messages are subscribe-to-publish mode, I think maybe rxjs might fit into this abstraction pattern so that developers can use batch processing in these scenarios more concisely and customally

So I looked at whether I could use rxjs to process data streams in a similar way so that the response events of the data stream could be more clearly abstracted, especially when the corresponding events were batched, which could reduce the response pressure of the UI.

See that there is already a discussion of observables in your documentation ( Observable support part ). How do you think about this part, and will there be a supportive plan later?

[oceanlvr]

2. For 2. I may have thought that MQTT.js could develop several lifecycle functions for developers to use in a subscription-multicast manner, so that users can develop related plugins and extend the user-defined functions of mqtt.js through subscription-multicast

Thank you.

[iamgabrielsoft]

Could we improve parallel messages on the next version

[BertKleewein]

@iamgabrielsoft - could you be more specific? What about the current version don't you like? @vishnureddy17 and I have been talking about this lately and I want to make sure we're not missing something here.

[modellking]

This is less about how the library is structured, but more about its Interface. Perspective of a light user.
The instantiation process seems weird to us: Create a Client with .connect() then add listeners for 'connect' and 'error'/'close'. Just the naming seems to imply raceconditions (we do not rely on this, but use it for logging).
Why not something like:

mqtt.createClient([options])
  .on('connect', ()=>{}) // setup listeners
  .connect([options])

[seriousme]

In my view the current version is way to heavy on dependencies.
I would appreciate a more lightweight version were complexity can be plugged by those that really need it.
E.g. on an espruino with little RAM you don't need f́ancy stuff. In other use cases people might want more features.

ESM would also help in treeshaking.

My 2cts ;-)

[vishnureddy17]

We are moving our development work for the next generation of MQTT.js to a separate repo. I'll close this discussion to avoid confusion and keep everything in that repo. If you have any thoughts or feedback on vNext, please open an issue in that repo.