Distributed PubSub for Microcontrollers

Thanks, @RyanMac!

Hi Carles,

Great to hear that you enjoyed the talk and Q&A session, and thank you for the kind words. If you dive into things more and have any questions, please don't hesitate to ask them here or email me.

Best regards,
-- Matt

Hi @RaulPando!

I am not that familiar with CANbus, but I just read csselectronics.com and Wikipedia. It seems that CANbus is just a robust many-to-many protocol, where all nodes receive all messages. According to Higher Layer Protocols by Kvaser, the data format is assigned by the application. You could certainly create a PubSub layer over CANbus, if one doesn't already exist. However, check out UAVCAN and the C implementation libcanard.

Before I stumbled upon UAVCAN, I wrote this: Since you only have 8 bytes of payload, you probably want to be more efficient than topic strings. For example, you could construct topic integers where you assign a certain number of bits for your topic hierarchy. Alternatively, you could create some type of dynamic global registration with one node assigning new integers to publishers based upon topic metadata. You could then have the remaining payload for integer data. You may also need to reserve some bits to allow for segmentation & reassembly if you want to send larger data over CANbus frames.

The Fitterbap PubSub implementation is definitely not optimized for CAN, but it looks like UAVCAN is. What do you think?

Best regards,
-- Matt

Hi Remco,

Thanks! If you find something that does not seem like the right tradeoff, let me know! I'd love to discuss it.

1. I didn't spend much time discussing network retransmission approaches in the talk, but there are three very common methods:
   a. STOP & WAIT
   b. GO BACK N
   c. SELECTIVE REPEAT

STOP & WAIT only has a single frame outstanding at a time. If the sender gets a NACK or times out, it retransmits. The problem is throughput as you only have the one frame outstanding and have to wait for the processing delay plus the ACK delay. The upside is that you only need a single frame buffer on both the transmit and receive side.

GO BACK N fixes the throughput issue in the normal case. Instead of one outstanding frame, you can have N. The upside is the we can utilize the full bandwidth, assuming that N is large enough to account for the processing and 2 x transmit delay. The buffer requirements grow to N on the transmit side and 1 on the receive side. However, on error, the transmitter has to repeat every frame since the errored frame. While we (meaning I) would like errors to be normally (Gaussian) distributed, in reality, we often have an underlying normal distribution of errors with some additional errors that "group". The "grouping" is usually due to other events that increase the error probability, such as ESD. So, GO BACK N has a much higher chance of data loss than the normally distributed error assumption often leads us to believe.

With Selective Repeat, the sender only repeats errored frames. The receiver records all correct frames and plays them out in order when possible. This gives the best throughput and error resiliancy at the expense of increased buffering. Both the transmitter and receiver must have an N frame buffer.

Fitterbap implements (c), selective repeat. So, it only needs to repeat the errored frames.

For further research, you can start with Wikipedia.

2. Yes, fitterbap includes timeouts, which are necessary for selective repeat to work reliably. Fitterbap allows you to configure the timeout for your data link. The longer the timeout, the more effect a lost packet has on your system and the more buffering you need to keep full throughput. Fitterbap is designed to be quick since I am considering the case of UARTs, SPI and I2C where propagation times are short and we want to keep the buffers small. My reference implementation has a 16 millisecond timeout.

Did I answer your questions? Any additional questions?

Best regards,
-- Matt

Hi @tcmichals,

Thanks for sharing! If I understand correctly, nanopb is a microcontroller-friendly version of Google protocol buffers. I have a few questions regarding how you are using it:

1. How are you communicating between your microcontroller and the desktop computer? For example, USB-CDC? USB-CDC uses USB bulk mode which provides reliable USB frame delivery. As long as you don't drop USB frames, you don't need the error detection and retransmission features in the fitterbap datalink.

2. You mentioned routing. Did you actually mean network-style routing, or do you really have a one-to-one message queue in each direction, where the messages are Google protocol buffers? If you mean network-style routing, could you elaborate?

3. How reliable has your setup been? Would you recommend it?

Thanks for checking out the talk, and look forward to hearing more about your experiences!

Best regards,
-- Matt

Thanks Harrison, and great to hear that you enjoyed the talk! Feel free to contact me if you have questions!

2. Yes. To change the state of the those modules, you would need to continue to talk to them from other parts of the system. This creates a dependency and new interactions that reduces the benefit of the initial dependency injection.

3. If you have a finite state machine (FSM) controlling your system, then the current FSM state is the global state, at least with respect to what it's controlling. As you mention, if you need to communicate that state to other parts of the system or other parts of the system need to behave differently based upon the current FSM state, then you have shared global state. You can easily use PubSub to carry the events to your FSM. The FSM can also publish it's state to keep other parts of the system in sync. You probably would not want other parts of the system publishing the FSM state, though, so you would have to enforce that by design.

Does this answer your questions? Any follow up questions?

Hi @nathanchalesjones!

The Fitterbap project does not currently have a list of modules that are available with their maturity. The project does use doxygen, but I have not published it. Something else for the list! For now, you can peruse the include directory. With the exception of pubsbu and the communication that were in the talk, everything else is in production code for products I have built.

1. This is a tough one to answer. I know at least one functional programming convert that would say it immediately by design! In practice, shared state tends to generate code smells like tightly coupled code, code that is difficult to test, intertwined dependencies, and that piece of code that no one wants to touch. As an consultant, I took on extending a C/C++ project where the entire codebases was effectively a single giant object. It was actually multiple objects, but they all had public members that other objects modify at will. It was a mess. My solution was to encapsulate the mess (the customer did not want to pay to fix it!) and extend around it. Last I heard, the customer was very happy with the new code, but unable to successfully make changes to that old code. Shared state has a way of getting out of hand over time, so any choices we can make to keep it under control during the initial design is valuable. I personally think PubSub is one option.

Thanks @EEngstrom77! I like the logo visual image!

I have used the ESD generator with USB, both high-speed and super-speed, which use differential signals. With USB, common-mode is also a huge problem, so the ESD generator works great for injecting errors. You don't have to hit your signals that hard with ESD to create bit errors. A discharge nearby is often enough, which many devices can handle without special ESD protection. If you have an expensive target board, you may only want to use this approach on signals that have ESD protection.

Great to hear that you enjoyed the talk, Matthew! While this PubSub implementation is not going to fit on the small ATtiny's, it is definitely suitable for many microcontrollers including Cortex-M family parts.

Hi Erwin.

1. PubSub at its simplest is a design pattern that you can implement within a single program or single microcontroller. Networked PubSub protocols, like MQTT, and PubSub brokers allow publishers & subscribers over a network. The distributed PubSub that I am talking about is a simplified version of network PubSub that works both within a single microcontroller and across multiple microcontrollers.

2. At the API of the data-link layer, you send a message and it is transferred to the microcontroller on the other side of the connection. The PubSub port packs & unpacks data-link layer messages that contain the topic and value. The data-link does not currently have a keep-alive, but it does use timeouts. This means that it can determine loss of connection, but only when it has something to say. I have considered adding an optional keep-alive for more guaranteed loss of connection detection.

3. Topics are visible up the hierarchy, but not down. Metadata is shared whenever a publisher posts $ or my/topic/$. Metadata is shared just like any other PubSub message, but with the reserved $ topic suffix.

4. Yes, state is repopulated. Check out this explanation.

5. C will continue operating, and potentially publish new topics. Normal non-retained messages will be passed to subscribers local to C, but A will never know. Retained messages will also be passed to subscribers local to C. When C establishes connection with B, then it will update B with the retained values. Likewise, A's retained values will be passed to B. If the retained values on both A and C changed while B was down, then the value will resolve, but the winner is not well-defined with the existing implementation. It could be the value from either A or C depending upon the order in which the connections come up.

PubSub itself is relatively simple. Anyone can publish a message. Subscribers receive the messages for which they subscribed. That's really about it conceptually! Does the figure on slide 12 make sense to you?

Now, implementation details vary greatly. You can check out Wikipedia. There are lots of variations, especially for how the PubSub instance decides how to forward which messages to which subscribers.

I hope to chat live during the Q&A!
-- Matt

Hi Steve,
The top-level of the topic must match the "owner" PubSub instance, which is typically one per microcontroller. The remaining topic hierarchy is up to you. I typically think of the hierarchy in terms of matching my module architecture, but it doesn't have to. In the case of retained values, the last part of the hierarchical name is a feature, which that module exposes so that other parts of the system can control it. That feature can do anything, including writing hardware registers, changing software behavior, and starting a waveform stream. Basically, you are free to assign the topics and meaning of values to fit your system.

The top-level topic allows some limited routing. As you mentioned this is not a typical network stack, but it is not that inefficient either. Let's take the case of the PubSub instances A, B, and C, where A is the top of the tree, B is in the middle and C is at the bottom. A module connected to PubSub instance C publishes C/enable = 1. The PubSub instance C forwards that to all matching subscribers, including the link subscriber for B. PubSub instance B receives the message and forwards it to all matching subscribers except the publisher, the PubSub instance C's link subscriber. One subscriber is the link subscriber for A. PubSub instance A receives the message and forwards it to all subscribers except the publisher. Any and all subscribers have now received the message.

Now a module connected to PubSub instance A can issue C/enable = 0. PubSub B subscribers to PubSub A with both top-level topics B & C. Upon publish, A forwards the message to all matching subscribers, including PubSub B. Likewise, B forwards the message to all matching subscribers, which includes PubSub C.

Now, let's say that a module connected to PubSub C publishes C/enable = 1, but there are no external subscribers at B or A. What happens? Well, the message is still forwarded to B and the only subscribers is A. A receives the message and discards it because it has no other matching subscribers. This is somewhat wasteful of bandwidth. We could architect the system so that C would know that it's parents have no subscribers. However, this requires more coordinatation and RAM, so the existing implementation just forwards the message regardless. We do have the "_" topic, which is never forwarded. This is a very simple way to keep more frequent communications within a single microcontroller from taking up distribution bandwidth unnecessarily.

Yes, the PubSub approach allows for many-to-many communications. By design, many-to-many allows multiple subscribers to receive the same message. However, it can also support one-to-one communication if a topic has one publisher and one subscriber. You do this by adding a topic with one and only one publisher and one and only one subscriber. However, if you need two subscribers at a later date, the PubSub approach makes this trivial.

If you have two modules that always communicate, especially at high bandwidth, using a one-to-one message-passing queue is a great solution. Nothing about this stops you from doing that. If you want to communicate directly over the data-link layer without pubsub, the extra unassigned ports are perfect for this. You saw a very brief example with the waveform.

Does this help? The live Q&A is tomorrow (Mon) at 10:30 am EDT. I'd love to chat more and answer any questions!

Best regards,
-- Matt

Hi Miro,
Distributed PubSub is definitely a great fit for sending events (messages) around your system! It may be a good way for QP to encompass a larger system of microcontrollers.

I definitely agree that I need to create an example project. I have been thinking of targeting Raspberry Pi Pico since no one can argue with the price. I started, but have not had enough time to really dive in.
Would the Pico (RP2040) work for you?

Best regards,

• Matt

Thank you, Sergio. Looking forward to your talk, too!

Hi Miro,
Thanks for attempting to try out pyfitterbap! The project definitely needed the finishing touches for python packaging. I just pushed an updated version 0.3.1 to GitHub that will make things much easier. You can now follow the normal python install process using pypi:
pip3 install -U pyfitterbap
You can then run the Fitterbap Communications UI:
fitterbap comm_ui
or
python -m pyfitterbap comm_ui
If you would prefer to run from the source, you do need to build the native code using Cython, which is a little more work to configure on Windows. First, you need Visual C++ or MinGW. Once the tools are installed, you can build pyfitterbap. Here are the instructions to run in place:
cd c:\path\to\fitterbap
git pull
pip3 install -U -r requirements.txt
python setup.py build_ext --inplace
python -m pyfitterbap comm_ui

Does this work for you?
-- Matt

The fitterbap GitHub repo is now available:
https://github.com/jetperch/fitterbap

Hi @esaias and thanks for the question. The fitterbap GitHub repo is not yet public, but it will be before the conference. While the code is already being used, I want to take another pass through the API, which gets much harder to change after making it public.