Tutorials:
Sensors, interfaces and bus systems (SENIN, BUSSY)

MQTT

last updated: 2021-01-30

Introduction

Song of this chapter: Rogue Wave > Descended Like Vultures > Publish My Love

MQTT is a lightweight publish-subscribe messaging protocol designed to be used with IoT devices (wiki).

MQTT stands for Message Queue Telemetry Transport and one of the main development objectives was to communicate from machine-to-machine (M2M) but also connect M2M to clouds (Big Data).

As HTTP, MQTT runs on top of Transmission Control Protocol / Internet Protocol (TCP/IP) stack. The most recent implementation is MQTT v5.0 (official OASIS standard), based on the earlier v3.1.1 standard. Our software sill uses the v3.1.1 standard, so we stick to v3.1.1.


MQTT OSI

Publishing and subscribing

Like e.g. ROS (Robot Operating System) MQTT uses a publish-subscribe pattern (pub-sub pattern). This is a messaging pattern where messages are not sent directly from transmitter to receiver (point to point), but are distributed by an MQTT server (formerly called MQTT broker).

The MQTT server is the centerpiece of an pub-sub architecture. It can be implemented very simply on an single board computer like the Raspberry Pi or a NAS but naturally also on mainframes or internet server. The server distributes messages and so has to be a publisher, but is never a subscriber!

Clients can publish messages (sender), subscribe messages (receiver) or both. A client (also called node) is an intelligent device like a microcontroller, or computer with a TCP/IP stack and software implementing the MQTT protocol.

A publishing client (called publisher) is only loosely coupled to a subscribing client. It does not even know if there are other clients. Same for a receiving client (subscriber) that is interested in a specific types of messages and will subscribe to these messages.

The messages are published under a topic allowing filtering. Topics are hierarchical divided UTF-8 strings. The different topic levels are separated by a slash /.

Let's take a look at the following setup. The photovoltaic power plant is publisher. The main topic level is "PV". The plant publishes two sub-levels "sunshine" and "data".

"PV/sunshine" is a bool value (yes/no, could also be 1/0) and is needed by the charging station to know if the electric vehicle should be loaded (only when the sun shines :)). The charging station (EVSE) is a subscriber and subscribes to "PV/sunshine" to get the information from the server.

"PV/data" on the other hand transports the momentary power generated by the plant in kW and that topic could e.g. be subscribed by computers or tablets to generate diagrams of the delivered power over a day.


MQTT publish subscribe

"Just do it" MQTT1:
"Just do it" MQTT2:


MQTT.fx

Topics and message filtering

The topic is are hierarchical divided UTF-8 string. The different topic levels are separated by a slash /, the topic level separator. There is no need to inform the broker over the topics, every syntactically right string is valid. Some examples of topics:

    myhome/firstFloor/Kitchen/temperature
    myhome/firstFloor/Kitchen/humidity
    Luxembourg/Luxembourg/Jofferchersgaass/4/heating/onOff
    6133/49609/btsiot/sc4

The first topic has 4 levels. Topics are case-sensitive, so firstfloor is not the same as firstFloor. Empty spaces are permitted but not very useful as such strings are prone to errors. Each topic must contain at least 1 character, but the forward slash alone is a valid topic. Topics should not beginn with $. The $-symbol topics are reserved for internal statistics of the MQTT broker, even if there is no official standardization. Commonly $SYS/ is used for all the following information as explained in the MQTT GitHub wiki.

It is important to maintain consistency in topic names and we should be aware that we can subscribe to multiple topics by using topic filters. Therefore, it is very important to choose the right topic names for a project.

Wildcards

To filter topics when subscribing, two wildcards can be used, the single-level wildcard: + or the multi-level wildcard: #.

Single-level wildcard +

For all temperatures on the first floor of myhome we could use:

    myhome/firstFloor/+/temperature

For all temperatures in myhome we could:

    myhome/+/+/temperature
Multi-level wildcard #

The the multi-level wildcard must be placed as the last character in the topic and preceded by a forward slash.

For all sensors placed in the Kitchen (first floor) of myhome we could use:

    myhome/firstFloor/Kitchen/#

And for all sensors from the first floor:

    myhome/firstFloor/#

Quality of Service levels

The quality of service is an important feature of MQTT. As we use TCP/IP the connections are already secured at a certain level. But in wireless networks interruptions and disturbances are frequent and MQTT helps here to avoid the loss of information with its quality of service levels. These levels are used when publishing. If a client publishes to the MQTT server, the client will be the sender and the MQTT server the receiver. When the MQTT server publishes messages the client, the server is the sender and the client is the receiver.

QoS 0: fire and forget

This is the normal quality of TCP/IP. There is no acknowledgement from the server and the message is not stored.

MQTT QoS 0

QoS 1: at least once delivery

QoS 1 promises that the message will be delivered at least once to the subscriber. The message has gets a message Id. The sender stores the message until it receives an acknowledgement (PUBACK) from the subscriber. The first time the sender sets the duplicate flag to 0 (DUP = 0). QoS 1 tells the destination device (receiver) that a confirmation requirement is necessary. If the message was received correctly the receiver sends a PUBACK (acknowledgement) with the right message Id. However if a certain time is exceeded without confirmation, the message is sent again, this time with the duplicate flag set to 1 (DUP = 1). This can happen more than once to the same destination device. This destination device must have the necessary logic to detect duplicates and react likewise.

MQTT QoS 1

QoS 2: exactly once delivery

With QoS 2 we have a warranty that the message is delivered only once to the destination. For this the message with its unique message Id is stored twice, first from the sender and later by the receiver. QoS level 2 has the highest overhead in the network because two flows between the sender and the receiver are needed. After the first sending (DUP = 0) the sender repeats the sending ((DUP = 1) until it receives a PUBREC that tells, that the message was stored by the receiver. With the second transaction the sender tells the receiver to stop the transmission with a PUBREL (release), clear the buffer used for the storage and send a PUBCOMP (complete). A published QoS 2 message is successfully delivered after the sender is sure that it has been successfully received once by the destination device.

MQTT QoS 2

Local MQTT server on a Raspberry Pi

Using free server in the Internet is simple but also a security risk that is not worse to take if we want to use MQTT e.g. locally in buildings. We will activate a server on a Raspberry Pi (Raspi) single board computer.

Burn the image and enable ssh

For this download the newest Raspian OS image (.zip) from https://www.raspberrypi.org/downloads/raspbian/ (lite version), unzip it and and flash the image on an SD card. More infos can be found e.g. on rapberrytips.com.

For security reasons, ssh (secured access over network) is no longer enabled by default on the Raspi, so our headless Raspi can't be accessed. To enable ssh we need to place an empty file named ssh (no extension) in the boot directory. So we insert the SD card in a computer. We can access boot easily because it has a FAT32 file system and create an empty file with the name ssh in the root folder.

Restart the Raspberry Pi and connect it to your Ethernet. For security reasons we will not use WiFi. To find the IP address of the Raspi we can use the nmap command (if not installed, install it (linux: sudo apt install nmap)):

    sudo nmap -sP 192.168.1.*

Look for the IP address of your Raspi (e.g. 192.168.1.125) and log in with ssh (putty on windows):

    sudo ssh pi@192.168.1.125

Now log in with the standard user pi and the standard password raspberry (with a German keyboard the password is raspberrz :)).

Change password and do an upgrade

As every one knows the Raspi password let's change it immediately with the passwd command. First you will be prompted for your old password, then you new password twice.

To get the latest versions of all programs use the following commands:

    sudo apt update
    sudo apt upgrade
    sudo apt dist-upgrade

It is also good idea to install the terminal file manager "midnight commander (mc) to search and edit files as root in terminal (sudo mc) and htop to look at processes.

    sudo apt install mc htop

Add a static IP address

It is simpler to know Raspi's IP address, so we set it static. Call the midnight commander sudo mc and open the file dhcpcd.conf in the /etc folder. Press F4 to open the nano text editor and uncomment the following lines in /etc/dhcpcd.conf. Change the IP addresses to your needs.

    interface eth0
    static ip_address=192.168.1.100/24
    static routers=192.168.1.1

Save the file with Ctrl-O, exit with Ctrl-X , exit mcwith F10 and reboot (sudo reboot). Now we can log in with the new IP.

Install the MQTT server

We will use an an open source (EPL/EDL licensed) message server from Eclipse. The server is called Mosquitto (https://mosquitto.org/) and implements the MQTT protocol versions 5.0, 3.1.1 and 3.1. Mosquitto is lightweight and so can be used on all devices from low power single board computers to full servers.

The installation on a Raspi is done with one command:

    sudo apt install mosquitto
"Just do it" MQTT3:
    sudo service mosquitto status
    netstat -an | grep 1883

Install an MQTT client on the Raspi and use it

We want to publish messages and subscribe to messages on the same Raspi where the server runs. For this we install a client program on the Raspi:

    sudo apt install mosquitto-clients
Command-line subscription

Start the subscribe client mosquitto_sub with:

    mosquitto_sub -V mqttv311 -t PV/sunshine -d

The Raspi mosquitto server and MQTT.fx use MQTT v3.1.1. So we use the -V option to specify the version. The -t option is needed to specify the topic to which we subscribe, and the -d option stands for debug an gives an verbose output with debug informations.

MQTT sub terminal output

"Just do it" MQTT4:
Command-line publishing

Start the publishing client mosquitto_pub with:

    mosquitto_pub -V mqttv311 -t PV/sunshine -m "yes" -d

The new -m option is needed to specify the message.

MQTT sub terminal output

"Just do it" MQTT5:

MQTT messages and connections

The message format

The MQTT protocol is flexible, trustworthy and efficient. MQTT messages combine a fixed header (all packets), with variable headers (some packets) and payload (some packets).


MQTT message format

Fixed header

The (nearly) fixed header has 2 byte. The first byte contains the message type and 3 flags (DUP, QoS and RETAIN). The flags are only used in MQTT 3.1.1 in the PUBLISH control packet. The second byte contains the message length (0-127). This is the length of the variable headers and the payload. With none of these the length byte is 0.

If bit 7 of this byte is set, one to three more bytes can be used to store the message length, wich gives a maximum length of 268435455 byte (0xFF, 0xFF, 0xFF, 0x7F). So the (nearly) fixed header can have from 2 to 5 bytes.

message type (name) value description direction
reserved  0 reserved --
CONNECT  1 client request to connect to server client ⟶ server
CONNACK  2 connect acknowledgment client ⟵ server
PUBLISH  3 publish message client ⟷ server
PUBACK  4 publish acknowledge client ⟷ server
PUBREC  5 publish received (assured delivery part 1) client ⟷ server
PUBREL  6 publish released (assured delivery part 2) client ⟷ server
PUBCOMP  7 publish complete (assured delivery part 3) client ⟷ server
SUBSCRIBE  8 client request to subscribe to topic client ⟶ server
SUBACK  9 subscribe acknowledgment client ⟵ server
UNSUBSCRIBE 10 client request to unsubscribe from topic client ⟶ server
UNSUBACK 11 unsubscribe acknowledgment client ⟵ server
PINGREQ 12 client requests PING client ⟶ server
PINGRESP 13 PING response client ⟵ server
DISCONNECT 14 client is disconnecting client ⟶ server
reserved 15 reserved --
Variable headers

With a variable content in the message (topics, client IDs, last will and testament ...) we need a variable header to transport this information. With e.g. 2 topics, we need 2 variable headers. The 2 first byte contain the length, the following bytes the information. If QoS 1 or QoS 2 are used a packet identifier (consecutive number), is appended.

Payload

The payload length is the message length minus the variable headers length. As the protocol uses a binary transmission, the payload can transport any data.

Delving deeper

With 14 messages we have the all possibilities to login, logout, publish, subscribe and surveillance. Often the client initiates the communication. The server answers with acknowledge messages.

task client server
login/logout CONNECT, DISCONNECT CONNACK
publish PUBLISH, PUBACK, PUBREC, PUBREL, PUBCOMP PUBLISH, PUBACK, PUBREC, PUBREL, PUBCOMP
subscribe SUBSCRIBE, UNSUBSCRIBE SUBACK, UNSUBACK
surveillance PINGREQ PINGRESP
Login/logout

The MQTT server is responsible for all connections. It has to authenticate and authorize the MQTT clients. If a client wants to establish a connection with the MQTT server, it must send a CONNECT control packet with a payload that includes all the necessary information to the server. The MQTT server will check the packet. After performing authentication and authorization it sends a CONNACK control packet to the client to signalise the successful connection. If the client sends an invalid CONNECT control packet, the server automatically closes the connection. The server will keep a successful connection open until the client sends a DISCONNECT control packet to the server or the connection is lost.

MQTT connectMQTT disconnect

The CONNECT control packet must include the following fields or flags in the payload:

MQTT ubsubscribe

MQTT sub terminal output

The following fields are optional in the CONNECT control packet:

After a valid CONNECT control packet the server responds with a CONNACK control packet.

The CONNACK packet has a header with the following fields and flags:

Publish

The used packets depend on the Quality of Service (QoS) levels (see above).

MQTT QoS 0MQTT QoS 1MQTT QoS 2

The PUBLISH packet has a header with the following fields and flags:

PacketId:
With QoS level > 0 the packet identifier has a number value. It is needed to identify the packet and make it possible to identify the responses related to this packet.

The payload contains the actual message. This message is as stated is data agnostic, and therefore, we can send any binary data, not only strings.

Subscribe

MQTT subscribe

A single SUBSCRIBE packet can request to subscribe to many topics. The SUBSCRIBE packet includes at least one topic filter and QoS (list of topic + QoS). For QoS 1 and 2 we also get a PacketId field.

The SUBSCRIBE packet has the following fields:

If the server gets a valid SUBSCRIBE packet it will respond with a SUBACK packet that confirms the receipt and processing of the SUBSCRIBE packet.

The SUBACK packet has the following fields:

Unsubscribe

MQTT ubsubscribe

The UNSUBSCRIBE packet contains a PacketId in the header and one or more topics in the payload. It is not necessary to include the QoS levels. A single UNSUBSCRIBE packet can request the server to unsubscribe a client from many topics. The server responds with an UNSUBACK packet. It confirms the receipt and processing of the UNSUBSCRIBE packet. The UNSUBACK packet contains the same PacketId in the header that was received with the UNSUBSCRIBE packet.

"Just do it" MQTT6:

MQTT client with Arduino

Let's get practical. We want to create an IoT device, capable of publishing and subscribing. An App on our mobile phone is used to get the information and to give us the possibility to act.

The device should have a light sensor and should be able to switch on an LED lamp and even dimm it.

For our device we need a microcontroller or sb-computer that has a TCP/IP stack. And we need an MQTT library for this device. Fortunately there are MQTT client libraries available for the most popular programming languages and platforms. Some of the libraries might not implement all the features of MQTT, so we must look if a library is suitable or not for our needs.

Possible solutions for IoT devices acting as MQTT client (subscriber, publisher or both) are Arduino or ESP boards (with WiFi, Ethernet or both), a Raspberry Pi board, a mobile phone, a laptop, tablet , computer , NAS, server etc..

"Just do it" MQTT7:

The MQTT library

Next we need an MQTT library for Arduino. One of the oldest and stablest libraries is the library of Nick O'Leary called PubSubClient.


MQTT pubsubclient

Publishing in Arduino

To understand how the library works we will look at first at an Arduino sketch that only publishes a message:

    // Basic ESP32 MQTT example to publish a message

    #include <WiFi.h>
    #include <PubSubClient.h>

    const char *WIFI_SSID = "myssid";       // SSID
    const char *WIFI_PASSWORD = "mypass";   // password

    // MQTT settings
    const char *MQTT_SERVER = "192.168.1.84";
    const char *MQTT_CLIENT_ID = "bussy_mqtt_pub_1";
    const char *MQTT_TOPIC = "bussy_mqtt_pub";
    const short MQTT_PORT = 1883;  // TLS=8883

    WiFiClient ESP32_Client;
    PubSubClient MQTT_Client(ESP32_Client);

    long prev_millis = 0;
    char mqtt_message[128];
    int message_counter = 0;
    int pub_delay = 3000; // in ms

    void setup() {
      Serial.begin(115200);
      init_wifi();
      MQTT_Client.setServer(MQTT_SERVER, MQTT_PORT);
    }

    void loop() {
      if (!MQTT_Client.connected()) {
        mqtt_reconnect();
      }
      MQTT_Client.loop();
      if (millis() - prev_millis > pub_delay) {
        prev_millis = millis();
        ++message_counter;
        snprintf (mqtt_message, 128, "hello world #%ld", message_counter);
        Serial.print("Publish MQTT message: ");
        Serial.println(mqtt_message);
        MQTT_Client.publish(MQTT_TOPIC, mqtt_message);
      }
    }

    void init_wifi() {
      Serial.print("Connecting to ");
      Serial.println(WIFI_SSID);
      WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
      while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
      }
      randomSeed(micros());
      Serial.println("\nWiFi connected\nIP address: ");
      Serial.println(WiFi.localIP());
    }

    void mqtt_reconnect() {                        // Loop until reconnected
      while (!MQTT_Client.connected()) {
        Serial.print("Attempting MQTT connection...");
        if (MQTT_Client.connect(MQTT_CLIENT_ID)) { // Attempt to connect
          Serial.println("connected");
          MQTT_Client.publish(MQTT_TOPIC, "connected");
        } else {
          Serial.print("failed, rc=");
          Serial.print(MQTT_Client.state());
          Serial.println(" try again in 5 seconds");
          delay(5000); // wait 5 seconds before retrying
        }
      }
    }

First we need to include the WiFi library (included in the core package) and the SubPubClient library. Next we initialise some constants as the SSID and the password for our WiFi connection and the IP of our MQTT server. We will use two functions, one to initialise and connect to WiFi (init_wifi()) a and a second to reconnect to the MQTT server if the connection is lost (mqtt_reconnect()). In our main loop we use the function millis() to get a delay of 2 s without blocking the loop.

The reconnect function is a blocking function meaning if a reconnect is not possible the other tasks of the sketch will not be done until a reconnection is possible.

For the message itself a buffer of 128 byte is reserved. With the help of the snprintf() function the text is printed into the buffer string using formatting aids to integrate variables. The format string follows the same specifications as format in the printf() function.

"Just do it" MQTT8:


MQTT.fx

Subscribing in Arduino
    // Basic ESP32 MQTT example to subscribe to a topic

    #include <WiFi.h>
    #include <PubSubClient.h>

    // WiFi and network settings
    const char *WIFI_SSID = "myssid";       // SSID
    const char *WIFI_PASSWORD = "mypass";   // password

    // MQTT settings
    const char *MQTT_SERVER = "192.168.1.84";
    const char *MQTT_CLIENT_ID = "bussy_mqtt_pub_1";
    const char *MQTT_OUT_TOPIC = "bussy_mqtt_pub";
    const char *MQTT_IN_TOPIC = "bussy_mqtt_sub";
    const short MQTT_PORT = 1883;  // TLS=8883

    WiFiClient ESP32_Client;
    PubSubClient MQTT_Client(ESP32_Client);

    void setup() {
      pinMode(LED_BUILTIN, OUTPUT);     // Initialize the LED_BUILTIN
      Serial.begin(115200);
      init_wifi();
      MQTT_Client.setServer(MQTT_SERVER, MQTT_PORT);
      MQTT_Client.setCallback(mqtt_callback);
    }

    void loop() {
      if (!MQTT_Client.connected()) {
        mqtt_reconnect();
      }
      MQTT_Client.loop();
    }

    void init_wifi() {
      Serial.print("Connecting to ");
      Serial.println(WIFI_SSID);
      WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
      while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
      }
      randomSeed(micros());
      Serial.println("\nWiFi connected\nIP address: ");
      Serial.println(WiFi.localIP());
    }

    void mqtt_reconnect() {                         // Loop until reconnected
      while (!MQTT_Client.connected()) {
        Serial.print("Attempting MQTT connection...");
        if (MQTT_Client.connect(MQTT_IN_TOPIC)) {      // Attempt to connect
          Serial.println("connected");
          MQTT_Client.publish(MQTT_OUT_TOPIC, "connected");
          MQTT_Client.subscribe(MQTT_IN_TOPIC);         // ... and resubscribe
        } else {
          Serial.print("failed, rc=");
          Serial.print(MQTT_Client.state());
          Serial.println(" try again in 5 seconds");
          delay(5000); // wait 5 seconds before retrying
        }
      }
    }

    void mqtt_callback(char* topic, byte* payload, unsigned int length) {
      Serial.print("Topic: ");
      Serial.print(topic);
      Serial.print("\nMessage: ");
      for (int i = 0; i < length; i++) {
        Serial.print((char)payload[i]);
      }
      Serial.println();
      if ((char)payload[0] == '1') {     // Switch LED on if first character is '1'
        digitalWrite(LED_BUILTIN, HIGH); // LED on
      } else {
        digitalWrite(LED_BUILTIN, LOW);  // LED off
      }
    }

In this example we get have 2 topics, one to publish and one to subscribe. We need the same init_wifi() but the mqtt_reconnect() function changes. It must reconnect and subscribe to the in topic. New is the mqtt_callback() function.

"Just do it" MQTT9:
    char input[128];
    for (int i = 0; i < length; i++) {
      input[i] = (char)payload[i];
    }
"Just do it" MQTT10:

MQTT security

Security is for the IoT an extremely important topic! Sometimes we we use MQTT to publish values that are neither confidential nor critical for other applications, but often it is important, that only the authorized persons can access a device e.g if we control a drone via MQTT.
Another aspect is that our device perhaps can be used to gain control over the network behind our device. If you search the web you find many of these exploits, like the hack of the Cherokee jeep or the HVAC hack at target.
The novel "Blackout" from author Marc Elsberg published in 2012 shows impressively how hacked smartmeters could be used to trigger a collapse of electrical grids across Europe. The book made clear that we didn't focus enough on security and helped to to draw attention to the problem. It was even noticed by Germany politics.

Security generates always an additional overhead. So it is important to keep a balance to avoid overheads that can make our project unusable. Also some cryptographic algorithms are not suitable for microcontroller with little processing power in our IoT boards. If security is needed we better choose a more powerful hardware for our IoT board. An ESP32 microcontroller for example has a separate hardware accelerator for cryptographic algorithm implementations, that neither an Arduino or ESP8266 can provide.
Many security levels also may require too many maintenance tasks (e.g. generate and distribute a certificate for each device), so the project gets unmaintainable and extremely complex.

For security we need need Authentication to check who (person or device) wants to publish or subscribe (login with name and password) and we need Authorisation to allow the access to all the data or parts of it or to deny the access.

Security on different levels

MQTT has some own security features, but uses also other standardized solutions like SSL/TLS. As MQTT resides in the top layers of the on top of the OSI model we can implement security in different layers:

Authentication with username and password

To create a password file we have the utility mosquitto_passwd. The switch -c is used to create a new password file (overwrites an existing file!) and the switch -b is convenient to run in batch mode and allow passing passwords on the command line. Here an example:
The file is named mosqui_passfile. Two logins with the usernames btsiot1 and btsiot2 are created. The passwords are btsiotpass1 and btsiotpass2.


MQTT create password file

The file can be found after we created it in /home/pi. We see that the username is cleartext, but the passwords are hashed and not readable!


MQTT password file

Now we need to tell mosquitto to use the file. This is done in a configuration file. The default configuration file is in /etc/mosquitto/mosquitto.conf. If we open this file we read at the beginning to place our own file into the folder conf.d:


MQTT config file

A link to the mentioned example file is here. We also see that a log file is already specified.
Let's create our own configuration file with the name mymosqui.conf and place it in the /etc/mosquitto/conf.d folder:

    # My personal config file for mosquitto

    # Logging (don't define "log_dest file" because already defined in default!)
    log_dest stdout
    log_type error
    log_type warning
    log_type notice
    log_type information
    connection_messages true
    log_timestamp true

    # Security
    #clientid_prefixes btsiot-
    allow_anonymous false

    # Default authentication and topic access control
    password_file /home/pi/mosquitto/mosqui_passfile
"Just do it" MQTT11:

Client Id with unique prefix

We can increase security with a unique prefix for the client Id.

"Just do it" MQTT12:

Encryption with TLS-PSK

Even if we use passwords and prefix, the password, client ID and the topics are only encrypted if WiFi is used. In a wired environment it is easy to get these information with a sniffer as wireshark (as seen in the just do it exercise).

The only secure solution is encryption. We have two possibilities. We can use TLS-PSK Transport Layer Security cipher suites with a pre-shared key or TLS/SSL encryption.

The pre-shared keys are symmetric keys (same key on client and server) shared in advance among the clients and server to establish a TLS connection. Pre shared keys are used e.g. in WiFi routers where all the clients need to know the WiFi key in advance. Clearly it is difficult to exchange secret keys with unknown clients over the Internet (e.g. to secure Internet shopping). This is the disadvantage of TLS-PSK, and for such applications TLS/SSL with asymmetric keys (private and public keys) is used.

So what are the advantages of TLS-PSK?
A first advantage of TLS-PSK (depending on the cipher suite) is that there are no public key operations (no key server) needed. So less performant microcontroller can be used. A second advantage is that it is also easier in closed environments (manually configuration in advance) to configure a PSK than to use certificates.

If we send a message of 72 Byte we get 1172 byte with TLS-PSK and 3742 byte with TLS/SSL. So clearly we get less overhead with TLS-PSK.

In MQTT the PSK keys are defined in a PSK file. The name of the file is arbitrary. PSK identities and keys are separated with a colon. The key is in hex and should have more than 20 digits. Let's try this with a file named mosqui_pskfile with the following content:

    btsiot1:0123456789abcdef0123

Now we add the following line to /etc/mosquitto/conf.d/mymosqui.conf:

    # Port to use for the default listener.
    port 8883
    # Pre-shared-key based SSL/TLS support (text "btsiot" is arbitrary)
    psk_hint btsiot
    # Default authentication and topic access control
    psk_file /home/pi/mosquitto/mosqui_pskfile

The default port for encrypted MQTT communication is 8883. The psk_hint line switches psk support on. The text is needed for authentication by the broker but is arbitrary.

Now we can publish and subscribe with the following lines. Because of our unique prefix the -i switch (ClientId) is not optional.

    mosquitto_sub -V mqttv311 -p 8883 -i btsiot-client1 --psk-identity btsiot1 --psk 0123456789abcdef0123 \
    -u btsiot1 -P btsiotpass1 -t PV/sunshine -d

    mosquitto_pub -V mqttv311 -p 8883 -i btsiot-client2 --psk-identity btsiot1 --psk 0123456789abcdef0123 \
    -u btsiot1 -P btsiotpass1 -t PV/sunshine -m Hallobts -d
"Just do it" MQTT13:
TLS-PSK with Arduino

To use TLS-PSK we include the WiFiClientSecure header file and use an object of type WiFiClientSecure instead of WiFiClient. In setup() we need to add the setPreSharedKey() method to submit the PSK identity and the key. Here is the code to publish a message:

    // Basic ESP32 MQTT example to publish an encrypted message with TLS-PSK

    #include <WiFi.h>
    #include <PubSubClient.h>
    #include <WiFiClientSecure.h>

    //const char *WIFI_SSID = "myssid";       // SSID
    //const char *WIFI_PASSWORD = "mypass";   // password

    // MQTT settings
    const char *MQTT_SERVER = "192.168.1.84";
    const char *MQTT_CLIENT_ID = "btsiot-client2";
    const char *MQTT_PSK_IDENTITY = "btsiot1";
    const char *MQTT_PSK_KEY = "0123456789abcdef0123"; // hex string without 0x
    const char *MQTT_TOPIC = "PV/sunshine";
    const short MQTT_PORT = 8883;  // TLS=8883
    const char *MQTT_USERNAME = "btsiot1";
    const char *MQTT_PASSWORD = "btsiotpass1";

    WiFiClientSecure ESP32_SEC_Client;
    PubSubClient MQTT_Client(ESP32_SEC_Client);

    long prev_millis = 0;
    int pub_delay = 3000; // in ms
    char mqtt_message[128];
    int message_counter = 0;

    void setup() {
      Serial.begin(115200);
      init_wifi();
      MQTT_Client.setServer(MQTT_SERVER, MQTT_PORT);
      ESP32_SEC_Client.setPreSharedKey(MQTT_PSK_IDENTITY, MQTT_PSK_KEY);
    }

    void loop() {
      if (!MQTT_Client.connected()) {
        mqtt_reconnect();
      }
      MQTT_Client.loop();
      if (millis() - prev_millis > pub_delay) {
        prev_millis = millis();
        ++message_counter;
        snprintf (mqtt_message, 128, "Hello with TLS-PSK #%ld", message_counter);
        Serial.print("Publish MQTT message: ");
        Serial.println(mqtt_message);
        MQTT_Client.publish(MQTT_TOPIC, mqtt_message);
      }
    }

    void init_wifi() {
      Serial.print("Connecting to ");
      Serial.println(WIFI_SSID);
      WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
      while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
      }
      randomSeed(micros());
      Serial.println("\nWiFi connected\nIP address: ");
      Serial.println(WiFi.localIP());
    }

    void mqtt_reconnect() {                        // Loop until reconnected
      while (!MQTT_Client.connected()) {
        Serial.print("Attempting MQTT connection...");
        if (MQTT_Client.connect(MQTT_CLIENT_ID,MQTT_USERNAME,MQTT_PASSWORD)) { // Attempt to connect
          Serial.println("connected");
          MQTT_Client.publish(MQTT_TOPIC, "connected");
        } else {
          Serial.print("failed, rc=");
          Serial.print(MQTT_Client.state());
          Serial.println(" try again in 5 seconds");
          delay(5000);  // Wait 5 seconds before retrying
        }
      }
    }
"Just do it" MQTT14:

Encryption with TLS/SSL

Usually, Transport Layer Security (TLS) uses public key certificates. We find many sites on the internet that explain how to use TLS/SSL on the Raspberry Pi with mosquitto or with Arduino. As this topic is already covered in other modules and can be reviewed in the net, we will not go into it further.

Interesting links: