Understanding MQTT protocol : IoT Part 14

Message Queue Telemetry Transport (MQTT) protocol is an application layer protocol. The application layer provides an interface between the end nodes (IOT devices) and the network. In case of computers, laptops and mobile devices, the application layer is usually implemented by the browser. In case of IOT devices, the application layer may be implemented either by the running operating system (if an embedded OS is running on it) or by the firmware.

In computing world, the most common and ubiquitous application layer protocol used is Hypertext Transfer Protocol (HTTP). The HTTP protocol has been designed for communication between a client and a server. The protocol uses request/response method in which when a client needs data from the server then it sends a request message to the server and server sends back a response message. HTTP is applied where only one end can start the data communication. That is, the client initiates data request and server responds to it. The protocol was designed to render application layer implementation for computers. It can still be used by IOT devices, however, IOT applications with constraint resources face limitations with the HTTP protocol.

First of all, as already mentioned, HTTP is designed for one-way communication where from one-way is meant that data communication can be initiated only from one side. The server cannot initiate communication with the client device on its own. Secondly, many, in fact, most of the IOT applications need to operate on limited network bandwidth. The IOT devices themselves have limited RAM and ROM on-board and limited computing resources. They are battery powered so need to operate on low power. They also need to transfer or communicate data in small code fragments that can be securely transmitted to the other end despite limited bandwidth and on-board resources. The data though being small in size must be not only be communicated reliably but must also be communicated in real time. These specific demands of an IOT environment make the HTTP and other common computer networking application protocols misfit in the IOT scenario.

So, to cater to specific requirements of IOT systems, protocols like MQTT has been designed. MQTT was designed by Andy Stanford-Clark (IBM) and Arlen Nipper in 1999 for connecting oil pipeline telemetry systems over satellite.

MQTT is a protocol or set of rules which is used for Device to Device (D2D) communication to remotely exchange the data from one client to another client. It involves device to device or machine to machine communication over the global network through an intermediate component called MQTT broker or server. MQTT is designed in such a way that client and server both can communicate with each other. With MQTT protocol, any end can initiate the data communication.

A device that requests data is referred as subscriber in the MQTT protocol and the device that provides data is referred as publisher. This is a lightweight publish/subscription protocol in which each individual MQTT client is connected to the MQTT broker in order to communicate with other MQTT clients. An IOT device can configure as publisher, subscriber or both. It needs to configure as publisher, subscriber or both with the MQTT broker. It is a message oriented protocol in which the clients send and receive the data in the form of a message with the particular topic. Any publisher or subscriber communicate over the network via the broker only. Any of them can initiate data communication by connecting through the broker. Like the publisher can provide data to the broker whenever required and the subscriber can also request data whenever required. The data communication though being initiated by any end is communicated in real time. So, with the help of the broker, there can be a two-way communication over the network.

Also, MQTT is applicable for the resource constrained and power limited devices. The protocol is designed to communicate small code footprints in a reliable and real-time manner despite limited network bandwidth. IoT applications are possible without MQTT but MQTT provides architectural advantages over common computer networking application layer protocols when resource constrained IOT devices have to communicate bi-directionally over limited bandwidth.

The publishers and subscribers can also be completely decoupled in the protocol stack.This protocol is developed on the top of the TCP/IP protocol stack to provide remote communication between the IOT devices. This protocol stack provides lossless, ordered, bi-directional connection between devices. As devices in Internet of things are more dispersed and have smaller code footprint and limited bandwidth, a small code footprint solution like MQTT is becoming the ever growing protocol in the market.

The reason for saying MQTT as the light weight protocol is that the packet overhead of this protocol is extremely small. The smallest packet has only 2 bytes overhead. This small transport overhead helps to reduce network traffic.

MQTT Architecture

MQTT uses Publish/Subscribe model which actually requires the use of Central Broker or Server. It is also known as client/server model. There are mainly two clients in MQTT – one acts as publisher and other acts as a subscriber. Publisher client consists of Payload and Topic. Publisher client publishes the message or payload to a particular topic (say X) to the MQTT broker and subscriber client subscribes to the particular topic (like X) from the MQTT broker and receives the message and perform accordingly.

In order to understand the architecture of MQTT for D2D communication, it is important to understand some terms related with it. These terms are as follow –

1) Payload – Payload is an actual message which a client (publisher) may want to send and share. The maximum payload that can be sent using MQTT protocol has size limited to 268, 435, 456 bytes which can be maximum 256 MB in a packet. The maximum size is limited by the MQTT specifications.

2) Topic – Topics are the virtual channels which are used to define the path where the data is being sent. The maximum length of the topic can be 65535 bytes according to the MQTT specifications. The Topics are created by Publishers and sent to the broker with the message and then those topics can be subscribed by the subscriber. As long as the topic name follows the standards, it is accepted by the broker.

MQTT clients do not use addresses like mobile numbers or email addresses. This means that a client does not need to mention these addresses to the other clients. Instead of host and destination addresses, they use Topics. Topics are UTF-8 (encoding scheme) string, which are used by the broker to filter messages for each connected client. A topic consists of one or more topic levels. Each topic level is separated by a forward slash. The topic is very lightweight string.

For example, topics as UTF-8 strings looks like the following – My_home/first_floor/dining_room/temperature

India/Delhi/India_gate/humidity

In both examples, topics are used to define the path of the message. In the first example, it is shown that the message (temperature) will be sent to the dining room which is on the first floor of the home. The topics are case sensitive which means My_home/first_floor will be different from My_Home/first_floor. So the topics must be written in the code with caution.

Learn more about the topics from the following tutorial –

MQTT Topics and Best Practices

3) MQTT Broker – The broker or server is mainly responsible for receiving all messages from publisher clients and filtering them. It decides which subscriber is interested in it and then send the messages to those subscribed clients. MQTT brokers are used as a mediator to receive and transmit the data. MQTT broker is the heart of this publish/subscribe protocol. MQTT broker can handle up to thousands of concurrently connected devices. The broker should be responsible for authentication and authorization of clients and the important thing is, the broker should be failure-resistant.

4) MQTT Publisher – An MQTT client registered as publisher sends payload to the MQTT broker. For example, a temperature sensor can be publisher type client in the protocol. It can send a temperature message to the broker corresponding to a related Topic, message being wrapped in a payload.

5) MQTT Subscriber – An MQTT client registered as subscriber requests messages (data) on Topics from the broker. For example, a mobile device might request temperature message to be displayed by an app from the broker.

MQTT Methods

MQTT defines methods to indicate the desired action to be performed on a device. Like there are GET, POST, PUT, DELETE methods in HTTP protocol, there are the following methods in MQTT protocol –

1) CONNECT

2) CONNACK

3) PUBLISH

4) PUBACK

5)PUBREC

6) PUBREL

7) PUBCOMP

8) SUBSCRIBE

9) SUBACK

10) UNSUBSCRIBE

11) UNSUBACK

12) PINGREQ

13) PINGRESP

14) DISCONNECT

CONNECT – To establish connection between broker and client, client sends a CONNECTpacket to the broker. The CONNECT packet helps the client to connect to the broker and start the rest of the communication. The CONNECT packet contains one or more encoded fields like Client Identifier for a client, a Will Topic, Will Message, User Name, Password etc. The CONNECT packet contains a 2-byte fixed header, a variable header (containing protocol name, protocol level, connect flags and Keep Alive) and the payload.

The input parameters in Connect packets are as follow –

void connect(char *ClientIdentifier, char UserNameFlag, char PasswordFlag, char *UserName, char *Password, char CleanSession, char WillFlag, char WillQoS, char WillRetain, char *WillTopic, char *WillMessage);

The payload contains the following fields which must be passed as the parameter in the Connect function –

Fig. 1: Table listing MQTT Commands and their Functions

Fig. 2: Table listing MQTT Commands and their Functions

Once the connection is established and CONNECT packet has been transferred, the client can send OnConnect packet to the MQTT broker. The OnConnect packet is set in a callback function which means this function will be called whenever MQTT connection is established. In this OnConnect callback function, the client can call publish and subscription functions (according to the client’s need). Whenever MQTT connection is established, the MQTT publisher client will automatically publish the message with a particular topic to the MQTT broker and will subscribe to the topic automatically.

CONNACK – This packet is sent by the server to the client in response to the CONNECT packet from the client. The packet contains a fixed 2-byte header and a variable header. The variable header contains Connect Acknowledge Flags and Connect Return Codes. This packet has no payload. The Connect Return Codes can be as follow –

Fig. 3: Table Listing Connect Return Codes

PUBLISH – This packet is sent by the client to the server or server to the client to transport application messages. The packet contains a 2-byte fixed header, a variable header and a payload. The fixed header contains flags for MQTT Control Packet Type, DUP Flag, QoS level and RETAIN flag. The variable header contains Topic Name and the packet identifier. The payload contains the application message that has to be published. The content and format of data in payload is always application specific.

The input parameters in Publish packets are as follow –

void publish(char DUP, char Qos, char RETAIN, unsigned intMessageID, char *Topic, char *Message);

The input parameters in the PUBLISH function should be as follow –

Fig. 4: Table Listing Input Parameters in the PUBLISH Function

PUBACK – This packet is sent either by client to server or server to client if QoS level is 1 in response to the PUBLISH packet. It contains a 2-byte fixed header along with a variable header. There is no payload in this packet. The variable header contains packet identifier bytes.

PUBREC – This packet is sent either by client to server or server to client if QoS level is 2 in response to the PUBLISH packet. It is the second packet of the QoS 2 protocol exchange. It contains a 2-byte fixed header along with a variable header. There is no payload in this packet. The variable header contains packet identifier bytes.

PUBREL – This packet is sent either by client to server or server to client if QoS level is 2 in response to the PUBLISH packet. It is the third packet of the QoS 2 protocol exchange. It contains a 2-byte fixed header along with a variable header. There is no payload in this packet. The variable header contains packet identifier bytes.

PUBCOMP – This packet is sent in response to a PUBREL Packet. It is the fourth and final packet of the QoS 2 protocol exchange. It contains a 2-byte fixed header along with a variable header. There is no payload in this packet. The variable header contains packet identifier bytes.

SUBSCRIBE – This packet is sent by the client to the server. It contains a 2-byte fixed header, variable header and a payload. It contains a list of Topic Filters indicating the Topics to which the Client wants to subscribe. The Topic Filters in a SUBSCRIBE packet payload MUST be UTF-8 encoded strings. The Server should support Topic filters that contain the wildcard characters. If it chooses not to support topic filters that contain wildcard characters it must reject any Subscription request whose filter contains them. Each filter is followed by a byte called the Requested QoS. The input parameters in Subscribe packets are as follow –

void subscribe(char DUP, unsigned intMessageID, char *SubTopic, char SubQoS);

The input parameters in the SUBSCRIBE function should be as follow –

Fig. 5: Table Listing Input Parameters in the SUBSCRIBE Function

It is used only in messages where the QoS level is greater than 0 (SUBSCRIBE message is at QoS=1)

SubTopic It is Topic Name to which subscription is needed.SubQoS It is QoS level at which the client wants to receive messages. It’s possible values are 0, 1 and 2 corresponding to QoS level 0, 1 and 2 respectively. The default value is 0.

After connecting, MQTT clients automatically publish and subscribe to the topic and message, they can send OnMessage packet to the MQTT broker. The OnMessage packet is also set in a callback function. This function is called when the message is received from another publisher client to the subscribed topic. Inside this function, they can call a custom code so that they can perform some activity. For Example, Client-01 is publishing the message “ON” with topic “Device/LED” on the MQTT broker and client-02 has subscribed the topic “Device/LED”. Whenever a message arrives at the client-02, the client-02 can perform some task based on that, like it can glow the LED connected to it.

OnMessage callback function includes the following parameters –

void OnMessage(char *Topic, intTopicLength, char *Message, intMessageLength)

The input parameters in the OnMessage function should be as follow –

Fig. 6: Table Listing Input Parameters in the OnMessage Function

SUBACK – This packet is sent by the Server to the Client to confirm receipt and processing of a SUBSCRIBE Packet. It contains a fixed 2-byte header, a variable header and a payload. The variable header contains packet identifier bytes. The payload contains a list of return codes, that specify the maximum QoS level that was granted in each Subscription that was requested by the SUBSCRIBE. The return codes can be as follow –

Fig. 7: Table Listing Return Codes of SUBACK Packet

UNSUBSCRIBE – If MQTT clients want to unsubscribe to the particular topic, an UNSUBCRIBE packet must be sent by the client to the broker. The packet contains a 2-byte fixed header, a variable header and a payload. The variable header contains the packet identifier bytes. The payload contains the list of Topic Filters that the Client wishes to unsubscribe from. The Topic Filters in an UNSUBSCRIBE packet MUST be UTF-8 encoded strings. The Payload of an UNSUBSCRIBE packet MUST contain at least one Topic Filter. An UNSUBSCRIBE packet with no payload is a protocol violation. The packet can be sent by calling unsubscribe function. This unsubscribe function will help MQTT clients to unsubscribe to the topic they have previously subscribed.

The input parameters for the Unsubscribe function includes the following –

void unsubscribe(char DUP, unsigned intMessageID, char *SubTopic);

UNSUBACK – This is sent by the Server to the Client to confirm receipt of an UNSUBSCRIBE Packet. It contains a 2-byte fixed header and a variable header. There is no payload in this packet. The variable header contains the packet identifier bytes.

PINGREQ – There is also a keepalive functionality for the clients. The keep alive functionality ensures that the connection is still open and both MQTT broker and MQTT clients are connected to each other. The keep alive time is the maximum time in seconds to which client and broker can be connected to each other even when they are not exchanging the data to each other. The MQTT specification says the following:

It is the responsibility of the Client to ensure that the interval between Control Packets being sent does not exceed the Keep Alive value. In the absence of sending any other Control Packets, the Client MUST send a PINGREQ Packet.

That means, as long as two devices are exchanging their information to each other and the keep alive time interval is not exceeded the desired limit, there is no need to send an extra message to check that connection is still open or not. But when clients stop exchanging their data and the keepalive time interval is also exceeded then the MQTT clients need to send PINREQ message to the MQTT broker to ensure the connection and the MQTT broker will response with PINRESP to ensure that the MQTT client is connected.

The PINGREQ packet simply contains a 2-byte fixed header. There is no variable header or payload in this packet.

PINGRESP – This packet is sent by the Server to the Client in response to a PINGREQ Packet. It indicates that the Server is alive. It simply contains a 2-byte fixed header. There is no variable header or payload in this packet.

DISCONNECT – If the client wants to disconnect from the MQTT broker, then it must send a DISCONNECT packet to the broker. The packet simply contains a 2-byte fixed header. There is no variable header or payload in this packet. It can be sent by calling a disconnect function. The disconnect function should have the following parameters –

void disconnect(void);

How MQTT Protocol works?

In MQTT, the devices act as clients and the server or broker acts as an access point which has its own SSID and password. The MQTT server continuously listens for a connection from clients. The clients do not talk to each other directly or are aware of each other. They communicate via MQTT broker with each other.

Suppose there are two clients – Client-01 and Client-02. The client-01 acts as a publisher client and client-02 act as subscriber client. The publisher client publishes two things to the MQTT broker – payload (actual message) and topic (the virtual channel) and the subscriber client subscribes to the topic from the MQTT broker. The publisher client and subscriber client both first connect to the MQTT broker so that they can start communicating with each other. Whenever the client initiates a connection to the MQTT broker, it needs to be connected through their unique IDs.

Next, the publisher client registers the topic with the Quality of Service level (QoS 0,1 or 2) to the MQTT broker and subscriber client subscribes to the topic with Quality of Service level from the MQTT broker. The MQTT broker has enough storage for device registry. It identifies the devices through the unique ID. Now, the subscriber client is continuously listening for the message on the particular topic from the MQTT broker. The publisher client publishes the message on the topic to the MQTT broker, depending on the QoS level at the publisher client side, the message arrives at the MQTT broker and the MQTT broker decides whether the topic is subscribed by any client or not. The subscriber client has subscribed to the topic so the MQTT broker will pass the message to the subscriber client and depend on the QoS level at subscriber side, the message will be received by the subscriber.

A publisher client can simultaneously act as subscriber client also and vice-versa. The whole working as discussed above can be implemented in reverse direction also. In this way, the device to device communication with MQTT protocol is managed.

Fig. 8: Image showing MQTT Protocol Architecture

Key Features of MQTT –

1. It is Lightweight message queuing and transport protocol which provides one-to-many message distribution and decoupling of applications.

2. It is based asynchronous communication model with messages.

3. It has low overhead (2 bytes header) for low network bandwidth applications.

4. It is based Publish / Subscribe (Pub-Sub) model.

5. It allows decoupling of data producer (publisher) and data consumer (subscriber) through topics (message queues).

6. It is a simple protocol, aimed at low complexity, low power, and low footprint (less space) implementations.

7. It runs on connection-oriented transport (TCP). It can be used in conjunction with 6LoWPAN (TCP header compression). For example with wireless sensor network.

Limitations of MQTT

Every protocol has some advantages and some disadvantages. MQTT protocol has also some disadvantages as follow –

1. It is ideal for only constrained networks where low bandwidth, high latency, data limits and fragile connections are demanded.

2.If There is a wireless sensor network that doesn’t support TCP/IP stack then MQTT has no use for the wireless sensor network. In such case MQTT-SN should be used. The MQTT-SN is a modified version of MQTT and is specifically designed for low power wireless sensor networks that don’t have TCP/IP stack with them.

Note that for more details on MQTT protocol and its implementation, check out the official documentation –

MQTT Version 3.1.1

In the next tutorial, discussions on MQTT Broker will be done.