HowTo setup fluent bit on Sensor

Agenda

  • Request  fluent bit instance for IOT Sensor
  • Learn configuration parameters for fluent bit

This tutorial teaches how to setup fluent bit on IOT Sensor.

Prerequisites

Before doing this tutorial make sure you have read and completed the following tutorials:

IMPORTANT: In order to successfully request a Fluentbit instance, you will need to have a validated Allocation Supply in your project with the following configuration:

  • Software Product: Fluent bit 1.0.351 software release with the default software type
  • Compute Node: the Node corresponding to your Olimex Sensor

The Software Product should already be ready for you to use, no need to create it.

If you wish to know more about the configuration of Fluentbit, you can find the official documentation here: Fluentbit Configuration File.

Request fluent bit instance for IOT Sensor

Step 1: Request instance

Please click on "Request" menu at left side of the page as shown on image below.

This will redirect you to your project page where you will have to select the appropriate project.

After selecting the project will will be given option to choose the product you want to install. In this case it is fluent bit.

Finally you will be redirected to your new instance where you will need to specify the needed instance's parameters.

Following actions are needed:

  1. Set a proper instance name (in the case it is "t20.mqtt-broker")
  2. Click on "Raw XML" button which will enable you to edit directly the instance's XML parameters
  3. Insert and adjust for your case the instance XML (the parameters ware wrapped in "<" and ">"). The XML itself you can find here. Please note that it is intentionally escaped XML!
  4. Click on "Request" button. Your instance will be created in background and its status will be visible from your "Service" tabs on your left side.

The instance itself

Description of parameters

TAG The MQTT tag name in this format (see example below). Please note that proper Data Supply is needed (see HowTo Create Data Supply)
FLUENTD-GATEWAY-IPv4 The IPv4 address which fluentd gateway will bind to. Usually something like 192.168.x.x

In the screenshots we use for:

  • TAG = sensor1.mqtt
  • FLUENTD-GATEWAY-IPv4 = 192.168.0.117

Step 2: Monitor instantiation of your instance

You can monitor the process by running

tail -f /opt/slapos/log/slapos-node-instance.log

on the respective machine.

Fluent bit Instance

This step can take some time.

In your SlapOS dashboard you can see your newly requested instance.

The instance is ready when the the Monitoring Status is green.

Note that the Fluentd configuration parameters we defined earlier can be found and edited in the Update parameter form .

Step 3: Control (start / stop) IOT fluent bit instance

Login to your IOT Sensor and run the following command :

sudo slapos node status

You see that a new service for fluent bit is added.

And the generated fluent bit configuration file is located here :

/srv/slapgrid/slappartX/etc/fluent-bit-config-file.cfg

Now your IOT fluent bit is ready to read data and to forward it over fluentd IOT Gateway to Wendelin.

Step 4: Add MQTT handling at IOT Gateway

This steps requires to ONLY change and adjust already setup-ed Fluentd IOT  Gateway here.
The only difference is that one instead of fluentd-gateway.xml one should use fluentd-gateway-with-mqtt.xml

Please reuse the fluentd instance created already and adjust the MQTT section for your setup inside SlapOS. Please see the diff between files below. Do not forget to adjust accordingly!

Step 5: Configure Wendelin's Data Supply

In order to make Wendelin store passed to it MQTT messages one needs to create respective "Data Supply" objects with proper "References". The process is described here. Still as minor adjustments are needed some screenshots follow.

Below you can see the important to set Reference on the "Data Supply" object. In this case as part of the tutorial it is "sensor3".

It is also required to set "Reference" on the "Data Product" as it can be seen below. In this case as part of the tutorial it is "mqtt".

Step 6: Test sending MQTT message to Wendelin

You need to further install additional packets for testing MQTT functionality, see below

# install needed packages for testing MQTT
root@t20:~#
apt install mosquitto-clients

In order to test sending of MQTT messages to Wendelin you need to know on which IPv4 fluent bit is listening. The IPV4 is stored in ¨/srv/slapgrid/slappartX/etc/fluent-bit-config-file.cfg" and defined by  "Listen" directive. It is something like "10.x.x.x"

Using this it is easy for example:

root@t20:~# mosquitto_pub -t sensor3.mqtt -m '{ "payload": { "message": "Hello, World from t20!" } }' -h <IPv4>

Step 7: Check MQTT message is stored in Wendelin

For this you need to go to your Wendelin instance and choose "MQTT Messages" from "Modules" selection. Then you can (if needed) use search criteria like "sensor3.mqtt" into "Title" column to filter MQTT messages for this TAG only. See below

And respectively the MQTT messages contents as it can be seen from below.