Wendelin Home Wendelin

    Wendelin - HowTo setup Fluentd on IOT Gateway

    HowTo setup Fluentd on IOT Gateway
    • Last Update:2023-01-17
    • Version:
    • Language:

    Agenda

    • Supply Fluentd software on IOT Gateway
    • Learn configuration parameters for Fluentd
    • Request Fluentd instance for IOT Gateway

     

    This tutorial teaches how to setup Fluentd on IOT Gateway. 

    Before doing this tutorial make sure you read and completed the following tutorials

    There are two ways how you can setup Fluentd. The first way uses the SlapOS web panel / dashboard. The second way makes use of the SlapOS Client. For the second way you will need to configure the SlapOS client. For that read SlapOS HowTo Use SlapOS Client tutorial from Get SlapOS Master Token chapter.

    Supply Software

    Supply via SlapOS Panel

    Login to the panel and click on Servers (c).

    Select the machine on which you want to supply Fluentd.

    Supply via SlapOS Panel Cont.

    Click on the Supply button.

    Supply via SlapOS Panel Cont.

    Select Fluentd.

    Supply via SlapOS Panel Cont.

    Choose the newest version.

    Supply via SlapOS Panel Cont.

    Click on Proceed.

    Wait until the compilation process has been finished.

    Supply via SlapOS Client

    First find the reference of your IOT Gateway.

    It is located in your Slapos Dashboard in Servers tab.

    Then using supply command of Slapos Client install Fluentd software release on that computer.

    
    supply("https://lab.nexedi.com/nexedi/slapos/raw/master/software/fluentd/software.cfg", "YOUR_COMPUTER_ID")
    
    

    Configuration Parameters

    
    <source>
    @type forward
    port 24224
    bind ::0
    </source>
    <match YOUR_TAG_NAME>
    @type wendelin
    streamtool_uri https://softinstXXXX.host.vifib.net/erp5/portal_ingestion_policies/default
    user      zope
    password  *****
    <buffer>
    flush_mode interval
    @type file
    path fluentd-buffer-file/
    flush_interval 1m
    </buffer>
    </match>
    
    

    About Fluentd configuration file you can learn from here.

    For input plugin we use Fluentd's standard forward plugin.

    For output we use our custom output plugin called wendelin.

    YOUR_TAG_NAME - it must consist of two parts separated by .(dot). For example sensor_1.sample-environment-raw-data. First part (sensor_1) is the name of the sensor and must be the same as the reference of the Data Supply we created in HowTo Create Data Supply tutorial. The second part (sample-environment-raw-data) is the type of data (Data Product  -  HowTo Create Data Product).

    streamtool_uri - is the URL of the Wendelin instance created in HowTo Request ipv4 frontend with Ciphers tutorial. 

    username and password  of that instance you can find in Connection Information tab of your webrunner as described in HowTo Install Wendelin on Webrunner.

     

     

    Configuration Parameters Cont.

    
    <source>
    @type forward
    port 24224
    bind ::0
    </source>
    <match YOUR_TAG_NAME>
    @type wendelin
    streamtool_uri https://softinstXXXX.host.vifib.net/erp5/portal_ingestion_policies/default
    user      zope
    password  *****
    <buffer>
    flush_mode interval
    @type file
    path fluentd-buffer-file/
    flush_interval 1m
    </buffer>
    </match>
    
    

    We also use file buffering plugin for fluentd. 

    For path give a relative path to a directory where you want to store buffer files.  

    flush_interval 1m  means that data will be flushed to wendelin every minute. 

     

    Request Instance

    Request via SlapOS Panel

    Go to the SlapOS dashboard and click on the Services (l) button.

    Click on the add button.

    Request via SlapOS Panel Cont.

    Select Fluentd.

    Request via SlapOS Panel Cont.

    Pick the newest version.

    Request via SlapOS Panel Cont.

    Select the machine on which you previously supplied Fluentd by clicking on the Computer Node drop-down list. This is the machine on which the instance will be requested.

    Request via SlapOS Panel Cont.

    Enter a title for your Fluentd instance

    Request via SlapOS Panel Cont.

    Fill in the Fluentd configuration parameters in the Instance Parameter field. You can use the <parameter> tag to specify certain parameters of your instance. The name for the Fluentd configuration parameter is conf_text. Because the Instance Parameter itself is written in XML we have to escape our Fluentd Configuration File. You can use an Online XML escape tool (for instance appdevtools).

    <?xml version="1.0" encoding="utf-8" ?><instance>
    <parameter id="conf_text">&lt;source&gt;
    @type forward
    port 24224
    bind ::0
    &lt;/source&gt;
    &lt;match YOUR_TAG_NAME&gt;
    @type wendelin
    streamtool_uri https://softinstXXXXXX.host.vifib.net/erp5/portal_ingestion_policies/default
    user      zope
    password  XXXXX
    &lt;buffer&gt;
    flush_mode interval
    @type file
    path fluentd-buffer-file/
    flush_interval 1m
    &lt;/buffer&gt;
    &lt;/match&gt;
    </parameter>
    </instance>

     

    Finally click Proceed to request the instance.

    Request via SlapOS Client

    
    config_params = """<source>
      @type forward
      port 24224
    bind ::0 
    </source>
    <match sensor_1.sample-environment-raw-data>
    @type wendelin
    streamtool_uri https://softinstXXXXXX.host.vifib.net/erp5/portal_ingestion_policies/default
    user zope
    password ******
    <buffer>
    flush_mode interval
    @type file
    path fluentd-buffer-file/
    flush_interval 1m
    </buffer>
    </match>"""
    request("https://lab.nexedi.com/nexedi/slapos/raw/master/software/fluentd/software.cfg", "IOT-Gateway-Tutorial-Fluentd",filter_kw={"computer_guid": "YOUR_COMPUTER_ID"}, partition_parameter_kw={"conf_text":config_params})

    As the fluentd configuration has many parameters for convenience we will write a short python script which will request the instance, instead of calling request function from Slapos Client.  

    Create a python script and name it request_fluentd.py and copy/paste the presented code. 

    ** Don't forget to correctly fill in your computer reference, Instance name, url etc. 

    As we are requesting instance of fluentd for the same computer the reference is the same as when we were supplying the software.

    "YOUR_INSTANCE_NAME" -  give your instance a distinguishing name so you can later easily find it. For example IOT-Gateway-Tutorial-Fluentd.

    Once it's ready run it by calling following command : 

    slapos console request_fluentd.py

    Monitor instantiation

    You can monitor the process by running

    tail -f /opt/slapos/log/slapos-node-instance.log

    on the respective machine.

    Fluentd Instance

    This step can take some time.  

    In your Slapos dashboard you can see your newly requested instance.

    The instance is ready when the the Monitoring Status is green.

    Note the Parameters section - there are the Fluentd configuration parameters we defined earlier.

    IOT Gateway

    Login to your IOT Gateway and run the following command : 

    sudo slapos node status

    You see that a new service for Fluentd is added.

    And the generated flunted configuration file is located here :

    /srv/slapgrid/slappartX/etc/fluentd-agent.conf

     

    Now your IOT Gateway is ready to receive data and to forward it to Wendelin.