Wendelin Home Wendelin

    Reading Fluentd Data With Wendelin

    A tutorial showing how to receive data from fluentd in Wendelin using a simulated IOT sensor.
    • Last Update:2016-06-24
    • Version:
    • Language:

    Todo: Create Ingestion Policy

    Wendelin-ERP5 Ingestion Policies
    • We now setup Wendelin to receive data from fluentd
    • Goto: http://softinstxxxx.host.vifib.net/erp5/portal_ingestion_policies/
    • Security Setting to prevent arbitrary stream from being sent
    • Currently fluentd and Wendelin are setup to receive streams of data
    • A stream is an existing file to which data is continually appended
    • The congestion policy "initializes" a Data Stream (creates a new file)

    Todo: Fast Input

    Wendelin-ERP5 Create Ingestion Policy
    • Hit the "Green Runner" for fast input
    • Enter "pydata" as our reference name and save
    • This creates a new ingestion policy and corresponding data stream
    • The ingestion policy includes default scripts to be called on incoming data
    • To modify data handling, you could now define your own script

    Todo: Check Data Streams

    Wendelin-ERP5 Data Stream
    • Switch to Data Stream Module
    • You will now have an empty stream, setup to ingest data from fluentd
    • Click on the stream. You can also upload data by hand if you like to experiment

    Audio: Simulate IOT Sensor

    Wendelin in Production

    Wendelin | Windpark
    • First Wendelin prototype in production to monitor wind turbines.
    • Wendelin used to collect data and manage wind parks.
    • Machine Learning for failure prediction, structural health monitoring.

    Wendelin in Production

    Wendelin | Wind Turbine
    • Turbine equipped with multiple sensors per blade, PC collecting sensor data
    • PC includes fluentd for ingesting into Wendelin (network availability!)
    • Tutorial: we use same setup, sensor - Fluentd - Wendelin

    Basic FluentD

    Wendelin | Basic FluentD Flowchart
    • FluentD is open source unified data collector
    • Has a source and destination, generic, easy to extend
    • Can handle data collection even under poor network conditions

    Complex FluentD

    Wendelin | Complex FluentD Flowchart
    • Setup of FluentD can be much more complex
    • In Wendelin production, every turbine has its own FluentD
    • Tutorial simulates the same, all VM have their own FluentD

    Todo: Record Audio

    Wendelin-ERP5 Audio Recorder

    Todo: Access Monitor using fluentD

    Wendelin-ERP5 Monitor Webrunner
    • Access Monitor at: https://softinst67162.host.vifib.net/
    • Go to the Editor, click "This Project"

    Todo: Upload to Monitor

    Wendelin-ERP5 Monitor Webrunner File Upload
    • We already made a folder for uploads for this tutorial
    • Left click and select upload file

    Todo: Forward File to fluentD

    Wendelin-ERP5 Webrunner FluentD Configuration
    • In the Editor, open folders project > slapos
    • Create a new file, give it your name
    • Code on next slide (please copy&paste)
    • We are now creating a configuration file to pass to fluentd
    • The file contains all parameters for fluentD regarding data source and destination
    • Normally this is set upfront, but for the tutorial we hardcode

    Todo: FluentD Configuration File (Gist)

        <source>
            @type bin
            format none
            path
            /srv/slapgrid/slappart9/srv/runner/PUT_YOUR_WAV_HERE/!!YOURNAME!!*.wav
            pos_file /srv/slapgrid/slappart9/srv/runner/!!YOURNAMEGOESHERE!!.pos
            enable_watch_timer false
            read_from_head true
            tag pydata
            </source>
            <match pydata>
            @type wendelin
            @id wendelin_out
            streamtool_uri
            http://!!URL_TO_YOUR_ZOPE!!/erp5/portal_ingestion_policies/pydata
            user zope
            password insecure
            buffer_type memory
            flush_interval 1s
            disable_retry_limit true
            </match>
    • Note that tag will be the name of your data-ingestion-policy/stream

    Todo: Save and send from Terminal

    Wendelin-ERP5 Webrunner FluentD File Transfer
    • Switch to the terminal and enter:
    • software/bd283b01bd665a31e41dce567f199491/bin/fluentd -c project/YOUR_NAME.cfg
    • To pass your configuration including what file to send to Wendelin

    Todo: Save and send from Terminal

    Wendelin-ERP5 - Verify File was received
    • Head back to Wendelin/ERP5
    • In the Data Stream Module, check the file size of the pydata stream
    • It should now show a file size larger than 0