Nexedi - Free Software R&D

• DREAM - Application decision support using discrete event simulation in Python
• WENDELIN - Big Data stack & Machine Learning, out-of-core computing
• OPEN PAAS - Next-generation collaboration platform for business

Stack: Wendelin Core

• "Out-Of-Core"
• Allows computation beyond limitation of physical RAM
• Embedded within Wendelin stack, but also usable standalone

Stack: NEO

• Distributed Transactional NoSQL Object Database
• Used as default storage in Wendelin

Stack: ERP5

• Based on 5 Theoretical Principles (Unified Business Model)
• Generic: handles Big Data the same way as millions of ERP-related objects
• ERP5 used under the hood of Wendelin

Stack: SlapOS

• Cloud Provisionment, Deployment, Orchestration for single machines or clusters
• Installable on any architecture
• Uses Ansible recipes, fully automated, everything is a service

Stack: re6st

• IPv6-based Resilient Mesh Overlay Network, "internet on top of the internet"
• Monitors possible connections between nodes, channels traffic on demand
• Mitigates risks of unavialable routes/packet loss

Stack: ShaCache

• MD5-signed cache, use signatures to validate cached files
• Allows to reuse trusted compilations by providing key value
• NoSQL storage with REST API, archives files through HTTP PUT method

Stack: Scikit.learn (et al)

• Integrate popular libraries for Data Science and Machine Learning
• Provide familiar interface which connects to ERP5/Wendelin.Core
• Scikit.Learn (project partner), Jupyter, SciPy, NumPy, Pandas

Setup: Explained

• Public Accessable IPv6, IPv4 tutorial fallback
• Load Wendelin via ShaCache
• Port forwarding via Socat

Setup: Undressed

• "Naked" Machine with a KVM
• Can be hosted anywhere (OVH, AWS, ...) any architecture (ARM, Intel...)
• Public IPv4 and IPv6 (Res6t)

Setup: Tutorial Preparation

• We use Slapos to deploy a cluster of VMs for the tutorial
• Using Two machines, 20 partitions
• Each partition with KVM Virtual Machine

Setup: SlapOS "OUTSIDE" - Manage Access

• SlapOS manages partitions and installs virtual machine
• Port forwarding from Public IPv6 to KVM private IPv4
• SlapOS simplifies all access by always going through IPv6

Setup: SlapOS "INSIDE" - Manage Wendelin

• SlapOS is installed on KVM, too (used to manage cluster and VM)
• Shortcut: Wendelin download pre-compiled via ShaCache
• Shortcut: Socat to automatically bind, port forward KVM

Todo: Access Terminal

• Click the terminal tab
• Enter SSH/VNC provided (like SSH slapos@2001:67c:1254:e:1b::XXXX -p 10022)

Todo: Socat Bind

• wget https://lab.nexedi.com/nexedi/wendelin/raw/master/utils/wendelin-standalone-bind.sh && bash wendelin-standalone-bind.sh
• This will activate urls provided to access Jupyter/ERP5

Todo: Fix Consistency

• Select All and Fix Consistencies

Todo: Install Configuration

• Select Configuration for Wendelin
• Click "Install" on next screen

Todo: HOTFIX: Add Business Template

• Goto My Favorites > Manage Business Templates
• HOTFIX are addable in live system via business templates
• Business Templates are standard way of adding functionality in ERP5/Wendelin

Todo: Enter Business Template URL

• Enter the HOTFIX Url provided and click to download
• http://www.wendelin.io/wendelin05_release_hotfix-0.5.bt5/Base_download
• Click download

Todo: Main Interface

• Go back to the main interface
• Your instance is now ready to use

Todo: Create Ingestion Policy

• We now setup Wendelin to receive data from fluentd
• Goto: http://softinstxxxx.host.vifib.net/erp5/portal_ingestion_policies/
• Security Setting to prevent arbitrary stream from being sent

Todo: Fast Input

• Hit the "Green Runner" for fast input
• Enter "pydata" as our reference name and save
• This creates a new ingestion policy and corresponding data stream

Todo: Check Data Streams

• Switch to Data Stream Module
• You will now have an empty stream, setup to ingest data from fluentd

Wendelin in Production

• First Wendelin prototype in production to monitor wind turbines.
• Wendelin used to collect data and manage wind parks.
• Machine Learning for failure prediction, structural health monitoring.

Basic FluentD

• FluentD is open source unified data collector
• Has a source and destination, generic, easy to extend
• Can handle data collection even under poor network conditions

Todo: Record Audio

• We use another SlapOS Webrunner hosting only a webpage:
• https://softinst56756.host.vifib.net/public/project/hyperconvergence/
• Click "Record" to Start/Stop audio and "Save" to save

Todo: Forward File to fluentD

• In the Editor, open folders project > slapos
• Create a new file, give it your name
• Code on next slide (please copy&paste)

Todo: FluentD Configuration File (Gist)

<source>
 @type bin
 format none
 path
/srv/slapgrid/slappart9/srv/runner/PUT_YOUR_WAV_HERE/!!YOURNAME!!*.wav
 pos_file /srv/slapgrid/slappart9/srv/runner/!!YOURNAMEGOESHERE!!.pos
 enable_watch_timer false
 read_from_head true
 tag pydata
</source>

<match pydata>
 @type wendelin
 @id wendelin_out 

 streamtool_uri
http://!!URL_TO_YOUR_ZOPE!!/erp5/portal_ingestion_policies/pydata
 user zope
 password insecure 

 buffer_type memory
 flush_interval 1s
 disable_retry_limit true
</match>

Todo: Head to Jupyter (Notebook)

• Head to Juypter http://[x].pydata-class.erp5.cn
• Start a new ERP5 Notebook
• This will make sure you use the ERP5 Kernel

Todo: Learn ERP5 Kernel (Notebook)

• Note you have to connect to Wendelin/ERP5
• The reference you set will store your notebook in the Date Notebook Module
• Passing login/password will authenticate Juypter with Wendelin/ERP5

Todo: Accessing Objects (Notebook)

• Import necessary libs
• Type context, this will give you the Wendelin/ERP5 Object
• Type context.data_stream_module["1"]to get your uploaded sound file

Todo: Accessing Data Itself (Notebook)

• Try to get the length of the file using getData and via iterate
• Note then when using ERP5 kernel all manipulations should be "Big Data Aware"
• Just loading a file via getData() works for small files, but will break with volume

Todo: Compute Fourier (Notebook)

• Proceed to fetch data using getData for now
• Extract one channel, save it back to Wendelin and compute FFT
• Note, that ERP5 kernel at this time doesn't support %matplotlib inline

Todo: Rerun using Big File Reader (Notebook)

• Rerun using the Big File Reader
• Now one more step is out of core compliant
• Verify graphs render the same

WebSite Module

• Website Module contains websites
• Open renderjs_runner - ERP5 gadget interface

Renderjs Runner

• Parameters for website module
• see ERP5 Application Launcher - base gadget
• Open new tab: http://softinstxxxx/erp5/web_site_module/renderjs_runner/

Todo: Publish Website

• Select action Publish and publish the site
• This changes object state from embedded to published
• Try to access: http://softinstxxxx/erp5/web_site_module/pydata_runner/

Todo: Layout Properties

• Change to Tab "Layout Properties tab"
• Update the router to custom_gadget_erp5_router.html
• Refresh your app (disable cache), it will be broken, as this file doesn't exist

Todo: Web Page Module

• Change to web page module
• Search for reference "router"

Todo: Graph Gadget HTML (Gist)

<html>
<head>
  <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
  <meta name="viewport" content="width=device-width, user-scalable=no" />
  <title>PyData Graph</title>
  <script src="rsvp.js" type="text/javascript"></script>
  <script src="renderjs.js" type="text/javascript"></script>
  <script src="gadget_global.js" type="text/javascript"></script>
  <script src="gadget_erp5_page_pydata.js" type="text/javascript"></script>
</head>
<body>

</body>
</html>

Todo: Update Graph Gadget HTML (Gist)

<!DOCTYPE html>
<html>
<head>
  <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
  <meta name="viewport" content="width=device-width, user-scalable=no" />
  <title>PyData Graph</title>
  
  <!-- renderjs -->
  <script src="rsvp.js" type="text/javascript"></script>
  <script src="renderjs.js" type="text/javascript"></script>
  <!-- custom script -->
  <script src="dygraph.js" type="text/javascript"></script>
  <script src="gadget_global.js" type="text/javascript"></script>
  <script src="gadget_erp5_page_pydata.js" type="text/javascript"></script>
</head>
<body>
  <div class="custom-grid-wrap">
    <div class="custom-grid ui-corner-all ui-body-inherit ui-shadow ui-corner-all"></div>
  </div>
   <div data-gadget-url="gadget_ndarray.html"
        data-gadget-scope="ndarray"
        data-gadget-sandbox="public">
  </div>
</body>
</html>

Todo: Graph Gadget JS (1) (Gist)

/*global window, rJS, console, RSVP, Dygraph */
/*jslint indent: 2, maxerr: 3 */
(function (rJS) {
  "use strict";

  var ARRAY_VALUE_LENGTH = 8,
    OPTION_DICT = {
      start_date: 0,
      time_factor: 1000,
      resolution: 1,
      xlabel: 'x',
      ylabel: 'y',
      key_list: ["Channel 1", "Date"],
      label_list: ["Date", "Channel 1"],
      series_dict: {
        "Channel 1": {
          axis : "y",
          color: "#00884B",
          pointSize: 1,
          visible : true,
          connectSeparatedPoints: true
        }
      },
      axis_dict: {
        y: {position : "left", axisLabelColor: "grey", axisLabelWidth : 40, pixelsPerLabel : 30},
        x: {drawAxis : true, axisLabelWidth : 60, axisLabelColor: "grey", pixelsPerLabel : 30}
      },
      connectSeparatedPoints: true
    };
  ...
  

Todo: Graph Gadget JS (2) (Gist)

  function generateInitialGraphData(label_list) {
    var i,
      data = [[]];
    for (i = 0; i < label_list.length; i += 1) {
      data[0].push(0);
    }
    return data;
  }

  function convertDateColToDate(gadget, array) {
    var label_list = gadget.property_dict.option_dict.label_list,
      time_factor = gadget.property_dict.option_dict.time_factor,
      time_offset = gadget.property_dict.option_dict.time_offset || 0,
      i,
      k;

    for (k = 0; k < label_list.length; k += 1) {
      if (label_list[k] === "Date") {
        for (i = 0; i < array.length; i += 1) {
          array[i] = [i, array[i]];
        }
      }
    }
    return array;
  }

  ...
  

Todo: Graph Gadget JS (3) (Gist)

  rJS(window)

    .ready(function (gadget) {
      gadget.property_dict = {};
      return gadget.getElement()
        .push(function (element) {
          gadget.property_dict.element = element;
          gadget.property_dict.option_dict = OPTION_DICT;
        });
    })

    .declareAcquiredMethod("jio_getAttachment", "jio_getAttachment")

    // render gadget
    .declareMethod('render', function () {
      var gadget = this,
        interaction_model = Dygraph.Interaction.defaultModel,
        option_dict = {},
        url;

      url = "http://softinstxxxx/erp5/web_site_module/pydata_runner/hateoas/data_array_module/[your_id]";

  ...
  

Todo: Graph Gadget JS (4) (Gist)

      return new RSVP.Queue()
        .push(function () {
          return gadget.jio_getAttachment("erp5", url, {
            start : 0,
            format : "array_buffer"
          });
        })
        .push(function (buffer) {
          var array_length,
          length,
          array,
          array_width = 1;

        array_length = Math.floor(
          buffer.byteLength / array_width / ARRAY_VALUE_LENGTH
        );
        length = buffer.byteLength - (buffer.byteLength % ARRAY_VALUE_LENGTH);
        if (length === buffer.byteLength) {
          array = new Float64Array(buffer);
        } else {
          array = new Float64Array(buffer, 0, length);
        }
        return nj.ndarray(array, [array_length, array_width]);
      })
  ...
  

Todo: Graph Gadget JS (5) (Gist)

    .push(function (result) {
        var i,
          data = [],
          ndarray = result,
          label_list =  gadget.property_dict.option_dict.label_list,
          key_list = gadget.property_dict.option_dict.key_list;

        for (i = 1; i < label_list.length; i += 1) {
          data = data.concat(
            nj.unpack(
              ndarray.pick(
                null,
                key_list.indexOf(label_list[i])
              )
            )
          );
        }
        data = convertDateColToDate(gadget, data);
        gadget.property_dict.data = data;
        return gadget
      });
    })
    .declareService(function () {
      var gadget = this;
      return gadget.property_dict.graph = new Dygraph(
        gadget.property_dict.element,
        gadget.property_dict.data,
        gadget.property_dict.option_dict
      );
    });
}(rJS));