Wendelin Big Data Industrial Monitoring Platform

2014-04-03 – Paris

Who are we?

Jean-Paul Smets
Nexedi CEO
Author of ERP5
jp@nexedi.com

Ivan Tyagov
Senior Developer
Wendelin project lead
ivan@nexedi.con

Who is missing?

Kirill Smelkov
Senior Developer
wendelin.core

Sebastien Robin
Project Director
Author of POC

Agenda

Where do we come from
Wendelin Architecture
Detailed Example
Future Roadmap

Where do we come from?

Nexedi

Possibly Largest OSS Publisher in Europe

ERP5: ERP, CRM, ECM, e-business framework
SlapOS: distributed mesh cloud operation system
NEO: distributed transactional NoSQL database
Wendelin: out-of-core big data based on NumPy
re6st: resilient IPv6 mesh overlay network
RenderJS: javascript component system
JIO: javascript virtual database and virtual filesystem
cloudooo: multimedia conversion server
Web Runner: web based Platform-as-a-Service (PaaS) and IDE
OfficeJS: web office suite based on RenderJS and JIO

Application Convergence

Case 1: Wind Turbines

Collect logs
Collect records
Predict failure
Plan maintenance
Reduce downtime
→ add X% profits

Case 2: Cars

Collect logs
Collect records
Predict failure
Plan maintenance
Reduce downtime
→ increase loyalty

Case 3: Solar Energy

Collect logs
Collect records
Predict degradation
Plan maintenance
Increase efficiency
→ add X% to profits

Wendelin Architecture

Standard Hardware no router / no SAN

2 x 10 Gbps
2 x 6 core Xeon CPU
512 GB RAM
4 x 1 TB SSD
1 x M2090 GPU

x 160

x 32

x 320

10 Gbps
Unmanaged

Wendelin Hypercube Datacenter

Take the Best Analytics scikit-learn.org

Add Distributed Storage neoppod.org

neoctl

Sate access

Command control

Master

OID & TID allocation

Synchronisation

Load balancing

Storage

Object data

Transaction data

Partition table

Application

ZODB

neo.client

Data

Control

Admin

State archival

Command proxy

“Magic” out-of-core for NumPy

ZBigArray

PyData Paris 2015 – 16h45 Kirill Smelkov

Add Elastic PaaS erp5.com

# Initialize data

data_size = 1000000

server_count = 1000

chunk_size = data_size / server_count

data = array(data_size)

# Process data in parallel on each server (Map Reduce, Batch, etc.)

for server in server_count:

data.activate().process(server*chunk_size, chunk_size)

PaaS

And Multicloud Deployment slapos.org

MMC Rus

Wendelin Platform 100% open source

NEO

SlapOS

Scikit Learn

ERP5

Multicloud Deployment

Elastic PaaS

Distributed Storage

Data Analytics

Multi Data Center

100% Python

Wendelin Options 100% open source

Time sequence processing

DataPad / JP Morgan

JIT compiler / type inference

Continuum / DARPA

Scikit Learn

Pandas

Numba / Parakeet

NEO

100% Python

Blaze

Full out-of-core arrays

Continuum / DARPA

NLTK

Natural Language Tookit

U. Texas / Chalmers

OpenCV-Python

Video Processing

Intel Russia / Willow / Itseez

Data Ingestion: fluentd

Based on MsgPack middleware
Created by TreasureData (BDaaS pioneers)
Used by Amazon
Numerous plugins
Scalable and resilient
Bandwidth saver

Wendelin UI

HTML5 Render RenderJS
Data vizualisation
Offline support JIO

Data access REST API
Batch processing

REST GET

JSON + HATEOAS

Javascript

Python

Wendelin Distinctive Advantages

Native out-of-core NumPy (scikit-learn, pydata)
Native parallel processing
Bare metal performance (GPU, FORTRAN)
Transactions (ingestion, processing)
NewSQL queries
Built-in PaaS
Lower deployment cost (10x less than...)

Detailed Example

Data Transportation fluentd

3 months benchmark

Frequent downtime (server, network)

Very poor networking (ADSL, 3G)

fluentd(Buffer)

DPU

Fluentd

Central server

< 0.001% loss

UI: HTML5 Components RenderJS

Extend UI Components RenderJS

UI : Responsive RenderJS

UI: Offline / Other Backends JIO

Data Science in Javascript vs. Python ?

Data Sciences in Javascript ? phantomjs

Small data on client side
Small data on server side
Medium data (> 1 GB) in JS
Out-of-core data in JS
PyData compiled in JS
PyData in NaCl / PNaCl

Storing large streams in NEO

Title, date

File (inhouse)

BtreeData

0-8

BtreeNode 0-4

BtreeNode 4-8

BtreeNode

2-4

BtreeNode

4-6

BtreeNode

6-8

BtreeNode 0-2

0M-1M

1M-2M

2M-3M

3M-4M

4M-5M

5M-6M

6M-7M

7M-8M

Access: O(log(N)

Overhead : 0.2%

Structure

Data

UBM Monitoring Model?

UBM Business Model

Movement – ingestion of data
Resource – type of data (ex. memory log)
Node – data source, data owner
Path – data source registration
Item – sensor, data itself, license, data set

UBM Business Model

What UBM gets us for free

Accounting, billing and payment
User registration and management
Rule based security model
Customer relationship management
Web Content Management

→ save 12+ months and > 200 K€ on any Big Data project

Future Roadmap

Roadmap

Mainly accelerate learning curve

Universal packaging
Ready to use examples
Act as a backend to ipython notebook
Port joblib to CMFActivity

Yet, you can start using part of Wendelin now!

wendelin.core out-of-core for NumPy
JIO abstract data access library
RenderJS components
UI sample application
Open Source

www.wendelin.io

PyData Paris 2015 – 16h45 Kirill Smelkov

http://learn.renderjs.org

https://lab.nexedi.cn/Tyagov/wendelin/

R&D Partners

Wendelin-IA (FSN)

Nexedi
Abilian
2nd Quadrant
Paris 13
IMT
INRIA / ENS
MMC Rus (Ru)
X Corp

Windelin (Eurostars)

Nexedi (FR)
MariaDB (FI)
Y Corp (DE)

www.wendelin.io

Wendelin Big Data Industrial Monitoring Platform

2014-04-03 – Paris

Wendelin Big Data: Industrial Monitoring Platform

Wendelin Big Data Industrial Monitoring Platform