Wendelin Exanalytics 2020 Big Data with MariaDB

2014-04-03 – Santa Clara updated 2014-06-16

www.wendelin.io

Agenda

Our background: ERP5
Our future: Wendelin Exanalytics
Our challenge: out-of-core

ERP5

MariaDB

NEO

Python

ERP5

Web Workflow

Document Management

Supply Chain

Finance

MRP

Customisation

Fine Grain Security

Full Traceability

Scalability

Flexibility

Rapid prototyping

Zope TTW on steroids

Banking

Aerospace

Health

Chemical

Government

NGO

Cloud Computing

Consulting

Mechanical

Online contribution for 3rd parties

To-do lists

Notifications

Careers and assignments

Payroll

Projects

CRM

Terra-SAR X Satellite

Accessible to Airbus partners and distributors

Interfaces with DLR

(Germany Space Agency)

« With ERP5, our partners all over the world can access our infrastructure and order online with complete security “ Ralf Duering

Management of sales and production of images

Compliant with ESA standard (ECSS)

SANEF Group

« Web has become our primary sales channel. » Frédéric Charlier

Online sales and customer relation for ETC Tolling

120.000 new customers / year

51.000 invoice/hour

7.000.000 contacts / year

250 users

Implemented in 4 months

Open Source ERP/CRM for S&P 100

Agenda

Our background: ERP5
Our future: Wendelin Exanalytics
Our challenges with MariaDB

Take the Best Analytics scikit-learn.org

Made by Great Mathematicians

http://en.wikipedia.org/wiki/Fields_Medal

Wendelin Werner

Add Distributed Storage neoppod.org

Add Elastic PaaS erp5.com

# Initialize data

data_size = 1000000

server_count = 1000

chunk_size = data_size / server_count

data = array(data_size)

# Process data in parallel on each server (Map Reduce, Batch, etc.)

for server in server_count:

data.activate().process(server*chunk_size, chunk_size)

PaaS

And Multicloud Deployment slapos.org

MMC Rus

Wendelin Exanalytics Core 100% open source

NEO

SlapOS

Scikit Learn

ERP5

Multicloud Deployment

Elastic PaaS

Distributed Storage

Data Analytics

Multi Data Center

100% Python

Wendelin User Interface renderjs.org

Wendelin Options 100% open source

Time sequence processing

DataPad / JP Morgan

JIT compiler / type inference

Continuum / DARPA

Scikit Learn

Pandas

Numba / Parakeet

NEO

100% Python

Blaze

Full out-of-core arrays

Continuum / DARPA

Reatime log collection

Treasure Data / Amazon

Fluentd

NLTK

Natural Language Tookit

U. Texas / Chalmers

OpenCV-Python

Video Processing

Intel Russia / Willow / Itseez

Wendelin Applications

Intrusion detection
Fraud detection
Business and economic prevision
Marketing
Media analysis
Public security
Brain Computer Interface
Internet Of Things

Business Model: German Style No VC

Nexedi (WendelinCo)

Scikit Learn

Big Data System User

Extension 1

Big Data System Supplier

Extension 2

100% open source

hardware

100%

1 - 10%

proprietary

Agenda

Our background: ERP5
Our future: Wendelin Exanalytics
Our challenge: out-of-core

Out-of-core arrays

# Numpy

np.ndarray(shape=(2,2), dtype=float, order='F')

# Out-of-core data

np.ndarray(shape=(1e18,2), dtype=float, order='F')

# Full out-of-core

np.ndarray(shape=(1e9,2e9), dtype=float, order='F')

2 Exabyte

Best out-of-core topology depends on the algorithm and array geometry

neo.ndarray out-of-core data

neo.ndarray

NEO Overview

neoctl

Sate access

Command control

Master

OID & TID allocation

Synchronisation

Load balancing

Storage

Object data

Transaction data

Partition table

Application

ZODB

neo.client

Data

Control

Admin

State archival

Command proxy

NEO Overview

neoctl

Sate access

Command control

Master

OID & TID allocation

Synchronisation

Load balancing

Storage

Object data

Transaction data

Partition table

Application

ZODB

neo.client

Data

Control

Admin

State archival

Command proxy

Object retrieval

Retrieve x :

hash(x._p_oid)

Parition	Node	State
S1	IP:PORT	?
S2	IP:PORT	Connected
S3	IP:PORT	?

Partition	Node	State
0	S1
	S3
1	S2
	S3
...	...	...

Variable

Roadmap

Q3 2014: neo.ndarray
Q3 2014: developer release of Wendelin
Q4 2014: neo.ndarray with simple optimizations
Q1 2015: mariadb embedded
Q2 2015: coloured caches
Q3 2015: coloured caches with C client cache
Q4 2015: GO storage

Challenges

Reduce latency → embedded mariadb ?
Reduce SQL overhead → precompile queries ?
Reduce copies → BLOB protocol ?
Accelerate storage → C++ ? GO ?
Optimize cache → colored caching

Wendelin Exanalytics 2020 Big Data with MariaDB

2014-04-03 – Santa Clara updated 2014-06-16

Wendelin Exanalytics - 2020 Big Data with MariaDB

Wendelin Exanalytics 2020 Big Data with MariaDB