Norikra in Action (ver. 2014 spring)

Norikra in Action
TAGOMORI Satoshi (@tagomoris)
LINE Corp.
2014/01/31 (Fri) at University of Tsukuba
the 2nd half

14年1月31日金曜日

Norikra:
Schema-less Stream Processing with SQL
Open source software (GPLv2)
http://norikra.github.io/
https://github.com/norikra/norikra


Norikra:
Schema-less event stream:
Add/Remove data ﬁelds whenever you want
SQL:
No more restarts to add/remove queries
w/ JOINs, w/ SubQueries
w/ UDF
Truly Complex events:
Nested Hash/Array, accessible directly from SQL


Norikra Queries: (1)

SELECT name, age
FROM events

target


{“name”:”tagomoris”,
“age”:34, “address”:”Tokyo”,
“corp”:”LINE”, “current”:”Tsukuba”}

SELECT name, age
FROM events

{“name”:”tagomoris”,”age”:34}

“address”:”Tokyo”,

SELECT name, age
FROM events

nothing


SELECT name, age
FROM events
WHERE current=”Tsukuba”

{“name”:”tagomoris”,”age”:34}

{“name”:”kawashima”,
“age”:99, “address”:”Tsukuba”,
“corp”:”Univ”, “current”:”Dream”}

SELECT name, age
FROM events

nothing


SELECT age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
GROUP BY age



GROUP BY age
every 5 mins
{”age”:34,”cnt”:3}, {“age”:33,”cnt”:1}, ...


FROM
events.win:time_batch(5 mins)
GROUP BY age

SELECT max(age) as max
FROM
events.win:time_batch(5 mins)

{”age”:34,”cnt”:3}, {“age”:33,”cnt”:1}, ...
{“max”:51}

every 5 mins

“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Tsukuba”,
“speaker”:true,
“attend”:[true,true,false, ...]
}

GROUP BY age

“current”:”Tsukuba”,
“speaker”:true,
}

SELECT user.age, COUNT(*) as cnt
GROUP BY user.age

“current”:”Kyoto”,
“speaker”:true,
}

AND attend.$0 AND attend.$1
GROUP BY user.age

Norikra and Esper
Esper:
CEP engine library, Java, GPLv2
EPL: Event Processing Language (SQL + window)
Streams: schema-full ﬂat ﬁeld set
Norikra:
Using Esper internally
Schema-less stream -> schema-full stream
conversion
Rewriting compiled queries


Norikra query execution
1.accept query
2.parse -> find target / field set
3.(if target is not opened) wait for first event
4.compile query
5.rewrite target name into stream name
6.rewrite field names
7.register query
8.input events

Target mapping

Ignore unused ﬁelds
Field set matching between streams and queries
Generate ﬁeld set inheritance tree


automated stream inheritance
of norikra's target
Base fieldset

Query fieldset

Data fieldset


b_xxxxxxxxx

minimal fieldset definition:
name: 'string'
id: 'long'
valid: 'boolean'
action_type: 'string'

of norikra's target
Base fieldset

b_xxxxxxxxx

Query fieldset

Data fieldset


e_xxxxxxxx1

event data fieldset definition:
name: 'string'
id: 'long'
valid: 'boolean'
product_code: 'string'
charge: 'integer'
shop_code: 'long'

of norikra's target
Base fieldset
name: 'string'
id: 'long'
valid: 'boolean'
product_code: 'string'
Query fieldset
charge: 'integer'
shop_code: 'long'
affiliate: 'string'
Data fieldset


e_xxxxxxxx1

b_xxxxxxxxx

e_xxxxxxxx2

of norikra's target
Base fieldset

b_xxxxxxxxx
new query:
SELECT count(*)
FROM target.win:time_batch(1min)
WHERE affiliate.length() > 0

Query fieldset

Data fieldset


e_xxxxxxxx1

e_xxxxxxxx2

new query:
SELECT count(*)
FROM target.win:time_batch(1min)

of norikra's target
Base fieldset

b_xxxxxxxxx

Query fieldset

Data fieldset


name: 'string'
id: 'long'
valid: 'boolean'
affiliate: 'string'

q_xxxxxxxx0

e_xxxxxxxx1

e_xxxxxxxx2'

Registered
automated stream inheritance EPL:
SELECT count(*)
of norikra's targetFROM q_xxxxxxxx0.win:time_batch(1min)

Base fieldset

b_xxxxxxxxx

Query fieldset

Data fieldset


q_xxxxxxxx0

e_xxxxxxxx1

e_xxxxxxxx2'

of norikra's target
Base fieldset

b_xxxxxxxxx

Query fieldset

q_xxxxxxxx0

q_xxxxxxxx1

Data fieldset e_xxxxxxxx1'

e_xxxxxxxx2'

e_xxxxxxxx3'


Query rewriting
“speaker”:true,
}

GROUP BY user.age

“speaker”:true,
}

GROUP BY user.age

“speaker”:true,
}
“user.age”:34,
“user.corp”:”LINE”,
“user.address”:”Tokyo”,
“speaker”:true,
“attend.$0”:true,
“attend.$2”:false,
...
}

“user.age”:34,
“user.corp”:”LINE”,
“user.address”:”Tokyo”,
“speaker”:true,
“attend.$2”:false,
...
}

{“user$age”:34,
“attend$$0”:true,
}

{“user$age”:34,
}

GROUP BY user.age


{“user$age”:34,
}
SELECT

user.age, COUNT(*) as cnt
GROUP BY user.age

Conversions for compiled query object in fact.

SELECT user$age, COUNT(*) as cnt
AND attend$$0 AND attend$$1
GROUP BY user$age

Norikra internal:
Jump
from schema-full world
to schema-less world.


Norikra in Action (ver. 2014 spring)

Recommended

More Related Content

What's hot (20)

More from SATOSHI TAGOMORI (20)

Recently uploaded (20)

Norikra in Action (ver. 2014 spring)