Oedo Ruby Conference 04: Ruby会議でSQLの話をするのは間違っているだろうか
•
11 likes•5,985 views
「Ruby会議でSQLの話をするのは間違っているだろうか」 @大江戸Ruby会議04, 2014-04-19
![コスト差ってどれくらいよ
package org.myorg;
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
!public class WordCount {
public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
! public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
output.collect(word, one);
}
}
}
!public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
int sum = 0;
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));
}
}
!public static void main(String[] args) throws Exception {
JobConf conf = new JobConf(WordCount.class);
conf.setJobName("wordcount");
! conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
! conf.setMapperClass(Map.class);
conf.setCombinerClass(Reduce.class);
conf.setReducerClass(Reduce.class);
! conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
! FileInputFormat.setInputPaths(conf, new Path(args[0]));
FileOutputFormat.setOutputPath(conf, new Path(args[1]));
! JobClient.runJob(conf);
}
}
select count(*) from (
select regexp_split_to_table(str, s+')
from text_table
) t;
MapReduceによるWordCount SQLによるWordCount](https://image.slidesharecdn.com/20140420oedoruby04public-140429081300-phpapp01/85/Oedo-Ruby-Conference-04-Ruby-SQL-26-320.jpg)
Oedo Ruby Conference 04: Ruby会議でSQLの話をするのは間違っているだろうか
- 2. 今日のお話について Theme of this session
- 3. 「技術的に濃い 話題がいいです」 Akira Matsuda said I expect you deep technical talk.
- 4. 「濃い話」is 何 What s deep talk???
- 5. Rubyの実装の話とか もう別に濃くない Ruby implementation is not deep already, so I speak about another theme.
- 6. 25分でわかる ビッグデータ分析 ∼MapReduce追悼∼ Big Data Analytics in 25 minutes
- 7. トータル100TBくらいの データを分析するとしよう Suppose you must analyze 100TB text data
- 8. 1CPUとか もうマヂムリ… コンピュータ プログラム データ Single CPU cannot handle 100TB
- 9. そうだ分散処理しよう ノード0 ノード1 ノード2 ノード3 プログラム プログラム プログラム プログラム データ データ データ データ You need more computers (distributed processing)
- 10. でも分散処理って めんどい…マヂムリ… Distributed processing is too difficult…
- 11. そこで並列RDBですよ Parallel RDB may help you.
- 12. Parallel RDB Node 0 Node 1 Node 2 Node 3 Front End Front End Front End Front End Back End Back End Back End Back End
- 13. 並列RDBの特長 1 ノードを増やせば線形に速くなる！ 2 標準SQLが使える！ 3 クライアントからは1台に見える！ has linear scalability You can use SQL Looks as the one computer Parallel RDB is great because…
- 14. 並列RDB超スゴイ age age マック Parallel RDB is great
- 15. いろいろな商用並列RDB Database Vendor Since Teradata Teradata 1983 Teradata Aster Teradata 2005 PureData System for Analytics IBM 2000 Exadata Oracle 2008 Greenplum Pivotal 2003 SQL Server PDW Microsoft 2010くらい Redshift Amazon 2012 Various parallel RDBs
- 16. Here Comes a New Challenger!!
- 17. since 2005
- 18. Hadoop Architecture HDFS: Distributed File System MapReduce: Compute Framework (Hive: SQL interface)
- 19. Hadoopの特徴 1 データはテーブルではなくファイル 2 処理にはMapReduceを使う（使っていた） 3 Hiveを乗せるとSQLっぽい言語でも書ける Hadoop data is plain file Processed by MapReduce Hive allows you to write SQL-like query
- 20. 猫も 子も MapReduce Big Data meant MapReduce few years ago
- 21. MapReduce k1, v1 k2, v2 k3, v3 k4, v4 k5, v5 k6, v6 M a p k'1, v'1 k'1, v'2 k'1, v'3 k'2, v'4 k'3, v'5 k'3, v'6 k''1, v''1 k''2, v''2 k''3, v''3 R e d u ce
- 22. Map関数とReduce関数を 書いたらよしなに 分散してくれるフレームワーク You just write Map&Reduce functions, Hadoop serves the rest
- 23. Q1. SQLとMapReduce どっちがいいの？！ Which is good, SQL and MapReduce
- 24. ビジネス的な答え： SQL
- 25. なぜSQLか 1 SQL書けてもJava書けない人は多い 2 既存のSQLを使ったアプリが動かない 3 MapReduce関数を書くのは高コスト Many people can write SQL but not Java Many applications rely on SQL Writing MR function needs more time
- 26. コスト差ってどれくらいよ package org.myorg; import java.io.IOException; import java.util.*; import org.apache.hadoop.fs.Path; import org.apache.hadoop.conf.*; import org.apache.hadoop.io.*; import org.apache.hadoop.mapred.*; import org.apache.hadoop.util.*; !public class WordCount { public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); ! public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); output.collect(word, one); } } } !public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { int sum = 0; while (values.hasNext()) { sum += values.next().get(); } output.collect(key, new IntWritable(sum)); } } !public static void main(String[] args) throws Exception { JobConf conf = new JobConf(WordCount.class); conf.setJobName("wordcount"); ! conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(IntWritable.class); ! conf.setMapperClass(Map.class); conf.setCombinerClass(Reduce.class); conf.setReducerClass(Reduce.class); ! conf.setInputFormat(TextInputFormat.class); conf.setOutputFormat(TextOutputFormat.class); ! FileInputFormat.setInputPaths(conf, new Path(args[0])); FileOutputFormat.setOutputPath(conf, new Path(args[1])); ! JobClient.runJob(conf); } } select count(*) from ( select regexp_split_to_table(str, s+') from text_table ) t; MapReduceによるWordCount SQLによるWordCount
- 27. 実際SQLが勝った。 Now SQL beats MapReduce
- 28. Q2. Hadoopと並列RDBは どっちがいいんですか！！！ Which is good, Hadoop or parallel RDB?
- 29. 速度は並列RDB データ構造はHadoop Parallel RDB is faster; Hadoop is more flexible
- 32. Hadoopは並列RDBに 似てきている DB filesystem backend parser, planner HDFS impala be impala fe Hadoop resembles to parallel RDB now
- 33. Hybrid DB comes in near future
- 36. MapReduceは 並列処理にJavaやCを はさみこめる MapReduce has better extendability
- 38. SQLからMapReduce呼べる select count(distinct user_id) from npath( on clicks partition by user_id order by timestamp mode(overlapping) pattern( H.S.P ) symbols( page_type = home AS H, page_type = search AS S, page_type = product AS P) result(first(user_id of H) as user_id) ); 最近、Hiveにもnpath入りました (MatchPath) You can combine MapReduce with SQL
- 39. Easy & Handy SQL + Extendable MapReduce
- 40. ☆ 最後にポエム ☆
- 41. よいものはよい Great product is anywhere
- 42. だが知識は偏在している but knowledge is maldistributed
- 43. OSS World Enterprise World Hadoop Ruby Python Excel 並列RDB Windows markdown VB Git Java
- 44. Cross the border
- 45. end