20160708 データ処理のプラットフォームとしてのpython 札幌
•
5 likes•3,287 views
インサイトテクノロジーさん主催の[7月8日@札幌] Sapporo Tech Barでの発表スライドです。 http://www.db-tech-showcase.com/events-seminars/db-tech-salon/20160708_sapporo_tech_bar
![1 1.2 1000000L Python2
‘abc’ u’ ’ Python2
[1, 2, 3,‘foo’,‘bar’,‘foo’]
(1, 2, 3,‘foo’,‘bar’,‘foo’)
{‘k1’:‘value1’,‘k2’:‘value2’}
set(1, 2, 3,‘foo’,‘bar’)](https://image.slidesharecdn.com/20160708python-160709173450/85/20160708-python-27-320.jpg)
![•
•
• split
s = ‘foo, bar, baz’
items = s.split(‘,’)
print items[0]
print items[-1]
print items[0][-2:]](https://image.slidesharecdn.com/20160708python-160709173450/85/20160708-python-28-320.jpg)
![•
list comprehension
•
dictionary comprehension
• lambda map, reduce, filter
sList = [‘foo’, ‘bar’, ‘baz’]
lList = [len(s) for s in sList]
lList = map(lambda s:len(s),
sList)
lDict = {s:len(s) for s in sList}](https://image.slidesharecdn.com/20160708python-160709173450/85/20160708-python-29-320.jpg)
![• http://sinhrks.hatenablog.com/entry/2015/01/28/073327
0 1
import pandas as pd
df[‘nValue’] = df[‘value’] / sum(df[‘value’])
id value color
sapporo 43 red
osaka 42 pink
matsumoto 40 green
id value color nValue
sapporo 43 red 0.344
osaka 42 pink 0.336
matsumoto 40 green 0.32
Python](https://image.slidesharecdn.com/20160708python-160709173450/85/20160708-python-33-320.jpg)
20160708 データ処理のプラットフォームとしてのpython 札幌
- 1. Python Sky
- 3. • • Python 2000 (**) • db tech showcase MongoDB • • FB: Ryuji Tamagawa • Twitter : tamagawa_ryuji
- 5. 2015
- 6. 2016
- 8. • Python • • Python • NumPy, SciPy, matplotlib, Pandas • Python • scikit-learn • TensorFlow • Python IPython, Jupyter notebook, Spyder, VisualStudio • Python • Python • Pandas • Spark - PySpark DataFrame API • matplotlib
- 9. Part 1 : Python
- 10. Python • • Google Guido Google Google 1 • NumPy, SciPy, matplotlib → Pandas • • -2000 Linux -2010 Web Trac Google
- 12. Python • • pyODBC • Web WSGI
- 13. Python • 2.x 3.x 32bit 64bit 64bit • 2.x • 3.x 3 • 2.x 3.x
- 14. • Ruby? • R? • Java? • Scala?
- 15. Python • Python ’CPython’ JIT PyPy JVM Jython .Net IronPython • CPython • CPython 2 • C • processing PySpark
- 16. Python • Python • 1 Linux Mac OS Python Python Mac • Python pip 3.x Python 2.7.9 2.x Python pip Linux Python pip yum apt • Python Anaconda Python conda • python 2016 http://qiita.com/y__sama/items/5b62d31cb7e6ed50f02c
- 17. NumPy, SciPy, matplotlib, Pandas • • NumPy SciPy • Pandas Pandas Pandas NumPy • Anaconda Python
- 20. Python IPython Jupyter, … IDE Spyder, Rodeo Visual Studio, PyCharm, PyDev
- 22. • IPython • • • Anaconda • pip
- 23. • Jupyter Notebook • Python • IPython Notebook Python • Apache Zeppelin http:// zeppelin.apache.org
- 24. IDE • R RStudio • IDE • • 2 Spyder Rodeo • Spyder
- 25. • • Visual Studio • Eclipse PyDev • PyCharm •
- 26. Part 2 : Python
- 27. 1 1.2 1000000L Python2 ‘abc’ u’ ’ Python2 [1, 2, 3,‘foo’,‘bar’,‘foo’] (1, 2, 3,‘foo’,‘bar’,‘foo’) {‘k1’:‘value1’,‘k2’:‘value2’} set(1, 2, 3,‘foo’,‘bar’)
- 28. • • • split s = ‘foo, bar, baz’ items = s.split(‘,’) print items[0] print items[-1] print items[0][-2:]
- 29. • list comprehension • dictionary comprehension • lambda map, reduce, filter sList = [‘foo’, ‘bar’, ‘baz’] lList = [len(s) for s in sList] lList = map(lambda s:len(s), sList) lDict = {s:len(s) for s in sList}
- 30. Pandas • Pandas • matplotlib / seaborn • NumPy SciPy Python • Pandas + matplotlib OK Pandas NumPy NumPy / SciPy
- 31. Pandas • Pandas DataFrame • R • RDB 2 • index Series Columns Columns Series Series SeriesIndex
- 32. Pandas I/O • CSV JSON RDB Excel • column • RDB • import pandas as pd pd.read_csv(<filename>) pd.read_json(<filename>) pd.to_csv(<filename>) pd.to_excel(<filename>) # pd.to_clipboard()
- 33. • http://sinhrks.hatenablog.com/entry/2015/01/28/073327 0 1 import pandas as pd df[‘nValue’] = df[‘value’] / sum(df[‘value’]) id value color sapporo 43 red osaka 42 pink matsumoto 40 green id value color nValue sapporo 43 red 0.344 osaka 42 pink 0.336 matsumoto 40 green 0.32 Python
- 34. Spark - PySpark DataFrame API • Python • Spark PySpark findSpark Spark • Python Spark API DataFrame API • Spark Pandas Spark PySpark Spark node Spark node Spark node Spark node driver
- 35. matplotlib / seaborn • • Python NumPy / Pandas • Jupyter Notebook Spyder
- 36. Questions ?