Dask & Ray
Another tool that deserves mention: Dask. The idea: recreate the API from Pandas (and NumPy and Scikit-Learn) as much as possible, but do lazy evaluation, and allow distributed computation (like Spark). As a demo, let's recreate the Most-Viewed Wikipedia Pages solution in Dask… [Complete code: dask_wikipedia.py] Dask Basic setup: import the Dask DataFrame functionality, and create a client to work
1100万行・32GB超ã®å·¨å¤§CSVファイルã®åŸºæœ¬çµ±è¨ˆé‡ã‚’4GBメモリマシンã§ç®—出ã™ã‚‹ - Qiita
ã¯ã˜ã‚ã« ã“ã®è¨˜äº‹ã¯ï¼ŒKaggle Advent Calendar 2022第6日目ã®è¨˜äº‹ã«ãªã‚Šã¾ã™ã€‚ 本記事ã§ã¯ã€ 32GB超ã®CSVデータã®åŸºæœ¬çµ±è¨ˆé‡ã‚’ã€å°è¦æ¨¡ãƒžã‚·ãƒ³ã§ã‚‚çœãƒ¡ãƒ¢ãƒªã‹ã¤é«˜é€Ÿã«è¨ˆç®—ã™ã‚‹ãƒ†ã‚¯ãƒ‹ãƒƒã‚¯ ã«ã¤ã„ã¦è§£èª¬ã—ã¾ã™ã€‚ Kaggleコンペã«é™ã‚‰ãšã€ マシンスペックãŒä½Žã„ãŸã‚ã€å¤§ããªãƒ‡ãƒ¼ã‚¿ã‚»ãƒƒãƒˆã‚’満足ã«å‡¦ç†ã§ããšå›°ã£ã¦ã„ã‚‹ 毎回行ã†ãƒ•ã‚¡ã‚¤ãƒ«èªã¿è¾¼ã¿ãŒé…ã„ã®ã§ã€ã‚‚ã£ã¨é«˜é€ŸåŒ–ã—ãŸã„ âš¡ ã¨ã„ã£ãŸæ‚©ã¿ã‚„課題を抱ãˆã¦ã„ã‚‹æ–¹ã®å‚考ã«ãªã‚Œã°å¹¸ã„ã§ã™ã€‚ モãƒãƒ™ãƒ¼ã‚·ãƒ§ãƒ³ データ分æžæ¥å‹™ã‚„Kaggleç‰ã®ã‚³ãƒ³ãƒšãƒ†ã‚£ã‚·ãƒ§ãƒ³ã§åˆã‚ã¦ã®ãƒ‡ãƒ¼ã‚¿ã‚»ãƒƒãƒˆã‚’扱ã†å ´åˆã€ã„ããªã‚Šæ©Ÿæ¢°å¦ç¿’アルゴリズムを行ã†ã“ã¨ã¯ã¾ãšç„¡ãã€æœ€åˆã«ãƒ‡ãƒ¼ã‚¿è¦³å¯Ÿã‚’è¡Œã†ã®ãŒä¸€èˆ¬çš„ã§ã™ã€‚ テーブルデータã§ã‚ã‚Œã°ã€å„カラムã®åŸºæœ¬çµ±è¨ˆé‡ï¼ˆæœ€å°å€¤ã€æœ€å¤§å€¤ã€å¹³å‡ã€åˆ†æ•£ã€å››åˆ†ä½æ•°ï¼‰ãªã©ã‚’計算・å¯è¦–化ã—ã€ãƒ‡ãƒ¼ã‚¿ã‚¯ãƒ¬ãƒ³ã‚¸ãƒ³ã‚°ã®è¦å¦ã‚„特徴é‡è¨è¨ˆã®æ–¹é‡ãªã©ã‚’検
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Understanding Performance — Dask documentation
Run a single task per worker with dask-mpi
MVAPICH2 for Intel MIC
Python for High Performance Computing: Dask for parallel data operations
dask-examples/machine-learning.ipynb at main · dask/dask-examples
dask-examples/dataframe.ipynb at main · dask/dask-examples
dask-mpi/dask_mpi/core.py at main · dask/dask-mpi
Start dask-mpi problem
How to Create dask-scheduler.json · Issue #3416 · dask/dask
Dask on HPC - Initial Work
GitHub - wmlba/ECS-Dask: Dask on ECS Fargate
NeMo Data Curator - NVIDIA Docs
blog/dedup.md at main · huggingface/blog
Large-scale Near-deduplication Behind BigCode
Dask, Pandas, and GPUs: first steps
Dask-MPI with GPUs — Dask-MPI 2022.4.0+68.g551707e.dirty documentation
How Dask-MPI Works — Dask-MPI 2022.4.0+68.g551707e.dirty documentation
GitHub - dask/dask-mpi: Deploy Dask using MPI4Py
{{#tags}}- {{label}}
{{/tags}}