[B! s3] kanbayashiã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

kanbayashi id:kanbayashi

s3ã«é–¢ã™ã‚‹kanbayashiã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (10)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

botoã‚’ä½¿ã£ã¦S3ã«ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã™ã‚‹ - åã£ãŸè¨€èªžä¿¡è€…ã®åž‚ã‚Œæµã—
ä½•åº¦ã‚‚ã‚„ã£ãŸã“ã¨ã‚ã‚‹ã®ã«ã€ãƒ–ãƒã‚°ã«æ›¸ã„ã¦ãªã‹ã£ãŸã£ã½ã„ã®ã§æ›¸ãã€‚ Pythonã‹ã‚‰botoã‚’ä½¿ã£ã¦AWSã®S3ã«ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã™ã‚‹æ–¹æ³•ã€‚ è©¦ã—ãŸPythonã®ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã¯2.6ã€‚botoã¯2.0rc1 botoã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ« pipã‚„easy_installã§å…¥ã‚Œã‚Œã°ã‚ˆã„ã€‚ $ pip install boto S3ã¸ã®ãƒã‚±ãƒƒãƒˆã®ç”¨æ„ ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰å…ˆã®ãƒã‚±ãƒƒãƒˆã‚’ç”¨æ„ã™ã‚‹ã€‚ botoã®APIã§ä½œæˆã™ã‚‹ã“ã¨ã‚‚ã§ãã‚‹ãŒã€ä»Šå›žã¯AWSã®ã‚³ãƒ³ã‚½ãƒ¼ãƒ«ã‹ã‚‰ä½œæˆã—ã¦ãŠãã€‚ S3ã®ã‚¿ãƒ–ã®Bucketsã‹ã‚‰"Create Bucket"ã§ãƒã‚±ãƒƒãƒˆåã‚’å…¥åŠ›ã—ã¦ä½œæˆã€‚ ä»Šå›žã¯ test-boto.nullpobug.com ã¨ã—ãŸã€‚(åˆ¥ã«ãƒ‰ãƒ¡ã‚¤ãƒ³åã«ãªã£ã¦ãªãã¦ã‚‚ã‚ˆã„ãŒã€CNAMEå‰²ã‚Šå½“ã¦ã§ã®ã‚¢ã‚¯ã‚»ã‚¹ã‚’ä½¿ã†ãªã‚‰ãƒ‰ãƒ¡ã‚¤ãƒ³åã«ã™ã‚‹å¿…è¦ãŒã‚ã‚‹) ã‚¢ã‚¯ã‚»ã‚¹ã‚ãƒ¼ã®ç¢ºèª AWSã®APIã‚’åˆ©ç”¨ã™ã‚‹ã«ã¯ã‚¢ã‚¯ã‚»ã‚¹ã‚ãƒ¼ã¨ã‚·ãƒ¼
kanbayashi 2016/11/15
aws

s3

boto

cli

ã‚³ãƒžãƒ³ãƒ‰ãƒ©ã‚¤ãƒ³ãƒ„ãƒ¼ãƒ«
ãƒªãƒ³ã‚¯
AWS CLIã®S3ã‚³ãƒžãƒ³ãƒ‰ã¾ã¨ã‚ - TASK NOTES
æ™®æ®µ AWS CLI ã‚’ä½¿ã†ã“ã¨ã¯ãã‚“ãªã«ãªã„ã‚“ã§ã™ãŒã€S3 ã‚³ãƒžãƒ³ãƒ‰ã ã‘ã¯ã‚ˆãä½¿ã†ã®ã§ã¾ã¨ã‚ã¾ã—ãŸã€‚ã¨ã„ã£ã¦ã‚‚å…¨ã¦ã§ã¯ãªãã€ãƒ•ã‚¡ã‚¤ãƒ«ã‚„ãƒ‡ã‚£ãƒ¬ã‚¯ãƒˆãƒªæ“ä½œã«é–¢ã™ã‚‹éƒ¨åˆ†ã§ã™ã€‚ åŸºæœ¬ã‚³ãƒžãƒ³ãƒ‰ AWS CLI ã® S3 ã‚³ãƒžãƒ³ãƒ‰ã¯ä»¥ä¸‹ã®å½¢å¼ã§å®Ÿè¡Œã—ã¾ã™ã€‚ aws s3 <Command> [<Arg> ...] åŸºæœ¬çš„ã« Arg ã®ç®‡æ‰€ã¯ãƒ‘ã‚¹ã‚’å…¥åŠ›ã™ã‚‹ã“ã¨ã«ãªã‚‹ã®ã§ã™ãŒã€ãƒãƒ¼ã‚«ãƒ«ã®ãƒ‘ã‚¹ã¯ã€Œãƒãƒ¼ã‚«ãƒ«ãƒ•ã‚¡ã‚¤ãƒ«ã¾ãŸã¯ãƒ‡ã‚£ãƒ¬ã‚¯ãƒˆãƒªã®çµ¶å¯¾ãƒ‘ã‚¹ã¾ãŸã¯ç›¸å¯¾ãƒ‘ã‚¹ã€ã€S3ã®ãƒ‘ã‚¹ã¯ã€Œs3://<bucket>/<prefix>/<object>ã€ã¨ãªã‚Šã¾ã™ã€‚prefix ã¨ã¯ãƒ‡ã‚£ãƒ¬ã‚¯ãƒˆãƒªã‚„ãƒ•ã‚©ãƒ«ãƒ€ã®ã‚¤ãƒ¡ãƒ¼ã‚¸ã§ã™ã€‚ Command Description cp ã‚ªãƒ–ã‚¸ã‚§ã‚¯ãƒˆã®ã‚³ãƒ”ãƒ¼ ls ã‚ªãƒ–ã‚¸ã‚§ã‚¯ãƒˆã®ä¸€è¦§ mb S3ãƒã‚±ãƒƒãƒˆã®ä½œæˆ mv ã‚ªãƒ–ã‚¸ã‚§ã‚¯ãƒˆã®ç§»å‹• rb S3ãƒã‚±ãƒƒãƒˆã®å‰Šé™¤ rm ã‚ªãƒ–ã‚¸ã‚§ã‚¯ãƒˆã®å‰Šé™¤ sync
kanbayashi 2016/11/15
aws

s3
ãƒªãƒ³ã‚¯
Forums
kanbayashi 2011/05/06
elastic map reduce

hdfs

hadoop

emr

s3

amazon s3
ãƒªãƒ³ã‚¯
Linuxç«¯æœ«ã‹ã‚‰Amazon S3ã«ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã™ã‚‹(s3cmdä½¿ç”¨) at nkjmkzk.net
Virtualization, Operating System, Storage, Cloud Computingæ€ã†ã¨ã“ã‚ã‚ã£ã¦4GByteã®ãƒ•ã‚¡ã‚¤ãƒ«ã‚’S3ã«ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã—ã‚ˆã†ã¨ã—ã¦ã„ã¾ã—ãŸã€‚å½“åˆä¸€ç•ªãƒ¦ãƒ¼ã‚¶ãƒ¼ãƒ•ãƒ¬ãƒ³ãƒ‰ãƒªãƒ¼ãªWeb Consoleã‚’ä½¿ç”¨ã—ã¦ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã‚’è©¦ã¿ã¾ã—ãŸã€‚ ã—ã‹ã—ãªãŒã‚‰300MByteã»ã©ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã—ãŸã¨ã“ã‚ã§åŽŸå› ä¸æ˜Žã®Errorã¨ãªã‚Šã†ã¾ãã„ãã¾ã›ã‚“ã§ã—ãŸã€‚AWSãƒ–ãƒã‚°ã§ã‚‚å¤§å®¹é‡ãƒ•ã‚¡ã‚¤ãƒ«ã®ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã«ã¯ã„ã•ã•ã‹æ”¹å–„ã®ä½™åœ°ãŒã‚ã‚‹æ—¨ãŒè¨˜è¼‰ã•ã‚Œã¦ã„ã¾ã™ã€‚ http://aws.typepad.com/aws_japan/2010/11/amazon-s3-multipart-upload.html åŒè¨˜äº‹å†…ã®Multipart Uploadã‚’è©¦ã¿ã‚ˆã†ã‹ã¨æ€ã„ã¾ã—ãŸãŒã€ã‚ã–ã‚ã–ãƒ—ãƒã‚°ãƒ©ãƒ ã‚’æ›¸ãã»ã©ç¹°ã‚Šè¿”ã—è¡Œã†ä½œæ¥ã§ã‚‚ãªã„ã®ã§ä½•ã‹ãŠæ‰‹è»½ãªãƒ„ãƒ¼ãƒ«ã¯ãªã„ã‹ã¨æ€ã£ã¦ã„ãŸã‚‰
kanbayashi 2011/05/06
s3cmd

install

setup

storage

amaon s3

s3

cloud

command line

api
ãƒªãƒ³ã‚¯
Amazon S3 Tools: Command Line S3 Client and S3 Backup for Windows, Linux: s3cmd, s3express
Amazon S3 Tools: Command Line S3 Client Software and S3 Backup AWS S3 Command Line Clients for Windows, Linux, Mac. Backup to S3, upload, retrieve, query data on Amazon S3. S3cmd : Command Line S3 Client and Backup for Linux and Mac Amazon S3 is a reasonably priced data storage service. Ideal for off-site file backups, file archiving, web hosting and other data storage needs. It is generally more
kanbayashi 2011/05/06
s3

elastic map reduce

storage

cloud

amazon s3

api

command line

s3cmd
ãƒªãƒ³ã‚¯
CUIã§Amazon S3ã‚’æ“ä½œã™ã‚‹Â·s3cmd MOONGIFT
Amazon EC2ã‚’ä½¿ã£ã¦ã„ã‚‹äººã§ã‚ã‚Œã°ã€æ·»ä»˜ãƒ•ã‚¡ã‚¤ãƒ«ã‚„ç”»åƒã®ç½®ãå ´ã¨ã—ã¦Amazon S3ã‚’ä½¿ã£ã¦ã„ã‚‹äººã¯å¤šã„ã¯ãšã ã€‚ä»–ã«ã‚‚ãƒãƒƒã‚¯ã‚¢ãƒƒãƒ—ç”¨é€”ã‚„ã€å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’ä¿å˜ã—ã¦ãŠãå ´æ‰€ã¨ã—ã¦Amazon S3ã¯ä¾¿åˆ©ãªWebã‚µãƒ¼ãƒ“ã‚¹ã ã€‚ ãƒ˜ãƒ«ãƒ— ã ãŒAmazon S3ã¯Web APIã«ç‰¹åŒ–ã—ã¦ãŠã‚Šãã®æ“ä½œã«ã¯ãƒ©ã‚¤ãƒ–ãƒ©ãƒªã‚’ä»‹ã—ã¦è¡Œã†ã®ãŒä¸€èˆ¬çš„ã ã€‚ã ãŒã“ã‚Œã§ã¯è‡ªå‹•åŒ–ã™ã‚‹éš›ã«ã¯ãƒ—ãƒã‚°ãƒ©ãƒ ã‚’çµ„ã‚€å¿…è¦ãŒã‚ã‚Šé¢å€’ã ã€‚ãã“ã§Amazon S3ã‚’ã‚¿ãƒ¼ãƒŸãƒŠãƒ«ã‹ã‚‰æ“ä½œã™ã‚‹s3cmdã‚’ä½¿ãŠã†ã€‚ ä»Šå›žç´¹ä»‹ã™ã‚‹ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ãƒ»ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã¯s3cmdã€Amazon S3ã®CUI ãƒ•ãƒãƒ³ãƒˆã‚¨ãƒ³ãƒ‰ã ã€‚ s3cmdã¯Pythonã§ä½œã‚‰ã‚ŒãŸã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã§ã€Amazon S3ã®æ“ä½œã‚’ã‚¿ãƒ¼ãƒŸãƒŠãƒ«ã‹ã‚‰è¡Œã†ã“ã¨ãŒã§ãã‚‹ã€‚ã“ã‚Œã§ã‚ã‚Œã°ãƒ•ã‚¡ã‚¤ãƒ«ã®é€ä¿¡ã‚„å—ä¿¡ãªã©ã‚’ã‚³ãƒžãƒ³ãƒ‰ã§è¡Œãˆã‚‹ã‚ˆã†ã«ãªã‚‹ã®ã§è‡ªå‹•åŒ–ã‚‚å®¹æ˜“ã«ãªã‚‹ã ã‚ã†ã€‚ ãƒªã‚¹ãƒˆã‚³ãƒžãƒ³ãƒ‰ã‚’å®Ÿè¡Œã—ãŸæ‰€
kanbayashi 2011/05/06
amazon s3

s3

cloud

ruby

storage

api

command line

elastic map reduce
ãƒªãƒ³ã‚¯
Ruby/Amazon S3ã‚’ä½¿ã† - ä¿ºã®åŸºåœ°
ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã¨è£½é€ å¹´æœˆæ—¥ 2009å¹´09æœˆ08æ—¥ Ruby 1.8.6 aws-s3 0.6.2 WindowsXP SP3 â–²Â â–¼ ç’°å¢ƒæ§‹ç¯‰ Amazonã®ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã‚µãƒ¼ãƒ“ã‚¹ã‚’S3ã‹ã‚‰ä½¿ã†ãƒ©ã‚¤ãƒ–ãƒ©ãƒªã€€aws-s3ã€€ãŒã‚ã‚‹ã®ã§ã“ã„ã¤ã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã™ã‚‹ gem install aws-s3 ã“ã„ã¤ã«ã¯3ã¤ã®ä¾å˜æ€§ãŒã‚ã‚Šgemãªã‚‰è‡ªå‹•çš„ã«ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã•ã‚Œã‚‹ã‚“ã ãŒã€ä¸€å¿œä¹—ã£ã‘ã¦ãŠã xml-simple http://rubyforge.org/projects/xml-simple/ builder http://rubyforge.org/projects/builder/ mime-types http://rubyforge.org/projects/mime-types/ â–²Â â–¼ æŽ¥ç¶šè¨å®š ã¾ãšæŽ¥ç¶šè¨å®šã‚’è¡Œã†ã€‚ã“ã„ã¤ã¯Amazonã®ã‚µãƒ¼ãƒ“ã‚¹ã§æ¯Žåº¦ä½¿ã†ã€ç”Ÿæˆidã¨keyã®
kanbayashi 2011/05/06
ruby

aws

s3

ec2

elastic map reduce

mapreduce
ãƒªãƒ³ã‚¯
ã€Amazon S3ã€‘S3ã®ãƒ„ãƒ¼ãƒ«S3Foxã®ã‚»ãƒƒãƒˆã‚¢ãƒƒãƒ—æ–¹æ³•
ä»Šå›žã¯Amazon Simple Storage Service(S3)ã®ãƒ„ãƒ¼ãƒ« ã€ŒFirefox Organizer for Amazon S3 and Amazon CloudFront(S3Fox)ã€ ã®ã‚»ãƒƒãƒˆã‚¢ãƒƒãƒ—æ–¹æ³•ã«ã¤ã„ã¦ã§ã™ã€‚ S3ã¯ãƒãƒƒã‚¯ã‚¢ãƒƒãƒ—ç”¨ã®ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã«åˆ©ç”¨ã•ã‚Œã‚‹ä¾‹ãŒã‚ˆãç´¹ä»‹ã•ã‚Œã¦ã„ã¾ã™ãŒ ã‚‚ã†ã²ã¨ã¤å¤§äº‹ãªå½¹å‰²ãŒã‚ã‚Šã¾ã™ã€‚ EC2ã§ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã«å¿…è¦ãªã‚½ãƒ•ãƒˆã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã—ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºã—ãŸå ´åˆã€ ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹çµ‚äº†(Terminate)å¾Œã€å†èµ·å‹•ã™ã‚‹ç‚ºã«ã¯è‡ªå‰ã®AMIã¨ã—ã¦ä¿å˜ã—ã¦ãŠãå¿…è¦ãŒã‚ã‚Šã¾ã™ã€‚ ãã®ä¿å˜å…ˆãŒS3ã«ãªã‚Šã¾ã™ã€‚ S3ã‚’æ“ä½œã™ã‚‹ã®ã«ä¾¿åˆ©ãªã®ãŒFireFoxã®ã‚¢ãƒ‰ã‚ªãƒ³ã§ã‚ã‚‹S3Foxã§ã™ã€‚ S3Foxã®ã‚»ãƒƒãƒˆã‚¢ãƒƒãƒ—ã¯ä»¥ä¸‹ã®æ‰‹é †ã«ãªã‚Šã¾ã™ã€‚ 1. Firefoxã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã€‚ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã•ã‚Œã¦ã„ãªã‘ã‚Œã°ã€ã“ã“ã‹ã‚‰ãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰ã—ã¦ä¸‹ã•ã„ã€‚ 2. æ¬¡ã«S3F
kanbayashi 2011/04/10
s3fox

amazon s3

s3
ãƒªãƒ³ã‚¯
ã‚¯ãƒ©ã‚¦ãƒ‰åž‹ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã€ŒAmazon S3ã€ã¯å®‰ã„ã‹ï¼Ÿ
Amazon S3ã¯å®‰ã„ã‹é«˜ã„ã‹ã¨ã„ã†è°è«–ã‚’ã€ã“ã®2æ—¥é–“ã§ç«‹ã¦ç¶šã‘ã«3å€‹æ‰€ã§è€³ï¼ˆç›®ï¼‰ã«ã—ãŸã€‚ 1äººã¯ãƒãƒƒãƒˆã‚µãƒ¼ãƒ“ã‚¹ãƒ™ãƒ³ãƒãƒ£ãƒ¼ã®CTOã§ã€ã‚¯ãƒ©ã‚¦ãƒ‰ã¸ã®ç§»è¡Œã‚’æ¤œè¨Žã—ã¦ã„ã‚‹ã‚‚ã®ã®ã€Amazon S3ã‚„EC2ã®å®Ÿéš›ã®ä¾¡æ ¼ã‚„ä½¿ã„å‹æ‰‹ãŒã‚ˆãåˆ†ã‹ã‚‰ãªã„ã¨ã„ã†è©±ã ã£ãŸã€‚ã‚³ã‚¹ãƒˆçš„ã«è¦‹åˆã†ãªã‚‰ã€ã‚‚ã†ã‚µãƒ¼ãƒé‹ç”¨ã«ç…©ã‚ã•ã‚ŒãŸããªã„ã¨æ¼ã‚‰ã—ã¦ã„ãŸã€‚ ã‚‚ã†1ã¤ã¯ã€ãƒ™ãƒ³ãƒãƒ£ãƒ¼ã‚ãƒ£ãƒ”ã‚¿ãƒ«ã¨ã—ã¦çŸ¥ã‚‰ã‚Œã‚‹Yã‚³ãƒ³ãƒ“ãƒãƒ¼ã‚¿ãƒ¼ã®æŽ²ç¤ºæ¿ã§è¦‹ã‹ã‘ãŸã€ŒDoes Amazon S3 really save moneyï¼Ÿã€ã§å§‹ã¾ã‚‹è°è«–ã€‚1GBå½“ãŸã‚Š1ã‚«æœˆã§0.15ãƒ‰ãƒ«ã¨èžãã¨å®‰ã„ã‚ˆã†ã«ã‚‚æ€ãˆã‚‹ãŒã€1TBã®æœˆé¡ã¯150ãƒ‰ãƒ«ã€2å¹´ã«ã™ã‚‹ã¨3600ãƒ‰ãƒ«ï¼ˆ1ãƒ‰ãƒ«92å††ã¨ã—ã¦ç´„33ä¸‡å††ï¼‰ã«ã‚‚ãªã‚‹ã€‚ã—ã‹ã‚‚è»¢é€é‡ã«å¿œã˜ãŸèª²é‡‘ã‚‚ã‚ã‚‹ãŸã‚ã€å®Ÿéš›ã«ã¯ã“ã‚Œä»¥ä¸Šã«ãªã‚‹ã€‚ä»Šã‚„1TBã®ãƒ‰ãƒ©ã‚¤ãƒ–å˜ä¾¡ã¯1ä¸‡å††ã‚’å‰²ã£ã¦ã„ã¦ã€2å¹´ã§3600ãƒ‰ãƒ«ã¨ã¯æ¯”è¼ƒã«ã‚‚ãªã‚‰ãªã„ã€‚ã€Œã‚¯ãƒ©ã‚¦ãƒ‰
kanbayashi 2011/04/10
amzon s3

s3

aws

storage

cloud

cost

elastic map reduce
ãƒªãƒ³ã‚¯
Googleã€Amazon S3ã®ãƒ©ã‚¤ãƒãƒ«ã€Google Storageã‚’I/Oã‚«ãƒ³ãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹ã§ç™ºè¡¨ã¸
kanbayashi 2010/05/19
google

amazon

s3

cloud

storage
ãƒªãƒ³ã‚¯
1