S3cmd tool for Amazon Simple Storage Service (S3)

=================================================

Author:

Michal Ludvig <[email protected]>

S3tools / S3cmd project homepage:

http://s3tools.sourceforge.net

Amazon S3 homepage:

http://aws.amazon.com/s3

What is Amazon S3

-----------------

Amazon S3 provides a managed internet-accessible storage

service where anyone can store any amount of data and

retrieve it later again. Maximum amount of data in one

"object" is 5GB, maximum number of objects is not limited.

S3 is a paid service operated by the well known Amazon.com

internet book shop. Before storing anything into S3 you

must sign up for an "AWS" account (where AWS = Amazon Web

Services) to obtain a pair of identifiers: Access Key and

Secret Key. You will need to give these keys to S3cmd.

Think of them as if they were a username and password for

your S3 account.

Pricing explained

-----------------

At the time of this writing the costs of using S3 are:

1) US$0.15 per GB-Month of storage used.

2) US$0.20 per GB of data transferred.

If for instance on 1st of January you upload 2GB of

photos in JPEG from your holiday in New Zealand, at the

end of January you will be charged $0.30 for using 2GB of

storage space for a month and $0.40 for transferring 2GB

of data. That comes to $0.70 for a complete backup of your

precious holiday pictures.

In February you don't touch it. Your data are still on S3

servers so you pay $0.30 for those two gigabytes, but not

a single cent will be charged for any transfer. That comes

to $0.30 as an ongoing cost of your backup. Not too bad.

In March you allow anonymous read access to some of your

pictures and your friends download, say, 500MB of them.

As the files are owned by you, you are responsible for the

costs incurred. That means at the end of March you'll be

charged $0.30 for storage plus $0.07 for the traffic

generated by your friends.

There is no minimum monthly contract or a setup fee. What

you use is what you pay for. At the beginning my bill used

to be like US$0.03 or even nil.

That's the pricing model of Amazon S3 in a nutshell. Check

Amazon S3 homepage at http://aws.amazon.com/s3 for more

details.

Needless to say that all these money are charged by Amazon

itself, there is obviously no payment for using S3cmd :-)

Amazon S3 basics

----------------

Files stored in S3 are called "objects" and their names are

officially called "keys". Each object belongs to exactly one

"bucket". Buckets are kind of directories or folders with

some restrictions: 1) each user can only have 100 buckets at

the most, 2) bucket names must be unique amongst all users

of S3, 3) buckets can not be nested into a deeper

hierarchy and 4) a name of a bucket can only consist of basic

alphanumeric characters plus dot (.) and dash (-). No spaces,

no accented or UTF-8 letters, etc.

On the other hand there are almost no restrictions on object

names ("keys"). These can be any UTF-8 strings of up to 1024

bytes long. Interestingly enough the object name can contain

forward slash character (/) thus a "my/funny/picture.jpg" is

a valid object name. Note that there are not directories nor

buckets called "my" and "funny" - it is really a single object

name called "my/funny/picture.jpg" and S3 does not care at

all that it _looks_ like a directory structure.

To describe objects in S3 storage we invented a URI-like

schema in the following form:

s3://BUCKET/OBJECT

See the HowTo later in this document for example usages of

this S3-URI schema.

Simple S3cmd HowTo

------------------

1) Register for Amazon AWS / S3

Go to http://aws.amazon.com/s3, click the "Sign up

for web service" button in the right column and work

through the registration. You will have to supply

your Credit Card details in order to allow Amazon

charge you for S3 usage.

At the end you should posses your Access and Secret Keys

2) Run "s3cmd --configure"

You will be asked for the two keys - copy and paste

them from your confirmation email or from your Amazon

account page. Be careful when copying them! They are

case sensitive and must be entered accurately or you'll

keep getting errors about invalid signatures or similar.

3) Run "s3cmd ls" to list all your buckets.

As you just started using S3 there are no buckets owned by

you as of now. So the output will be empty.

4) Make a bucket with "s3cmd mb s3://my-new-bucket-name"

As mentioned above bucket names must be unique amongst

_all_ users of S3. That means the simple names like "test"

or "asdf" are already taken and you must make up something

more original. I sometimes prefix my bucket names with

my e-mail domain name (logix.cz) leading to a bucket name,

for instance, 'logix.cz-test':

~$ s3cmd mb s3://logix.cz-test

Bucket 'logix.cz-test' created

5) List your buckets again with "s3cmd ls"

Now you should see your freshly created bucket

~$ s3cmd ls

2007-01-19 01:41 s3://logix.cz-test

6) List the contents of the bucket

~$ s3cmd ls s3://logix.cz-test

Bucket 'logix.cz-test':

~$

It's empty, indeed.

7) Upload a file into the bucket

~$ s3cmd put addressbook.xml s3://logix.cz-test/addrbook.xml

File 'addressbook.xml' stored as s3://logix.cz-test/addrbook.xml (123456 bytes)

8) Now we can list the bucket contents again

~$ s3cmd ls s3://logix.cz-test

Bucket 'logix.cz-test':

2007-01-19 01:46 120k s3://logix.cz-test/addrbook.xml

9) Retrieve the file back and verify that its hasn't been

corrupted

~$ s3cmd get s3://logix.cz-test/addrbook.xml addressbook-2.xml

Object s3://logix.cz-test/addrbook.xml saved as 'addressbook-2.xml' (123456 bytes)

~$ md5sum addressbook.xml addressbook-2.xml

39bcb6992e461b269b95b3bda303addf addressbook.xml

39bcb6992e461b269b95b3bda303addf addressbook-2.xml

Checksums of the original file matches the one of the

retrieved one. Looks like it worked :-)

10) Clean up: delete the object and remove the bucket

~$ s3cmd rb s3://logix.cz-test

ERROR: S3 error: 409 (Conflict): BucketNotEmpty

Ouch, we can only remove empty buckets!

~$ s3cmd del s3://logix.cz-test/addrbook.xml

Object s3://logix.cz-test/addrbook.xml deleted

~$ s3cmd rb s3://logix.cz-test

Bucket 'logix.cz-test' removed

Hints

-----

The basic usage is as simple as described in the previous

section.

You can increase the level of verbosity with -v option and

if you're really keen to know what the program does under

its bonet run it with -d to see all 'debugging' output.

After configuring it with --configure all available options

are spitted into your ~/.s3cfg file. It's a text file ready

to be modified in your favourite text editor.

Multiple local files may be specified for "s3cmd put"

operation. In that case the S3 URI should only include

the bucket name, not the object part:

~$ s3cmd put file-* s3://logix.cz-test/

File 'file-one.txt' stored as s3://logix.cz-test/file-one.txt (4 bytes)

File 'file-two.txt' stored as s3://logix.cz-test/file-two.txt (4 bytes)

Alternatively if you specify the object part as well it

will be treated as a prefix and all filenames given on the

command line will be appended to the prefix making up

the object name. However --force option is required in this

case:

~$ s3cmd put --force file-* s3://logix.cz-test/prefixed:

File 'file-one.txt' stored as s3://logix.cz-test/prefixed:file-one.txt (4 bytes)

File 'file-two.txt' stored as s3://logix.cz-test/prefixed:file-two.txt (4 bytes)

This prefixing mode works with "s3cmd ls" as well:

~$ s3cmd ls s3://logix.cz-test

Bucket 'logix.cz-test':

2007-01-19 02:12 4 s3://logix.cz-test/file-one.txt

2007-01-19 02:12 4 s3://logix.cz-test/file-two.txt

2007-01-19 02:12 4 s3://logix.cz-test/prefixed:file-one.txt

2007-01-19 02:12 4 s3://logix.cz-test/prefixed:file-two.txt

Now with a prefix to list only names beginning with "file-":

~$ s3cmd ls s3://logix.cz-test/file-*

Bucket 'logix.cz-test':

2007-01-19 02:12 4 s3://logix.cz-test/file-one.txt

2007-01-19 02:12 4 s3://logix.cz-test/file-two.txt

For more information refer to:

* S3cmd / S3tools homepage at http://s3tools.sourceforge.net

* Amazon S3 homepage at http://aws.amazon.com/s3

Enjoy!

Michal Ludvig

* [email protected]

* http://www.logix.cz/michal