Sereal - A binary serialization format

This repository is the home of the Sereal data serialization format. This format was started because the authors had technical reasons for producing a better Storable.

Before we embarked on this project we had a look at various prior art. This included a review of Google Protocol Buffers and of the MessagePack protocol. Neither suited our needs so we designed this, liberally borrowing ideas from the other projects.

We wanted to be able to serialize shared references properly. Many serialization formats do not support this out of the box.

Given that perl uses a reference counting garbage collection scheme, Perl has the concept of a special type of reference called a "weakref" which is used to create cyclic reference structures which do not leak memory. Unlike most of the existing solutions, we need to handle these structures correctly, thus avoiding a perfectly valid data structure to be converted to one that will cause a memory leak on a remote system. For cross-language compatibility, weak references can very easily be ignored by other decoder implementations.

Perl supports aliases. These are a special kind of reference which is effectively a C level pointer instead of a Perl language-level reference. We needed to be able to represent these as well.

Promoting a plain data structure reference to an object, as is customary in dynamic languages, can be dangerous in some circumstances. We needed to be able to serialize objects safely and reliably, and we wanted a sane control mechanism for doing so.

In Perl, a regular expression is a native type. We wanted to be able serialize these at a native level without losing data such as modifiers.

We want to be able to represent common structures as small as is reasonable. Although not to the extreme that this makes the protocol error prone and ludicrously difficult to implement. The steps taken include removing redundancy from the serialized structure (such as hash keys or classnames) automatically. The protocol supports this kind of redundancy removal, but an encoder implementation can choose to which extent it makes use of the technique.

We want to be able to serialize and deserialize quickly. Some of the design decisions and trade-offs were aimed squarely at performance.

We wanted to separate the functions of serializing from deserializing so they could be upgraded independently.

We wanted the protocol to be robust to forward/backwards compatibility issues. It should be possible to partially read new formats with an old decoder, and output old formats with a new encoder.

We want the format to be usable by other languages, especially dynamic languages. In aim of making this easier we have structured our repo so that implementations from other languages can be easily added, and we would welcome any contributions along these lines.

There are some graphs of how the Perl implementations Sereal performs as compared to the alternatives at Sereal Performance Graphs

You can find the specification at sereal_spec.pod

There is a low-traffic announcement mailing list sereal-announce as well as a more general development list sereal-dev

Yves Orton <[email protected]>

Damian Gryski <[email protected]>

Steffen Mueller <[email protected]>

Rafaël Garcia-Suarez

Ævar Arnfjörð Bjarmason <[email protected]>

This protocol was originally developed for Booking.com. With approval from Booking.com, the code and specification were generalized and published as Open Source on github and CPAN, for which the authors would like to express their gratitude.

Copyright (C) 2012, 2013, 2014 by Steffen Mueller

Copyright (C) 2012, 2013, 2014 by Yves Orton