Safe64 is a binary data encoding scheme that is safe to be passed through processing systems expecting human readable text.

• Safe for use in JSON, SGML formats, source code string literals, URLs, without escaping
• Safe for use in filenames
• Safe for use in formatted documents
• Safe for use in legacy text processing systems
• Alternate Form with Support for length fields

• All of the abovementioned features
• Smaller alphabet
• No padding characters
• No escaping necessary
• Liberal whitespace rules
• Safe for use in filenames
• Reliable truncation detection
• Sortable in generic text sorting algorithms (such as file listings)

The following bolded, capitalized terms have specific meanings in this document:

• Safe64 Encoding
  • Features:
  • Advantages over base64:
  • Terms and Conventions
  • Contents
  • Encoding
    • Encoding Process
      • 3 Byte Full Group
      • 2 Byte Partial Group
      • 1 Byte Partial Group
    • Final Group
    • Alphabet
  • Whitespace
  • Examples
  • Filenames
• Safe64L Encoding
  • Encoding
  • Whitespace
  • Truncation Detection
  • Examples
    • Example: Length field & data:
  • Filenames
  • Advantages over base64 padding
  • Version History
  • License

Safe64 encoding uses an alphabet of 64 characters from the single-byte printable UTF-8 set to represent radix-64 chunks (where each chunk has an individual value from 0 - 63). These chunks are grouped magnitudally into big-endian sequences of 4 chunks, giving a range of 64^4 = 16777216 (0x1000000), which allows 24 bits (3 bytes) of data storage per group. Such an ecoding scheme multiplies the size of the data by a factor of 1.33.

Layout:

Bytes:  [aaaaaaaa] [bbbbbbbb] [cccccccc]
Chunks: [aaaaaa] [aabbbb] [bbbbcc] [cccccc]

The encoding process encodes groups of 3 bytes, outputting 4 chunks per group. If the source data length is not a multiple of 3, then the final group is output as a partial group, using only as many chunks as is necessary to encode the remaining bytes, with the unused high portion of the highest chunk cleared.

First, an accumulator is filled big-endian style with 3 bytes of data:

accumulator = (bytes[0] << 16) | (bytes[1] << 8) | bytes[2]

Next, the accumulator is broken down big-endian style into radix-64 chunks:

chunk[0] = (accumulator >> 18) & 0x3f
chunk[1] = (accumulator >> 12) & 0x3f
chunk[2] = (accumulator >> 6) & 0x3f
chunk[3] = accumulator & 0x3f

Bytes:  [aaaaaaaa] [bbbbbbbb]
Chunks: [00aaaa] [aaabbb] [bbbbbb]

accumulator = (bytes[0] << 8) | bytes[1]
chunk[0] = (accumulator >> 12) & 0x3f
chunk[1] = (accumulator >> 6) & 0x3f
chunk[2] = accumulator & 0x3f

Bytes:  [aaaaaaaa]
Chunks: [0000aa] [aaaaaa]

accumulator = bytes[0]
chunk[0] = (accumulator >> 6) & 0x3f
chunk[1] = accumulator & 0x3f

In the last (possibly partial) group, the number of remaining characters indicates how many bytes of data remain to be decoded, and whether truncation has been detected. Note that truncation detection is not guaranteed - for that you would need to use Safe64L encoding.

Chunk values are output as characters according to the following alphabet:

The alphabet is ordered according to the characters' ordinal positions in UTF-8, so that the resulting encoded text will sort in the same natural ordering as the data it represents.

An encoded stream CAN contain whitespace at any point. A decoder MUST accept and discard all whitespace characters while processing the stream.

For the purposes of this spec, only the following characters qualify as whitespace:

Data:    {0x39, 0x12, 0x82, 0xe1, 0x81, 0x39, 0xd9, 0x8b, 0x39, 0x4c, 0x63, 0x9d, 0x04, 0x8c}
Encoded: DG91sN3tqNgtI5DS-HB

Data:    {0xe6, 0x12, 0xa6, 0x9f, 0xf8, 0x38, 0x6d, 0x7b, 0x01, 0x99, 0x3e, 0x6c, 0x53, 0x7b, 0x60})
Encoded: tW9abzVsQMg0aItgJrhV

Data:    {0x21, 0xd1, 0x7d, 0x3f, 0x21, 0xc1, 0x88, 0x99, 0x71, 0x45, 0x96, 0xad, 0xcc, 0x96, 0x79, 0xd8})
Encoded: 7S4xEm60X8_lGOPhn8Ot2N

Files containing safe64 data SHOULD have the extension s64, for example mydata.s64.

While safe64 is sufficient for most systems, there are transmission mediums where no clear end marker exists for the encoded data field, or where no guarantee exists for detecting truncated data. In such cases, it is sometimes desirable to prefix a length field so that the receiving end will be sure of a complete transfer.

Safe64L works essentially the same as safe64, except that it is prefixed by a length field. The length field is built incrementally with 6-bit chunks encoded using the same encoding alphabet as is used for regular safe64 data, until sufficient bits are available to encode the length of the data.

In each chunk, the lower 5 bits contain data, and the high bit is a continuation bit:

• c = continuation bit
• x = data

While the continuation bit is set to 1, the length field is continued in the next character. Building of the length field continues until a continuation bit of 0 is encountered. The 5-bit data chunks are interpreted in big endian order (the first character represents the highest 5 bits, then the next lower 5 bits, and so on).

Note: The length field encodes the length of the non-encoded source data - not the encoded result, and not including the length field itself.

The length field CAN contain whitespace at any point in the stream, following the same rules as for safe64.

If truncation occurs anywhere in the encoded sequence (length or data), one of two things will happen:

1. The decoded data length won't match the length field.
2. The length field won't have a character with the continuation bit cleared.

Data:

0x21, 0x7b, 0x01, 0x99, 0x3e, 0xd1, 0x7d, 0x3f,
0x21, 0x8b, 0x39, 0x4c, 0x63, 0xc1, 0x88, 0x21,
0xc1, 0x88, 0x99, 0x71, 0xa6, 0x9f, 0xf8, 0x45,
0x96, 0xe1, 0x81, 0x39, 0xad, 0xcc, 0x96, 0x79,
0xd8

Encoded:

W07Mg0aIvGUIwWXn_BNw577R57aM5abzW4_i50DPrB_bbN

In this case, the length field is W0 (33)

Files containing safe64l data SHOULD have the extension s64l, for example mydata.s64l.

• No extra alphabet character is necessary. The length encodes using the exact same alphabet as the data encoding.
• Truncation is always detected. With base64 padding, truncation on a 4-character boundary will not be detected.
• Lower data usage for smaller data. base64's padding scheme uses an average of 1.5 characters no matter the length of the data. Safe64L uses only 1 byte for lengths 31 and under. By the time its size performance suffers in comparison to base64 (at length 32), the character length difference is already less than 2% of the total payload size, and shrinks from there.

• January 29, 2019: Version 1
• January 1, 2019: Preview Version 1

Copyright (c) 2019 Karl Stenerud. All rights reserved.

Distributed under the Creative Commons Attribution License: https://creativecommons.org/licenses/by/4.0/legalcode License deed: https://creativecommons.org/licenses/by/4.0/