SliceDB is a tool for capturing and restoring a subset of a PostgreSQL database. It also supports scrubbing sensive data.

pip3 install slice-db

docker pull rivethealth/slicedb

For all commands and options, see Usage.

First, query a database to create a schema file.

slicedb schema > schema.yml

Second, dump a slice:

slicedb dump --root public.example 'WHERE id IN (7, 56, 234)' --schema schema.yml > slice.zip

Third, restore that slice into another database:

slicedb restore < slice.zip

For a complete working example, see Example.

Use the libpq environment variables to configure the connection.

PGHOST=myhost slicedb schema > slice.yml

See dump.yml for the JSONSchema.

SliceDB can produce multiple formats:

• slice - ZIP archive. This can be restored with slicedb restore.
• sql - SQL file. This can be restored with psql or another client. If restoring into existing schema, foreign keys must first be disabled, e.g. SET session_replication_role = replica.

Schema can optionally be included. Restoring with schema requires an existing empty database.

The schema command uses foreign keys to infer relationships between tables. It is a suggested starting point.

You may want to prune the slice by removing relationships, or expand the slice by adding relationships that don't have explicit foreign keys.

slicedb schema-filter can help modify the schema, or generic JSON tools like jq.

The slicing process works as follows:

1. Starting with the root table, query the physical IDs (ctid) of rows.

2. Add the row IDs to the existing list.

3. For new IDs, process each of the adjacent tables, using them as the current root.

Do this in parallel, using pg_export_snapshot() to guarantee a consistent snapshot across workers.

Hundreds of thousands of rows can be exported in only a few minutes and several dozen MBs of memory.

See transform.yml for the JSONSchema.

Replacements are deterministic for a given pepper. By default, the pepper is randomly generated each run. You may specify it as --pepper. Note that possession of the pepper makes the data guessable.

Transformation may operate on an existing slice (TODO), or happen during the dump.

Transforms are specified by:

• class, the Python class
• config, transform-specific options
• module, defaults to slice_db.transforms

The name given to the transform is appended to the global pepper.

To create custom transforms, implement slice_db.transform.Transform, expose the class on a module, and install the module so that is accessible by slicedb.

The slice_db.transforms package has many common transforms.

See transforms.md for the full list.

SliceDB can restore slices into existing databases. In practice, this should normally be an empty existing database.

Foreign keys may form a cycle only if at least one foreign key in the cycle is deferrable.

That foreign key will be deferred during restore.

A restore may happen in a single transaction or not. Parallelism requires multiple transactions.

• Multiple databases
• Databases other than PostgreSQL

Install: make install

Format: make format

1. Update slice_db/version.py
2. Commit git add slice_db/version.py && git commit -m "Version <version>"
3. Tag git tag v<version>
4. Push git push origin master --tags
5. Publish to PyPI: make upload