What’s in Store for pg_probackup 3

A Quick Trip Down Memory Lane

Pg_probackup first appeared in 2016 and has been evolving ever since. Version 2.5.15 is available on GitHub, while version 2.8.6 is bundled with our Postgres Pro product line.

Pg_probackup supports 14 platforms, offers three types of incremental backups, allows merging them, enables catchup for quick replica restoration, works with S3 and CFS, and, of course, integrates fully with Postgres Professional products. There’s even a dedicated Telegram chat for pg_probackup.

Versions of pg_probackup:

• 2.5.15 Community: Available on GitHub. Only critical bug fixes are applied. Open-source.

• 2.8.6 STD: A heavily reworked, optimized, and more performant version designed primarily for backing up large datasets. Comes with Postgres Pro Standard. Closed-source.

• 2.8.6 ENT: Same as STD but adds CFS support and S3 compatibility. Closed-source.

• 3 and above: A complete overhau l— faster, better, and more efficient. Closed-source for the next couple of years.

The New Architecture of pg_probackup 3

The backlog for pg_probackup 3 was bursting at the seams:

• Tons of client requests we wanted to implement.

• Even more of our own ideas.

• The 2016 architecture had reached a critical mass of "duct-tape fixes", making maintenance a nightmare.

• Without a radical change, implementing the first two points was impossible.

What clients weren’t happy with:

• Managing multiple Postgres versions meant maintaining separate pg_probackup builds for each version.

• The backup format inherited from PG’s data structure was a simple file backup — a directory tree with a massive number of files. This was inconvenient (and costly) when working with S3 and tape storage. Even tar archives didn’t help much, as they still required sequential file placement.

• SSH required separate authorization permissions.

• Two separate connections for copying the database and WAL files.

After careful consideration, we decided to start from scratch — and we haven’t looked back since. The key difference is that pg_probackup is no longer a monolith but consists of three independent components:

1. A DB core extension responsible for reading files and tracking changes for incremental backups.

2. LibProbackup3, the heart of pg_probackup, the library that connects the backup application to the database. This library can be used to build custom solutions or integrate with broader infrastructure. Think of it as a full-fledged SDK.

3. The backup application itself, which handles communication with a DB and manages writing backup data to disk.

Originally, the plan was:

1. Develop the core and basic functions based on the new architecture.

2. Reimplement all features from version 2.

3. Add features that were impossible in version 2.

4. Profit!!!

But during planning, we realized this wasn’t the best approach. While working on the architecture, we came up with many tweaks, some of which seemed questionable. So, we took a different route.

The client's main wish: not to lose data! It means not only successfully creating backups but also quickly restoring them in case of failure. So, we changed the plan:

1. First, implement the key elements ensuring backup creation and fast recovery. This includes the new architecture, core, and replication protocol (more on that later).

2. Then, analyze the current functionality and redesign the entire user experience, aiming to get as close as possible to the previous feature set without aiming to 100% replication.

Here’s what we ended up with:

• You can create backups and restore from them.

• Compatibility with backups made in earlier versions.

• No more separate SSH connections.

• No need to configure separate access to PDGATA — everything runs under the postgres role.

• Most features from pg_probackup 2 are implemented.

• Support for multiple data sources for incremental backups.

• A new backup storage format: "one backup, one backup file" (with a metadata file alongside).

• Improved support for tape and S3. Backups are created as either a single file or split into files of a specified size.

• A new replication protocol using an enhanced walsender, enabling multi-threaded data reading and writing.

• Support for various retention policies.

• Support for vanilla Postgres.

A Closer Look at Libprobackup3

As mentioned, Libprobackup3 handles communication with the Postgres Pro core and allows integration with third-party backup systems. Written in C++, it exposes a C interface, making it easy to integrate with Python scripts, Go, or C++ applications.

Interacting directly with the DB core extension, Libprobackup3 manages archival file streams, metadata, backup logic, and incremental merges. However, it doesn’t know where backups are stored or handle database file I/O — that’s left to the backup application.

Here’s a simple Go integration example:

~$ golang_sample backup -pgport $SOURCEPORT -pgdata $BASE -backup-id $FULL -backup-mode FULL -backup-source=pro -storage=fs

/*Import the necessary package that contains functions for working with libpgpropbackup3*/
package pgpro

/*Parse the command line for parameters required for backup*/
backupCmd Parse(os.Args[2:])

copt := pgpro.ConnectionOptions{PGDatabase: *pgdatabase_flag, PGHost: *pghost_flag, PGPort: *pgport_flag}

/*Initialize variables*/
bopt := pgpro.BackupOptions{
    NumThreads: *num_threads_flag,
    PGData: *pgdata_flag,
    BackupMode: backup_mode_flag,
    BackupSource: source_flag,
    Storage: storage_flag,
    BackupId: *backup_id_flag,
    ParentBackupId: *parent_backup_id_flag,
    do_backup(copt, bopt)

/*Call the library function do_backup*/
func do_backup(copt pgpro.ConnectionOptions, bopt pgpro.BackupOptions, root string, tblspc string) {
    enc := &DummyEncoder{}
    enc.Root = root
    enc.DataFiles = make(map[int]*os.File)
    enc.NonDataFiles = make(map[int]*os.File)
    enc.TablespaceMapping = build_tablespaces(tblspc)
    enc.DataBlockSize = 0
    enc.WalDataBlockSize = 0

    res := pgpro.Backup(copt, bopt, enc)
    if res != nil {
        fmt.Println("Backup error: ", res)
    }
}

The New Replication Protocol in pg_probackup 3

There are three fundamental approaches to backing up a Postgres database:

1. Dump everything via SQL script.

2. Copy files from PGDATA.

3. Physical or logical replication (copying WAL files).

We developed a custom communication protocol for pg_probackup to avoid relying on physical files and directories. This gives us more flexibility in optimizing data reading, writing, and transfer over the network.

Every tool has its own way of doing things. For instance, pg_dump transfers data using SQL, while pg_basebackup relies on replication commands baked into the walsender to kick off a separate backend process that handles the actual file operations.

Before diving into development, we laid out a clear set of goals we wanted to achieve:

• One version to rule them all: A single utility version that works with Postgres versions.

• No direct DB access: Full and incremental backups should be created without the application needing direct access to the database’s internal data.

• Multi-threaded I/O: Reading and writing data in multiple threads for better performance.

• Transparent handling of CFS and other quirks: The application should seamlessly work with CFS (Compressed File System) and other installation-specific features.

• Data and WAL encryption: Ensuring data security through encryption.

• Simple incremental backups: A straightforward mechanism for handling incremental backups.

In pg_probackup 2, the backup application connects to the server using the libpq library in both regular and replication modes. It starts by calling pg_start_backup to fetch metadata at the start of the backup. Then, it transfers the data files, followed by the WAL files, and finally captures the metadata at the end of the backup. All this is written as a collection of separate files.

When the database grows large, and the directory structure inside PGDATA becomes overly complex, copying everything becomes a headache. Sure, you can still copy the files, but if you’re writing backups to S3, this approach becomes expensive. Even if you’re backing up to a local or network drive, writing data and WAL files separately becomes time-consuming. And if you’re copying sequentially, there’s a risk that by the time you’ve copied the data, the corresponding WAL files might have already been deleted. What’s missing here are snapshots.

The new protocol is inspired by pg_basebackup. We added our own replication command, resulting in something like this:

PG_PROBACKUP LABEL label [( option ['value'] [, ...] )]
LABEL 'label'
PROGRESS [ boolean ]
CHECKPOINT { 'fast' | 'spread' }
WAL [ boolean ]
COMPRESS_ALG 'method'
COMPRESS_LVL number
START_LSN [ number ]  -- for incrementals
INCR_MODE [MODE]
VERIFY_CHECKSUMS [ boolean ]

The key differences are START_LSN and INCR_MODE. If START_LSN is zero, it’s a full backup. If non-zero, it’s an incremental starting from that LSN. INCR_MODE lets you specify the incremental mode (e.g., DELTA, PTRACK).

What sets this apart from the old version is the introduction of a separate backend on the server and the ability to transfer both data and WAL files within a single connection. This move nipped the time drift issue in the bud. And just like that, our libprobackup library steps into the picture, encoding data on the fly on the server side, while the backup application takes care of packing it into the backup file and writing it to the chosen storage.

In the console, it looked something like this:

BACKUP COMMAND PG_PROBACKUP(LABEL '0-full', WAL true, VERIFY_CHECKSUMS true, COMPRESS_ALG 'none', COMPRESS_LVL 1);

PG_PROBACKUP 33554472 tli=1
PG_PROBACKUP-STOP 33554744 tli=1 bytes written=61210881 bytes compressed=61210881
[2024-07-08 12:04:55.781912] [93397] [8620067840] [info] Backup time 710
INFO: Backup 0-full completed successfully.
BACKUP INSTANCE 'test'

Instance Version ID  Start time               Mode  WAI  Mode  TLI  Duration  Data  WAI  Zaig  Zratio  Start LSN  Stop LSN  Status
-------------------------------------------------------------------------------------------------------------------------------------
test        17       0-full 2024-07-08 12:04:55  FULL  STREAM  1    58MB  -      none  1.00  0/2000028  0/2000138  DONE

Here, the replication command for pg_probackup is clearly reflected. 0_full is the name of the backup without compression but with checksum verification. And at the end, there’s the success message.  

At first glance, it seems like everything’s okay, but not so fast. This was a custom solution for Postgres Pro, and ditching support for vanilla Postgres was never on the table. Plus, we still hadn’t cracked the old issue: any addition of new commands or changes to the logic of existing ones required patching the core, and rebuilding your own DBMS kernel is no walk in the park. Configuring multi-threading within a single command also required atomic efforts. So, we decided to push further.  

First, we split the single command into three parts. Then, we developed a universal plugin so that instead of calling PG_PROBACKUP LABEL [(option [‘value’] [, …]) ]), you could use the scheme: Plugin > Plugin name > Command PGPRO_CALL_PLUGIN <plugin name> <command> [(option [‘value’] [, …]) ]).  

The overall architecture became even more intriguing. The unified data processing flow was divided into three stages:  

• Start_backup: To fetch metadata at the beginning.  

• Copy_files: For actually moving data into the backup. Multiple instances of this process can run simultaneously, each with different parameters. This enabled the much-needed controlled multi-threading.  

• Stop_backup: To capture metadata at the end of the process and finalize the write operation.  

And the pg_probackup 3 backend started being launched by that very plugin.

Accordingly, one command was split into three. Before:

BACKUP COMMAND PG_PROBACKUP (LABEL '0-full', WAL true, VERIFY_CHECKSUMS true, COMPRESS_ALG 'none', COMPRESS_LVL 1);

After:

[info]START BACKUP COMMAND = PGPRO_CALL_PLUGIN pgpro_bindump start backup(LABEL '0-full');
[info]PG_PROBACKUP 0/200028 tli=1
[info]BACKUP COMMAND PGPRO_CALL_PLUGIN pgpro_bindump copy_files (VERIFY_CHECKSUMS true, COMPRESS_ALG 'none', COMPRESS_LVL 1);
[info]BACKUP COMMAND PGPRO_CALL_PLUGIN pgpro_bindump stop_backup (COMPRESS_ALG 'none', COMPRESS_LVL 1);
BACKUP INSTANCE 'test'

Instance Version ID Start time Mode NAL Mode TLI Duration Data Wal Zratio Status
=================================================================
test 17 0-full 2025-02-08 13:14:09+0000 1 full stream 58MB none 1 0/2000028 0/0 DONE

It looks like everything is working, and we’re ready to release. But not so fast! Simply transferring data is great, but we also need to handle catchup_data and track data changes to create increments. So, we decided to add a few more commands.

Here’s the current state of replication protocols: DIRECT was introduced back in version 2 and supports five modes. With pg_probackup 3, we now have a BASE option based on basebackup and a brand-new PRO protocol.

Summary:

• DIRECT works the same way as in pg_probackup 2. It requires permissions for the PGDATA directory and copies it as is.

• PRO doesn’t need access rights to PGDATA, scales based on workload, and is a newcomer to the PostgreSQL backup world — still unlocking its full potential.

Let's go over the backup modes:

• FULL — creates a complete database backup.

• DELTA — incremental mode based on page-by-page file comparison.

• PTRACK — incremental mode that tracks changes and records the corresponding LSN at the moment of the event.

• PAGE — incremental mode where a change map is created during backup using LSN and checksums.

A bit of practice

To perform a backup using the new protocol, start by adding a few lines to the config file.

# ------------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
# ------------------------------------------------------------------------------

# Add settings for extensions here
wal_level=replica
archive_mode=always
walsender_plugin_libraries = 'pgpro_bindump' #launch the plagin
shared_preload_libraries = 'pgpro_bindump'

On the application side, there aren’t any major changes. You simply call pg_probackup with the usual set of options. The only difference from the previous version is the addition of --backup-source=pro.

Pg_probackup3 backup -B $BACKUPDIR3 --instance $INSTANCE -b FULL --backup-source=pro --backup-id=0-full

[info] Backup catalog '/Users/supauser/projects/pgpro/../backup3' successfully initialized
[info] Instance 'test' successfully initialized
[info] This PostgreSQL instance was initialized with data block checksums. Data block corruption will be detected
[info] START BACKUP COMMAND= PGPRO_CALL PLUGIN pgpro_bindump start_backup (LABEL '0-full');
[info] PG PROBACKUP 0/2000028 tli=1
[info] BACKUP COMMAND PGPRO_CALL_PLUGIN pgpro_bindump copy_files (VERIFY_CHECKSUMS true, COMPRESS_ALG 'none', COMPRESS_LVL 1);
[info] BACKUP COMMAND PGPRO_CALL_PLUGIN pgpro_bindump stop_backup (COMPRESS_ALG 'none', COMPRESS_LVL 1);
[info] PG_PROBACKUP-STOP 0/2000138 tli=1 bytes written=48885446 bytes compressed=48885446
[info] Backup time 817
[info] Backup 0-full completed successfully.
INFO: Backup 0-full completed successfully.
[info] Start validate 0-full...
[info] Validating backup 0-full
[info] Validate time 107
[info] Backup 0-full is valid

As we can see, the logs now display the new replication commands.

What’s New with Recovery in pg_probackup 3?

One of the coolest features we added is the ability to restore data directly from a backup without entirely unpacking it. This feature is called FUSE.

Imagine you’ve backed up a large database, and then production goes down because a few files were lost due to a disk failure. With the old approach, you’d replace the disks, wipe everything, and then download and unpack the backup — a process that could take hours, if not days.

With FUSE, the backup file isn’t unpacked to disk but mounted in memory via a special driver. This is similar to how the mount command works with regular disks. Since pg_probackup 3 backups are structured blocks with metadata, we can read specific files directly from the mounted backup and copy them where needed. You can even start the instance directly from the mounted backup.

pg_probackup3 fuse –B "/task_data/backups“ --mnt-path /tmp/mntpoint --instance=node -i SQA7X7
pg_ctl -D /tmp/mntpoint start

But don’t expect the same performance as if everything were already unpacked and stored on local disks. This is more of an emergency startup option, allowing you to take a calmer approach afterward, since the most critical business processes, though slowly, are up and running.

Database recovery options are:

• Restoration happens in a separate instance.

• Only the data for the required database is copied.

• The same goes for WAL files.

• Service data (e.g., configuration tables, database lists) is also copied.

• A recovery mode is started, where Postgres automatically fixes discrepancies in service files.

pg_probackup3 restore -B <backups_dir> --instance <instance_name> -D new_pgdata_dir> -i <backup_id> --db-include=db1

For remote recovery, where you don’t need DB server access, you can run pg_probackup remote-restore on the remote machine and netcat in listener mode on the target machine to start the recovery over the network.

Performance Testing

We conducted a full cycle of comparative testing between version 2.8.6 and the latest stable build of pg_probackup 3.0.0. Tests were run on a randomly generated 1 TB database with ten tables. We compared performance in creating full and incremental backups in PRO and DIRECT modes, with multi-threading ranging from 1 to 96 threads.

Here is the commands:

#pg_probackup2.8
pg_probackup backup -B /probackup_repo --instance=demo -b full -j 96 --no-validate --compress-algorithm=lz4

pg_probackup backup -B /probackup_repo --instance=demo -b ptrack -j 96 --no-validate --compress-algorithm=lz4

#pg_probackup3.0.0
pg_probackup3 backup -B /probackup_repo --instance=demo -b FULL --backup-source=pro -j 96 --compress-algorithm=lz4 --no-validate --log-level-console=info

pg_probackup3 backup -B /probackup_repo --instance=demo -b PTRACK --backup-source=pro -j 96 --compress-algorithm=lz4 --no-validate --log-level-console=info --parent-backup-id 2025-01-22-16-18-05-247

pg_probackup3 backup -B /probackup_repo --instance=demo -b PTRACK --backup-source=direct -j 96 --compress-algorithm=lz4 --no-validate --log-level-console=info --parent-backup-id 2025-01-22-16-18-05-247

As a result, we obtained graphs and findings as follows:

Results:

• For full backups, version 2.8.6 was faster in single-threaded mode, but from 8 threads onward, version 3.0.0 outperformed it.

• For PTRACK incrementals in PRO mode, version 3.0.0 also showed better performance, except for an anomaly at high thread counts where version 2.8.6 struggled.

What's next?

The roadmap for pg_probackup 3 is straightforward:

1. Release pg_probackup 3.

2. Prepare an open-source version.

3. Optimize multi-threading.

4. Implement everything planned for pg_probackup 3.1.

If you have suggestions, don’t hesitate to share them! We’re all ears.