Skip to content

RFC: support for writing UTXO set dump (dumptxoutset RPC) to a named pipe #31373

Open
@theStack

Description

Please describe the feature you'd like to see added.

While the primary obvious use-case for the dumptxoutset RPC is to create AssumeUTXO snapshots (to be distributed and loaded on newly created nodes via the loadtxoutset RPC later), it can also be useful as input for external tooling like converters to other UTXO set formats, e.g. #27432. For those, the intermediate step of writing a >10GB file to disk and then reading it again is wasteful and annoying, as it consumes both more time and space than necessary. By supporting writing to a named pipe, the output data could be fed directly into another process instead. Thanks to the UNIX "everything is a file" philosophy, no logic changes in the tooling are even needed -- the reader only sees an input stream and doesn't notice or care if the input file represents an actual physical file on disk or if the data is generated on-the-fly from another process.

Currently needed steps for external tools:

  1. call dumptxoutset to create utxo.dump (>10GB on mainnet)
  2. call external tool with utxo.dump as input (run only after step 1 is finished)
  3. delete utxo.dump

Needed steps for external tools with named pipe support:

  1. create a named pipe utxo.pipe (e.g. via https://linux.die.net/man/3/mkfifo)
  2. call dumptxoutset to write to utxo.pipe
  3. call external tool with utxo.pipe as input (run in parallel to step 2)
  4. delete utxo.pipe

I've tried this yesterday and it works as expected with minimal changes (see proposed solution below). Will push the branch later with concrete instructions, if people feel that this is worthwhile to support.

Is your feature related to a problem, if so please describe it.

No response

Describe the solution you'd like

The dumptxoutset call only needs two minor behaviour modificiations. If the passed path is a named pipe (trivially detectable via C++ standard library routine std::filesystem::is_fifo), then:

  • don't error if the file already exists
  • don't create a temporary file with .incomplete suffix, but write directly into the specified path

Describe any alternatives you've considered

No response

Please leave any additional context

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions