Skip to content

scripted-diff: Type-safe settings retrieval #31260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

ryanofsky
Copy link
Contributor

@ryanofsky ryanofsky commented Nov 8, 2024

This PR changes the way settings are registered and retrieved to provide more compile-time safety. Currently settings are registered and retrieved like:

// Register setting
argsman.AddArg("-pid=<file>", strprintf("Specify pid file. Relative paths will be prefixed by a net-specific datadir location. (default: %s)", BITCOIN_PID_FILENAME), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);

// Retrieve setting
args.GetPathArg("-pid", BITCOIN_PID_FILENAME)

But this is not ideal because nothing checks that setting names are spelled correctly, or that default values (BITCOIN_PID_FILENAME) are used consistently in the help string and retrieval sites, or that settings are retrieved with consistent types (bool, int, string, path, or list). This PR addresses these issues by adding setting declarations which allow settings to be registered and retrieved like:

// Declare setting
using PidSetting = common::Setting<
    "-pid=<file>", fs::path, {.legacy = true},
    "Specify pid file. Relative paths will be prefixed by a net-specific datadir location. (default: %s)">
    ::DefaultFn<[] { return BITCOIN_PID_FILENAME; }>;

// Register setting
PidSetting::Register(argsman);

// Retrieve setting
PidSetting::Get(args)

Suggestions for review

All the real changes in this PR are in the last scripted-diff commit:

If you take a few minutes to look at the changes applied in this scripted-diff commit, you'll understand everything this PR is doing. A good place to start looking around in this commit is the generated src/init_settings.h file which declares the settings that get registered in src/init.cpp. Then you can look at the surrounding diffs and see they are just replacing AddArg and GetArg calls.

The other notable commit is second commit, which implements the Setting class:

The other commits do minor things like moving code or updating linters. The python script that implements the scripted diff is a temporary artifact that gets deleted. The python script is complicated because it does things like parsing c++ code to extract help strings, and figuring out the right types to declare settings with so code compiles. But the entire scope of the script is to (1) generate Setting type definitions, (2) add #includes, and (3) replace AddArg() calls with Register() calls and GetArg() calls with Get() calls. So there is not much the script can actually do wrong without triggering build and test failures.


Extensions

This PR only adds the ability to declare individual settings with built-in types. It doesn't provide any new runtime behavior, but a branch in issue #22978 extends the Setting class implemented here to support runtime setting validation, additional types like std::variant, additional conversion options, custom types, custom validation, and groups of settings declared as options structs.

This change is mostly orthogonal to #16545. #16545 only provides runtime type checking while this PR only provides compile-time checking with no new runtime behavior. But this change does allow a nicer way of declaring types in #16545, using c++ types like int instead of flags like ALLOW_INT, orstd::vector<std::string> instead of ALLOW_STRING | ALLOW_LIST.

@DrahtBot
Copy link
Contributor

DrahtBot commented Nov 8, 2024

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31260.

Reviews

See the guideline for information on the review process.

Type Reviewers
Concept ACK l0rinc

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #31974 (Drop testnet3 by Sjors)
  • #31961 (Require sqlite when building the wallet by Sjors)
  • #31896 (refactor: Remove redundant and confusing calls to IsArgSet by maflcko)
  • #31887 (CLI cleanups by jonatack)
  • #31845 (Add -pruneduringinit option to temporarily use another prune target during IBD by luke-jr)
  • #31723 (qa debug: Add --debug_runs/-waitfordebugger [DRAFT] by hodlinator)
  • #31664 (Fees: add Fee rate Forecaster Manager by ismaelsadeeq)
  • #31649 (consensus: Remove checkpoints (take 2) by marcofleon)
  • #31492 (Execute Discover() when bind=0.0.0.0 or :: is set by andremralves)
  • #31278 (wallet, rpc: deprecate settxfee and paytxfee by polespinasa)
  • #31250 (wallet: Disable creating and loading legacy wallets by achow101)
  • #30951 (net: option to disallow v1 connection on ipv4 and ipv6 peers by stratospher)
  • #30343 (wallet, logging: Replace WalletLogPrintf() with LogInfo() by ryanofsky)
  • #30342 (kernel, logging: Pass Logger instances to kernel objects by ryanofsky)
  • #30221 (wallet: Ensure best block matches wallet scan state by achow101)
  • #29700 (kernel, refactor: return error status on all fatal errors by ryanofsky)
  • #29641 (scripted-diff: Use LogInfo over LogPrintf [WIP, NOMERGE, DRAFT] by maflcko)
  • #29365 (Extend signetchallenge to set target block spacing by starius)
  • #29278 (Wallet: Add maxfeerate wallet startup option by ismaelsadeeq)
  • #28802 (ArgsManager: support subcommand-specific options by ajtowns)
  • #28792 (Embed default ASMap as binary dump header file by fjahr)
  • #28710 (Remove the legacy wallet and BDB dependency by achow101)
  • #28463 (p2p: Increase inbound capacity for block-relay only connections by mzumsande)
  • #26988 (cli: rework -addrinfo cli to use addresses which aren’t filtered for quality/recency by stratospher)
  • #26022 (Add util::ResultPtr class by ryanofsky)
  • #25722 (refactor: Use util::Result class for wallet loading by ryanofsky)
  • #25665 (refactor: Add util::Result failure values, multiple error and warning messages by ryanofsky)
  • #24539 (Add a "tx output spender" index by sstone)
  • #17783 (common: Disallow calling IsArgSet() on ALLOW_LIST options by ryanofsky)
  • #17581 (refactor: Remove settings merge reverse precedence code by ryanofsky)
  • #17580 (refactor: Add ALLOW_LIST flags and enforce usage in CheckArgFlags by ryanofsky)
  • #17493 (util: Forbid ambiguous multiple assignments in config file by ryanofsky)
  • #16545 (refactor: Implement missing error checking for ArgsManager flags by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@ryanofsky ryanofsky marked this pull request as draft November 8, 2024 22:34
@ryanofsky
Copy link
Contributor Author

Current status of this PR is that bitcoind and test_bitcoin binaries work and functional and unit tests pass, but there are compile errors in the other binaries that need to be fixed, and this also needs to be rebased. The PR is complete with all functionality described above implemented, but it probably needs more documentation. I also would like to add more commits replacing last remaining GetArg / GetIntArg / GetBoolArg / GetArgs / IsArgSet / IsArgNegated method uses with Setting::Get and dropping all those methods.

@ryanofsky ryanofsky force-pushed the pr/scripty branch 2 times, most recently from 47e30b4 to d8a4a0a Compare November 13, 2024 00:31
@ryanofsky
Copy link
Contributor Author

ryanofsky commented Nov 13, 2024

Updated 416860f -> 47e30b4 (pr/scripty.1 -> pr/scripty.2, compare) getting the remaining binaries (not just test_bitcoin and bitcoind) to build, simplifying the way optional and default values are used, making many other cleanups and fixes.
Rebased 47e30b4 -> d8a4a0a (pr/scripty.2 -> pr/scripty.3, compare) due to conflicts
Updated d8a4a0a -> ad32a27 (pr/scripty.3 -> pr/scripty.4, compare) with a fix for the feature_logging.py test that got broken by the previous optional/default changes. Also includes cmake changes to fix some CI build errors.
Updated ad32a27 -> 534f971 (pr/scripty.4 -> pr/scripty.5, compare) replacing IsArgSet calls with Setting::Value().isNull() instead of using std::optional and Setting::Get() to avoid changing any behavior since Get() fully parses the setting and can throw exceptions, and to reduce uses of std::optional which complicated code. Also add basic unit tests and make various script improvements.
Rebased 534f971 -> 7e57237 (pr/scripty.5 -> pr/scripty.6, compare) with a number of changes intended to fix CI errors, and with a fix for silent merge conflict with #31174
Rebased 7e57237 -> 1f24346 (pr/scripty.6 -> pr/scripty.7, compare) due to conflict #31287 and many workarounds for various compiler issues in CI and fixes for more linters

@DrahtBot
Copy link
Contributor

🚧 At least one of the CI tasks failed.
Debug: https://github.com/bitcoin/bitcoin/runs/32896439725

Hints

Try to run the tests locally, according to the documentation. However, a CI failure may still
happen due to a number of reasons, for example:

  • Possibly due to a silent merge conflict (the changes in this pull request being
    incompatible with the current code in the target branch). If so, make sure to rebase on the latest
    commit of the target branch.

  • A sanitizer issue, which can only be found by compiling with the sanitizer and running the
    affected test.

  • An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

Copy link
Contributor Author

@ryanofsky ryanofsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated 215f555 -> 31fe4c6 (pr/scripty.15 -> pr/scripty.16, compare) changing HelpFn arguments as suggested to make them more extensible and support querying default values

using ConsolidateFeeRateSetting = common::Setting<
"-consolidatefeerate=<amt>", std::string, common::SettingOptions{.legacy = true},
"The maximum feerate (in %s/kvB) at which transaction building may use more inputs than strictly necessary so that the wallet's UTXO pool can be reduced (default: %s).">
::HelpFn<[](const auto& fmt) { return strprintf(fmt, CURRENCY_UNIT, FormatMoney(DEFAULT_CONSOLIDATE_FEERATE)); }>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #31260 (comment)

This means that the HelpFn would (ideally) have a way to query the default value as well.

This is now implemented in the latest push.

@DrahtBot
Copy link
Contributor

🚧 At least one of the CI tasks failed.
Debug: https://github.com/bitcoin/bitcoin/runs/37504402027

Hints

Try to run the tests locally, according to the documentation. However, a CI failure may still
happen due to a number of reasons, for example:

  • Possibly due to a silent merge conflict (the changes in this pull request being
    incompatible with the current code in the target branch). If so, make sure to rebase on the latest
    commit of the target branch.

  • A sanitizer issue, which can only be found by compiling with the sanitizer and running the
    affected test.

  • An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

//! which can be formatted at runtime using HelpFn/HelpArgs or
//! DefaultFn/DefaultArgs features (described below).
//!
//! Setting template class definiting-time Setting types which are
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: is definiting-time a typo? maybe definition-time?

Avoid string concatenation and add missing namespace names so AddArg calls can
be moved to header files in an upcoming scripted-diff.
Move constant declarations referenced in AddArg calls to headers so AddArg
calls be moved to header files in an upcoming scripted-diff.
Drop hidden_args vector so all AddHiddenArgs calls can be replaced in upcoming
scripted diff.
Linter changes to deal with _settings.h headers added in the next commit.

In circular-dependencies.py, prevent _settings.h headers from being included
transitively. Add a new check that makes it an error to include _settings.h
from another header file, rather than a .cpp file. Use this fact to avoid
errors about .cpp files including _settings.h files being circularly dependent
based on each other.

In lint-format-strings.py, skip _settings.h because they contain lines like
strformat(fmt, ...) where fmt is a compile time constant that is checked at
compile time by the compiler and can't be checked by this linter.

In check-doc.py update test to look for Setting::Get and Setting::Register
instead of GetArg and AddArg.
…/ Get calls

This commit is a pure refactoring and does not change behavior in any way.

-BEGIN VERIFY SCRIPT-
python contrib/devtools/reg-settings.py
git add -N src/bench/bench_bitcoin_settings.h src/bitcoin-tx_settings.h src/bitcoin-util_settings.h src/bitcoin-wallet_settings.h src/chainparamsbase_settings.h src/common/args_settings.h src/init/common_settings.h src/init_settings.h src/qt/bitcoin_settings.h src/test/argsman_tests_settings.h src/test/logging_tests_settings.h src/wallet/init_settings.h src/dummywallet_settings.h src/qt/test/optiontests_settings.h src/test/getarg_tests_settings.h
git rm contrib/devtools/reg-settings.py
-END VERIFY SCRIPT-
@ryanofsky
Copy link
Contributor Author

Rebased 31fe4c6 -> b396835 (pr/scripty.16 -> pr/scripty.17, compare) due to conflict with #31916. Also debugged the linker errors that happened in https://cirrus-ci.com/task/4624196052975616?logs=ci#L3843, implemented a workaround and reported a bug upstream llvm/llvm-project#129778

@DrahtBot
Copy link
Contributor

🐙 This pull request conflicts with the target branch and needs rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants