Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability for node to reject tx from mempool by number of tx inputs #2342

Merged
merged 8 commits into from
Jun 21, 2017

Conversation

bitcartel
Copy link
Contributor

@bitcartel bitcartel commented May 4, 2017

Implement short-term solution described in #2343 so that users can respond promptly to critical short-term problems caused by quadratic validation scaling, such as the getblocktemplate latency, block propagation latency, and mempool size inflation issues described in #2333.

Copy link
Contributor

@daira daira left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor changes needed. Also needs a test.

src/init.cpp Outdated
@@ -332,6 +332,7 @@ std::string HelpMessage(HelpMessageMode mode)
strUsage += HelpMessageOpt("-dbcache=<n>", strprintf(_("Set database cache size in megabytes (%d to %d, default: %d)"), nMinDbCache, nMaxDbCache, nDefaultDbCache));
strUsage += HelpMessageOpt("-loadblock=<file>", _("Imports blocks from external blk000??.dat file") + " " + _("on startup"));
strUsage += HelpMessageOpt("-maxorphantx=<n>", strprintf(_("Keep at most <n> unconnectable transactions in memory (default: %u)"), DEFAULT_MAX_ORPHAN_TRANSACTIONS));
strUsage += HelpMessageOpt("-mempooltxvinlimit=<n>", _("If set, limit the number of inputs to a transaction which this node will accept into the mempool for relay and mining"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest "mempooltxinputlimit"; vin is internal jargon.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it's the number of transparent inputs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

src/init.cpp Outdated
if (mapArgs.count("-mempooltxvinlimit")) {
unsigned int limit = GetArg("-mempooltxvinlimit", UINT_MAX);
if (limit == 0) {
return InitError(_("Mempool transaction vin limit cannot be 0."));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also change this if vin is dropped from the option name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

src/init.cpp Outdated
@@ -1009,6 +1010,14 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
}
#endif

if (mapArgs.count("-mempooltxvinlimit")) {
unsigned int limit = GetArg("-mempooltxvinlimit", UINT_MAX);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the default should be 40, but let's discuss that on #2343.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Default is now 0 which means no limit (code has changed in forced push)

src/init.cpp Outdated
@@ -1009,6 +1010,14 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
}
#endif

if (mapArgs.count("-mempooltxvinlimit")) {
unsigned int limit = GetArg("-mempooltxvinlimit", UINT_MAX);
if (limit == 0) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

int64_t limit = GetArg("-mempooltxvinlimit", std::numeric_limits<int64_t>::max());
if (limit < 0) {
    return InitError(_("Mempool limit on transparent inputs to a transaction must be greater than 0."));
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have adopted language.

src/main.cpp Outdated
// Node operator can choose to reject tx by number of inputs
if (mapArgs.count("-mempooltxvinlimit")) {
unsigned int n = tx.vin.size();
unsigned int limit = GetArg("-mempooltxvinlimit", UINT_MAX);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

static_assert(std::numeric_limits<size_t>::max() >= std::numeric_limits<int64_t>::max(), "size_t too small");
size_t n = tx.vin.size();
size_t limit = (size_t) GetArg("-mempooltxvinlimit", std::numeric_limits<int64_t>::max());

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@zookozcash
Copy link

Red flag: Nathan mistakenly thought that the Incident Response team had required this ticket to be prioritised for 1.0.9. In fact the opposite was true, the Incident Response team (me as Executive Coordinator) had decided that we would take steps to fix the problem in the short term explicitly under the assumption that this would not imply that this ticket would be prioritised for 1.0.9.

@bitcartel bitcartel force-pushed the 1081__mempoolpatch branch from 607ef82 to 9c2a621 Compare May 5, 2017 04:25
@bitcartel
Copy link
Contributor Author

@daira Changes made. No test yet, is someone available to write that?

src/init.cpp Outdated
@@ -332,6 +332,7 @@ std::string HelpMessage(HelpMessageMode mode)
strUsage += HelpMessageOpt("-dbcache=<n>", strprintf(_("Set database cache size in megabytes (%d to %d, default: %d)"), nMinDbCache, nMaxDbCache, nDefaultDbCache));
strUsage += HelpMessageOpt("-loadblock=<file>", _("Imports blocks from external blk000??.dat file") + " " + _("on startup"));
strUsage += HelpMessageOpt("-maxorphantx=<n>", strprintf(_("Keep at most <n> unconnectable transactions in memory (default: %u)"), DEFAULT_MAX_ORPHAN_TRANSACTIONS));
strUsage += HelpMessageOpt("-mempooltxinputlimit=<n>", _("Set the maximum number of transparent inputs in a transaction which the mempool will accept (default: 0 = no limit applied)"));
Copy link
Contributor

@daira daira May 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: s/which/that/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Contributor

@daira daira left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than the double-negative in the error message, ACK on the changes since my last review. Still needs a test.

src/init.cpp Outdated
if (mapArgs.count("-mempooltxinputlimit")) {
int64_t limit = GetArg("-mempooltxinputlimit", 0);
if (limit < 0) {
return InitError(_("Mempool limit on transparent inputs to a transaction cannot not be negative"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/cannot not/cannot/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@daira daira requested a review from str4d May 5, 2017 06:14
@veikkoeeva
Copy link

veikkoeeva commented May 5, 2017

A kind proposal, if there would a comment in the code about the problems and the solution, it would be great. I have been following the discussion in chat and I think I'd agree that "limiting this is good", but necessarily the real reasonin and why some particular limit was chosen (even commenting in vein "this limit is an educated guess to quell most of the problems known at the time") and what would happen to the transactions if the check fails (transactions in limbo?).

@daira
Copy link
Contributor

daira commented May 5, 2017

@veikkoeeva I'll answer that on #2343.

@nathan-at-least
Copy link
Contributor

Concept NACK: #2343 (comment)

@nathan-at-least nathan-at-least self-requested a review May 10, 2017 21:41
@nathan-at-least nathan-at-least added this to the 1.0.10 Release milestone Jun 2, 2017
@nathan-at-least
Copy link
Contributor

Considering for 1.0.10 release juxtaposed to related tickets.

Copy link
Contributor

@str4d str4d left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK on the patch, but I agree with daira's string requests and that this needs a test.

@str4d
Copy link
Contributor

str4d commented Jun 14, 2017

I'm also +1 on merging this in 1.0.10. I agree with @daira that we need something that users can leverage now, while we work on the longer-term improvements. I also think that the release notes should be explicit about how this is a short-term fix, and that it will be replaced (not extended) in future (likely with some metric that takes into account the entire mempool, not just individual transactions).

@bitcartel bitcartel force-pushed the 1081__mempoolpatch branch from 9c2a621 to da6d939 Compare June 17, 2017 02:32
CValidationState state3;
CTransaction tx3(mtx);
EXPECT_FALSE(AcceptToMemoryPool(pool, state3, tx3, false, &missingInputs));
EXPECT_NE(state3.GetRejectReason(), "bad-txns-version-too-low");
Copy link
Contributor Author

@bitcartel bitcartel Jun 17, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are all the error messages in the test related to 'bad-txns-version-too-low' ? I only see it fired here:

bool CheckTransactionWithoutProofVerification(const CTransaction& tx, CValidationState &state
)
{
    // Basic checks that don't depend on any context

    // Check transaction version
    if (tx.nVersion < MIN_TX_VERSION) {
        return state.DoS(100, error("CheckTransaction(): version too low"),
                         REJECT_INVALID, "bad-txns-version-too-low");
    }

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. I intentionally set tx.nVersion = 0 to trigger this error, as it's the first check that occurs after the -mempooltxinputlimit check, and it means that I didn't have to mock out a heap of global state. Think of that error as "the transaction passes the tx input limit check". The actual check itself doesn't return a reason, which is why I check that the reject reason is not bad-txns-version-too-low; we could possibly make things more explicit by checking instead that it is empty.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you copy the above explanation into the test? I take it we don't want to return state.DoS which updates global CValidationState state in init.cpp, rather than what we do right now which is simply return false (like some of the other code paths in AcceptToMemoryPool).

Copy link
Contributor

@str4d str4d left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NACK

if (isfromtaddr_) {
size_t limit = (size_t)GetArg("-mempooltxinputlimit", 0);
if (limit > 0) {
size_t n = t_inputs_.size();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this point in the function, this is the list of all available inputs, not selected inputs; z_sendmany therefore fails incorrectly. This check needs to be shifted to just below where t_inputs_ is re-used for storing the selected inputs. We also need to re-write this code (likely as part of #1360) to prevent this kind of bug.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Copy link
Contributor

@str4d str4d left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK

Also reduce number of z_sendmany calls made so test runs quicker.
@bitcartel
Copy link
Contributor Author

@str4d can you check latest commit; fixed test and reduced time to run.
@daira can you review the tests which were added to the PR since you last commented?

@daira
Copy link
Contributor

daira commented Jun 21, 2017

I'll review now.

Copy link
Contributor

@daira daira left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment about the order of checking tx version and number of inputs needs a response, otherwise utACK.

// Create an obviously-invalid transaction
// We intentionally set tx.nVersion = 0 to reliably trigger an error, as
// it's the first check that occurs after the -mempooltxinputlimit check,
// and it means that we don't have to mock out a lot of global state.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, input limiting occurs before the tx version check? How can we parse the transaction if it has an unknown version? The spec explicitly says you're not supposed to try to parse fields other than nVersion if the version is unrecognised. (This could lead to misreporting a transaction as being invalid for having too many inputs, when in fact that field at that position has changed its meaning entirely and the transaction should instead have been rejected for having an unknown version. Not critical, but it could lead to confusing log messages.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did wonder this myself. However:

  • The version check is solely that tx.nVersion >= MIN_TX_VERSION, where MIN_TX_VERSION = 1, so any positive integer is already accepted. This is in fact one way in which transaction replay prevention has been proposed upstream: make the version negative.
  • Transaction parsing already happens before AcceptToMemoryPool is called, so the spec should still be satisfied:
bool static ProcessMessage(...)
{
    ...
    else if (strCommand == "tx")
    {
        vector<uint256> vWorkQueue;
        vector<uint256> vEraseQueue;
        CTransaction tx;
        vRecv >> tx;

        CInv inv(MSG_TX, tx.GetHash());
        pfrom->AddInventoryKnown(inv);

        LOCK(cs_main);

        bool fMissingInputs = false;
        CValidationState state;

        pfrom->setAskFor.erase(inv.hash);
        mapAlreadyAskedFor.erase(inv);

        if (!AlreadyHave(inv) && AcceptToMemoryPool(mempool, state, tx, true, &fMissingInputs))
        {
            ...
        }
        ...
    }
    ...
}

bool ProcessMessages(...)
{
    ...
        bool fRet = false;
        try
        {
            fRet = ProcessMessage(pfrom, strCommand, vRecv, msg.nTime);
            boost::this_thread::interruption_point();
        }
        catch (const std::ios_base::failure& e)
        {
            ...
        }
    ...
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bystander note here, feel free to ignore, but could this reasoning be added as a comment to the code? I think it would greatly help others too and later when one is perhaps considering refactoring the code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the reminder @veikkoeeva - I've updated the first post in the conversation above, so when this is merged, it will also be in the commit message for the merge. I think anybody in the future who wants to refactor the code around mempooltxinputlimit will have the information on hand to do so.

Copy link
Contributor

@daira daira Jun 21, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see this in the first post.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@daira You don't see the text beginning "Implement short-term... " ? Also the PR number should be part of the zkbot commit message, so a developer can find their way to this conversation.

@daira
Copy link
Contributor

daira commented Jun 21, 2017

Actually I'll file another ticket about the parsing-of-unknown-tx-versions issue. That is not going to affect consensus so it can be addressed separately and does not need to block this PR.

Copy link
Contributor

@str4d str4d left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs one more adjustment, which I will make now.

assert_equal(set(self.nodes[1].getrawmempool()), set())

# Should use two UTXOs and succeed
spend_taddr_id2 = self.nodes[1].sendtoaddress(node0_taddr, spend_taddr_output + spend_taddr_output - Decimal('1'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first call was changed from sendtoaddress to sendfrom; this one should be too.

self.call_z_sendmany(node0_zaddr, node1_taddr, spend_taddr_amount - spend_taddr_output - spend_taddr_output - Decimal('0.0001')) # note amount less fees
recipients.append({"address":self.nodes[1].getnewaddress(), "amount": spend_taddr_output})
recipients.append({"address":self.nodes[1].getnewaddress(), "amount": spend_taddr_output})
recipients.append({"address":self.nodes[1].getnewaddress(), "amount": spend_taddr_amount - spend_taddr_output - spend_taddr_output})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK; the first variant was left over from an earlier version where node 1 already had a balance, so I was trying to get everything into one t-addr, but couldn't make multiple TXOs in the same z_sendmany.

Copy link
Contributor

@str4d str4d left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK

@str4d
Copy link
Contributor

str4d commented Jun 21, 2017

@zkbot r+

@zkbot
Copy link
Contributor

zkbot commented Jun 21, 2017

📌 Commit 6ea58d1 has been approved by str4d

@zkbot
Copy link
Contributor

zkbot commented Jun 21, 2017

⌛ Testing commit 6ea58d1 with merge 59de56e...

zkbot added a commit that referenced this pull request Jun 21, 2017
Add ability for node to reject tx from mempool by number of tx inputs

Implement short-term solution described in #2343 so that users can respond promptly to critical short-term problems caused by quadratic validation scaling, such as the getblocktemplate latency, block propagation latency, and mempool size inflation issues described in #2333.
@zkbot
Copy link
Contributor

zkbot commented Jun 21, 2017

☀️ Test successful - pr-merge
Approved by: str4d
Pushing 59de56e to master...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants