See #174

Originally we thought that this was similar to #174, but @daira pointed out:

Committing to a recent block isn't the same as committing to a fork, since you can't use a block to specify a fork decision before the fork has occurred

Note that if we want to enable users to select which side of the fork to go with before it happens, then zcashd needs to support this mechanism well in advance, so that we have explicit support for both opt-in and opt-out cases. If we only use this as a counter, then by default nodes that don't upgrade are in the opt-out category, and thus we have two states:

• Transactions made after the hard fork are valid on whichever side they were made on.
• Transactions made before the hard fork are valid on the blockchain pre-fork and on the opt-out side after, never on the opt-in side.
  • As a consequence, transactions made before the hard fork that are incremented (to be opt-in) will not be mined until after the hard fork.

It is the latter state that lacks the symmetry of an explicit opt-in/opt-out mechanism.

Wouldn't implementing this make merging forks impossible (#713)?

This is for planned hard forks. You typically don't want to merge a planned hard fork.

How about this: if there is a planned fork coming, then there are identifiers (not names [*]) that the user agent must use to specify which future branch their transactions would be valid on. If they don't, then the consensus rules reject those transactions after the fork happens.

([*] Because I don't want there to be an advantage to whichever branch squats "Zcash" first, and I don't want there to be conflict over which branch has the better claim to being the True "Zcash". So instead make them non-human-meaningful identifiers, i.e. the secure hash of something that the creators/users of the intended future branches can't easily grind.)

There's something interesting about this… people who pre-specify which future branch their transactions are valid for are performing a kind of proof-of-stake voting on which branch they are committing to.

Hm… Actually I don't immediately see how the consensus rules can know that a planned fork is coming, nor how they can know when it happened, so I don't see at all how to implement the above, and it is starting to get complicated. :-/

I think people will have learned their lesson from Coinbase throwing away ETC, and if we have adequate post-chain-fork disambiguation of which branch a given tx is valid for (even if the user agent has to opt-in to the disambiguation), then that will be sufficient and we don't need this idea of pre-chain-fork disambiguation.

The reason I suggested names for the forks, is that it allows software that is unaware of the specific fork to present configuration options that humans have a chance of remembering. As a compromise, how about using a ZIP number, i.e. enable-zipnn or disable-zipnn. This does centralise the assignment of fork proposal numbers, but I think that is a feature, in that it requires each proposal to have a specification that goes through the ZIP process.

(Perhaps if Ethereum had gone through a similar review process they would have had a chance to avoid three serious technical mistakes -- the denial-of-service vulnerability that caused the soft fork to fail, the failure to enable an existing replay protection mechanism that had been used on their testnet, and another window of vulnerability they got away with because no-one exploited it: ethereum/go-ethereum#2832 )

As to how this can be implemented technically: have each transaction include a list of ZIP numbers that it opts into, and a list of ZIP numbers that it opts out of. (There are some tricky user interface problems and anonymity set partitioning problems here, but this has some other advantages for future proofing, so I'd like to discuss the idea in tomorrow's engineering meeting.)

We could also do a bit-flag field similar to what upstream does at the block level. If we are trying transactions to ZIPs then the software doing the fork has to have support early for mapping the numbers to block heights, so at the same time we can allocate a particular set of bits to certain numbers, and keep a number in the config flag but not need the additional space in the transactions.

Bit reuse is possible because once a fork is decidedly enabled, a checkpoint can be added. It would need to be in a way that didn't re-enable replay attacks, but that is the same for including numbers (otherwise you'd have to include long-past forks in perpetuity). Perhaps that could be done as a pairing of the numbers or bit field, and a commitment to a block height or hash.

The advantage of having the numbers directly in the transaction is that anyone (particularly clients) can opt-in our opt-out of any ZIP without needing to upgrade. But miners will of course have to upgrade in order to support the potential for the fork in the first place. So it covers down to balancing the usability of non-miner client software against the potential effect of the additional data per-transaction, which will be significantly higher if numbers are included.

If the motivation of using a bit field over a set of compact integers is space saving, then I don't see the point. It only costs one byte per hard-fork (in each transaction) for the compact integer option, until there are more than 253 ZIPs. Anyway, let's argue semantics first, not encodings. (See Wadler's Law.)

Note that checkpoints allow removing explicit fork opt-in/out indicators regardless of how they are encoded. The next software release after a hard fork should absolutely checkpoint it (or in the case of a split like the Ethereum DAO hard-fork situation, both software forks should checkpoint their own chain).

Another advantage of including opt-in/out indicators in transactions is that it provides some information about support for/opposition to a fork from transacting users, as opposed to miners. (Note, however, that this is not entirely reliable because either side could attempt to flood blocks with transactions indicating their point of view, so this probably shouldn't be used for automated fork gating criteria.)

Here's an example of what we are trying to prevent: https://mobile.twitter.com/desantis/status/758104013766266881

In step 5, Coinbase intends to send just ETH to Poloniex, but the transaction is replayed and they end up sending both ETH and ETC. If the transaction had been required to specify the fork, this couldn't have happened. (I'll refrain from assigning blame here since that's independent of the technical issue, but any way you look at it it's a total mess.)

See also https://fieryspinningsword.com/2015/08/25/how-i-learned-to-stop-worrying-and-love-the-fork/ , which predicted the possibility of this kind of attack in the context of Bitcoin-XT (search for "A prudent user").

Some half-baked thoughts about locking transactions to specific forks, that might help expand the potential solution space...

(1) An intentional hard fork could decree that after the cut-over block, acceptable signatures are no longer over the raw messages, but rather messages with some extra implicit fork-specific transformation – such as an implied message prefix, or xor-mask – with a fork-identifier. This yields a zero-overhead convention which prevents classic-transactions from being valid on the fork or vice-versa.

(2) The fork-identifier could be an autogenerated value – such as the hash of the first block in the fork. (This might imply the 1st conscious fork block must be empty – no transactions can be locked to the fork until its 'second genesis' exists - or perhaps, that one block is the only one with transactions valid in both forks?)

(3) More generally – for instance with respect to temporary forks – a transaction might choose to name a necessary predecessor identifier, for example via a block-height. The hash of the block at that height would be the fork-identifier which must be implicitly-mixed with the transaction before signing. Thus, at space-cost of height-value (~4 bytes?) in transactions, and validating clients keeping an indexed log of all block hashes, any transaction could ensure it is only valid given a specific prior history. (This is essentially the ~gmaxwell 'transaction block commitments` quoted in #174. However I believe the 'implicit pre-sign transformation' approach suggested above means only the block-height need be specified – no full or truncated hashes.)

(4) Further, perhaps the 'predecessor identifier(s)' of a transaction should have a zero-overhead default, such as: (a) the blockhash at the actual-height of the 1st referenced in-transaction; or (b) the xor of all block-hashes holding all referenced in-transactions. Yes, this implies every transaction relying on such defaults would need to know & commit-to existing blocks containing its inputs. If you wanted to issue transactions when all inputs aren't yet in a history-you're'-ready-to-commit-to, you'd use an alternate explicit mechanism to specify a predecessor block (such as the genesis block to 'wildcard' your transaction to all forks). In this way, most transactions leisurely built on other confirmed-transactions enforce strong history-affinities without extra overhead, but can still express looser preferences at extra cost when necessary.

(5) Things like payment channels, where valid transactions are held but only released if necessary, may require such flexible anchoring (only to their own inputs, or a consciously chosen prior state), as opposed to a firm lock-to-original-fork, in order to retain their value on both sides of forks that have otherwise tried to lock themselves against replays. (So: the simple scheme in (1) isn't enough in the presence of such 'pocket' transactions.)

Two thoughts:

• First of all we already have "block anchoring" for JoinSplits. If you anchor a JoinSplit to a commitment tree that only exists in one fork, your transaction cannot be replayed. To support other kinds of transactions wallets should move all funds through a single JoinSplit anchored to one of the forks.
• Can we just rely on the transaction version numbers if we don't support transactions emitted "near the hardfork height"? There could be a social rule saying "don't emit transactions within 10 blocks of the split target height".

I propose both of these as "good enough" assuming sufficient community engagement / communication mechanisms so that we don't need to incorporate any feature / code changes.

• I don't think we should try to predict whether a hard fork will be controversial. The mechanism for ensuring that a transaction can't be replayed between split chains should "just work" for any fork that ends up causing a split.
• The mechanism should also work for transactions that contain no JoinSplits.

Since transactions don't expire (see #754, but that won't be implemented before the first hard fork because it requires a hard fork to implement :-) ), there's no way to guarantee that a transaction submitted before the fork will get into the block chain before the fork. Therefore the social rule @nathan-at-least mentioned doesn't really work reliably.

Re: @gojomo's (very welcome!) suggestions,

(1) This seems too magical and implicit to me. It's generally a good idea, especially in protocols based on signing or hashing, for the inputs to the signing/hashing to be communicated explicitly by the protocol as opposed to computing them implicitly; the latter tends to produce more error-prone and complicated protocols.

(2) and (3) This option doesn't allow identifying the post-fork chain in transactions submitted before the forking block is known. That might not be a problem if we have a rule that transactions specifying a dependent block before the fork cannot be included in blocks after the fork (and are discarded in that case). However, mining and verifying software that is unaware of the fork won't follow this rule.

(4) This seems to be an optimization of (3), and the same objection applies.

(5) Good point; we need to consider this when specifying things like BOLT.

Let's keep considering the alternatives here, but this will definitely not be in z8, thus not in the "core consensus freeze". So if we do implement something for 1.0, we may need to insert a new milestone. We're still considering options to enable this kind of future proofing post-1.0 launch, on the assumption that it's extremely-widely accepted and no other controversial HF occurs prior to that.

I put this in the Beta release (z10) even though we may not do anything, simply so we don't drop this decision.

This conversation feels like it is growing out of control. I can't keep all the ideas and possibilities straight.

First of all, let's separate two things. I've just renamed this ticket to "transactions specify in advance which branch they will be valid on" and I've renamed #174 to "transactions specify retrospectively which branch they are valid on".

Second of all, I really question whether it would be a good thing if transactions could specify in advance that they would be invalid in certain branches. I think if I give you some Zcash today, it should be assumed to be valid on both of the branches of a future fork. Of course, once the fork happens, a branch might adopt rules that reject your transactions and make your Zcash invalid on that branch. That's the branch's prerogative — we can't (and wouldn't want to) stop them from doing that. But I suspect it may be more harmful than helpful if I can give you Zcash today which I get to pre-emptively choose will be invalid on certain branches tomorrow.

Also, that would raise the deep, vexing question of "How do I specify which branch I meant when I say this cash is invalid on that branch?". Simple names/numbers are obviously a terrible answer, because that induces a race and a conflict between branches to lay claim to the names/numbers that have the most valid transactions on them. Something like "the hash of the first block on the branch" is perfect, except that it is useless for a priori constraints like this. The only answer I can come up with is "Whatever branch(es) is(are) signed by a certain private key.".

So, this whole idea ("transactions specify in advance which branch they will be valid on") seems to boil down to "I can give you a transaction that says you can only spend it on a blockchain which is signed by a given private key". This is just another signal to me that this whole thing is a bad idea. :-)

I'm removing "engineering meeting agenda" from this ticket because I don't want us to spend time on it before Zcash 1.0 "Sprout" launches, and possible not ever.

Now the other feature — #174 — that is absolutely a good idea. It is necessary and sufficient to eliminate all (?) of the problems with replay of transactions from one branch to the other that we heard about during the ETH/ETC split. That one is also not going into Zcash 1.0 "Sprout", but I think we should probably add it soon after Sprout.

@zookozcash I think you're confusing whether cash is valid on a branch, with whether transactions are valid on a branch.

No-one disputes that if a cryptocurrency splits, then the cash at the split (i.e. created by transactions that were already in the blockchain at the split) is valid on both branches. But it is clearly incorrect for any given transaction to be implicitly valid on both branches. If you want that, you should have to explicitly submit the transaction on both branches. All we're arguing about is how to implement that, and whether certain restrictions can simplify the implementation.

Also, #174 does not solve the problem of transactions submitted before the split being valid on the unforked chain, even when they were intended not to be. A safe resolution to that problem would be to not allow any submitted transactions to be valid across a fork; that avoids any issue of how to refer to the branches before they exist.

@zookozcash wrote:

Also, that would raise the deep, vexing question of "How do I specify which branch I meant when I say this cash is invalid on that branch?". Simple names/numbers are obviously a terrible answer, because that induces a race and a conflict between branches to lay claim to the names/numbers that have the most valid transactions on them. Something like "the hash of the first block on the branch" is perfect, except that it is useless for a priori constraints like this. The only answer I can come up with is "Whatever branch(es) is(are) signed by a certain private key.".

ZIP numbers, as I proposed, seem to suffice perfectly well for this, or at least I do not see the problem with using them.

We've decided to go with #754 and #174 for HF0 rather than this (although this ticket has a lot of useful discussion).

We plan to do #2584 along with #2286 (comment), which isn't precisely the same as what I understand as the goal of this ticket, but it seems close enough and is simpler.

The main UX difference is that on the upgrading side of the chain split, the mempool will get cleared at the activation height, so there will be a window during which users may need to resubmit transactions. Note: because of #754, after NU0, clients of future upgrades should set expirations at the activation boundaries and follow the standard UX for expiry.

Therefore we're not going to implement this.