Refactor consensus/types#7827
Conversation
6bcaa2a to
9da67f9
Compare
9da67f9 to
ffe0a71
Compare
|
This pull request has merge conflicts. Could you please resolve them @macladson? 🙏 |
ffe0a71 to
68c9d29
Compare
|
There seems to be some issue with the genesis sync tests but I don't believe they are specific to this PR |
|
Some required checks have failed. Could you please take a look @macladson? 🙏 |
|
This pull request has merge conflicts. Could you please resolve them @macladson? 🙏 |
| #[derive(Debug)] | ||
| pub enum Error { | ||
| ArithError(ArithError), | ||
| InvalidCustodySubnetCount(u64), | ||
| } | ||
|
|
||
| impl From<ArithError> for Error { | ||
| fn from(e: ArithError) -> Self { | ||
| Error::ArithError(e) | ||
| } | ||
| } |
There was a problem hiding this comment.
I removed this error type since from what I could tell it was unused
|
This pull request has merge conflicts. Could you please resolve them @macladson? 🙏 |
pawanjay176
left a comment
There was a problem hiding this comment.
This is looking great 🎉 Thanks a lot for leading this effort and sorry for the delay in reviewing.
Just a couple minor nits and a suggestion to adhere to module_inception lint. Lmk what you think
Co-authored-by: Pawan Dhananjay <[email protected]>
michaelsproul
left a comment
There was a problem hiding this comment.
Happy to merge too, thanks Mac for sticking with this, and Pawan for the thorough review as well!
Merge Queue Status
✅ The pull request has been merged This pull request spent 39 minutes 38 seconds in the queue, including 37 minutes 2 seconds running CI. Required conditions to merge
|
|
My favourite type of PRs! |
* Update local testnet scripts for the fulu fork (sigp#8489) * Remove `fulu-devnet-3` testing on CI * Delete `scripts/local_testnet/network_params_das.yaml` and consolidate it into the main `network_params.yaml` file we use on CI * Delete enclave before building image, so it doesn't cause slow image building. Co-Authored-By: Jimmy Chen <[email protected]> * Fix data columns sorting when reconstructing blobs (sigp#8510) Closes sigp#8509 Co-Authored-By: Antoine James <[email protected]> * Instrument attestation signing. (sigp#8508) We noticed attestation signing taking 2+ seconds on some of our hoodi nodes and the current traces doesn't provide enough details. This PR adds a few more spans to the `attestation_duty_cycle` code path in the VC. Before: <img width="842" height="154" alt="image" src="https://github.com/user-attachments/assets/cfc5c8c0-e6f2-4f56-a8e4-65001af4a005" /> After: <img width="496" height="217" alt="image" src="https://github.com/user-attachments/assets/c91cfa88-af1b-456e-8c64-625809eb6881" /> Co-Authored-By: Jimmy Chen <[email protected]> * Always use committee index 0 when getting attestation data (sigp#8171) * sigp#8046 Split the function `publish_attestations_and_aggregates` into `publish_attestations` and `handle_aggregates`, so that for attestations, only 1 task is spawned. Co-Authored-By: Tan Chee Keong <[email protected]> Co-Authored-By: chonghe <[email protected]> Co-Authored-By: Michael Sproul <[email protected]> Co-Authored-By: Michael Sproul <[email protected]> * Move deposit contract artifacts to /target (sigp#8518) Alternative to: - sigp#8488 Refactors deposit_contract crate to comply with Rust build conventions by placing generated artifacts in the build output directory. Co-Authored-By: Michael Sproul <[email protected]> * Move beacon state endpoints to a separate module. (sigp#8529) Part of the http api refactor to move endpoint handlers to separate modules. This should improve code maintainability, incremental compilation time and rust analyzer performance. Co-Authored-By: Jimmy Chen <[email protected]> * Refactor `consensus/types` (sigp#7827) Organize and categorize `consensus/types` into modules based on their relation to key consensus structures/concepts. This is a precursor to a sensible public interface. While this refactor is very opinionated, I am open to suggestions on module names, or type groupings if my current ones are inappropriate. Co-Authored-By: Mac L <[email protected]> * Move validator http endpoints to a separate module (sigp#8536) Continuation of: * sigp#8529 Moving `/validator` endpoints out of `http_api` to a separation module. This should improve code maintainability, incremental compilation time and rust analyzer performance. This is a tedious but straight forward change, so we're going with a pair & insta-merge approach to avoid painful & slow async review. @michaelsproul and I paired on the first commit - I believe we are almost done, will pair with @pawanjay176 tomorrow to wrap it up and merge tomorrow. (cc @macladson ) Co-Authored-By: Jimmy Chen <[email protected]> * Move beacon pool http api to its own separate module (sigp#8543) Continuation of: * sigp#8536 Moving `/beacon/pool` endpoints out of `http_api` to a separation module. This should improve code maintainability, incremental compilation time and rust analyzer performance. This is a tedious but straight forward change, so we're going with a pair & insta-merge approach to avoid painful & slow async review Co-Authored-By: Jimmy Chen <[email protected]> * Reduce `eth2` dependency space (sigp#8524) Remove certain dependencies from `eth2`, and feature-gate others which are only used by certain endpoints. | Removed | Optional | Dev only | | -------- | -------- | -------- | | `either` `enr` `libp2p-identity` `multiaddr` | `protoarray` `eth2_keystore` `eip_3076` `zeroize` `reqwest-eventsource` `futures` `futures-util` | `rand` `test_random_derive` | This is done by adding an `events` feature which enables the events endpoint and its associated dependencies. The `lighthouse` feature also enables its associated dependencies making them optional. The networking-adjacent dependencies were removed by just having certain fields use a `String` instead of an explicit network type. This means the user should handle conversion at the call site instead. This is a bit spicy, but I believe `PeerId`, `Enr` and `Multiaddr` are easily converted to and from `String`s so I think it's fine and reduces our dependency space by a lot. The alternative is to feature gate these types behind a `network` feature instead. Co-Authored-By: Mac L <[email protected]> * Clarify `alloy` dependencies (sigp#8550) Previously, we had a pinned version of `alloy` to fix some crate compatibility issues we encountered during the migration away from `ethers`. Now that the migration is complete we should remove the pin. This also updates alloy crates to their latest versions. Co-Authored-By: Mac L <[email protected]> * Remove `consensus/types` re-exports (sigp#8540) There are certain crates which we re-export within `types` which creates a fragmented DevEx, where there are various ways to import the same crates. ```rust // consensus/types/src/lib.rs pub use bls::{ AggregatePublicKey, AggregateSignature, Error as BlsError, Keypair, PUBLIC_KEY_BYTES_LEN, PublicKey, PublicKeyBytes, SIGNATURE_BYTES_LEN, SecretKey, Signature, SignatureBytes, get_withdrawal_credentials, }; pub use context_deserialize::{ContextDeserialize, context_deserialize}; pub use fixed_bytes::FixedBytesExtended; pub use milhouse::{self, List, Vector}; pub use ssz_types::{BitList, BitVector, FixedVector, VariableList, typenum, typenum::Unsigned}; pub use superstruct::superstruct; ``` This PR removes these re-exports and makes it explicit that these types are imported from a non-`consensus/types` crate. Co-Authored-By: Mac L <[email protected]> * Fix testnet script (sigp#8557) Fix an issue where a kurtosis testnet script was failing because no supernodes were provided ``` There was an error interpreting Starlark code Evaluation error: fail: Fulu fork is enabled (epoch: 0) but no supernodes are configured, no nodes have 128 or more validators, and perfect_peerdas_enabled is not enabled. Either configure a supernode, ensure at least one node has 128+ validators, or enable perfect_peerdas_enabled in network_params with 16 participants. at [github.com/ethpandaops/ethereum-package/main.star:83:57]: run at [github.com/ethpandaops/ethereum-package/src/package_io/input_parser.star:377:17]: input_parser at [0:0]: fail ``` Co-Authored-By: Eitan Seri-Levi <[email protected]> Co-Authored-By: Pawan Dhananjay <[email protected]> * Do not request attestation data when attestation duty is empty (sigp#8559) Co-Authored-By: Tan Chee Keong <[email protected]> * Rust 1.92 lints (sigp#8567) Co-Authored-By: Eitan Seri-Levi <[email protected]> --------- Co-authored-by: Jimmy Chen <[email protected]> Co-authored-by: Jimmy Chen <[email protected]> Co-authored-by: 0xMushow <[email protected]> Co-authored-by: Antoine James <[email protected]> Co-authored-by: chonghe <[email protected]> Co-authored-by: Michael Sproul <[email protected]> Co-authored-by: Michael Sproul <[email protected]> Co-authored-by: Mac L <[email protected]> Co-authored-by: Eitan Seri-Levi <[email protected]> Co-authored-by: Pawan Dhananjay <[email protected]> Co-authored-by: Tan Chee Keong <[email protected]>
Proposed Changes
Organize and categorize
consensus/typesinto modules based on their relation to key consensus structures/concepts.This is a precursor to a sensible public interface.
While this refactor is very opinionated, I am open to suggestions on module names, or type groupings if my current ones are inappropriate.
Reviewer Guide
While this is a 2000+ line diff, I don't believe the review should be difficult as the vast majority of the changes are from the following:
use crate::*to a more explicit and verbose import. The imports will take the orderstd, external, internalwith a new line between each category of import. For example:This format is one of the available options in nightly
rustfmt: https://rust-lang.github.io/rustfmt/?version=latest&search=#group_imports and is my personal preference.Errornames in favour of more specific error names e.g.BeaconStateError.mod.rsfiles.lib.rscan then just import each module directly.Future Work
Import Structure
To (mostly) keep backwards compatibility with the rest of Lighthouse (and to stop this PR from exploding in size and scope), we export all public types at the root level. For example, you can do
use types::BeaconBlock. There is also various facade modules to keep the original import structure. These can be removed 1-by-1 over time in small discreet PRs.For us to transition the
typescrate to a public library, it is my belief that we should require imports be at the module level (except for very core types such asHash256,EthSpec,Slot, etc). So we should instead requireuse types::block::BeaconBlock. In my view, this helps keep the namespaces and docs cleaner and enables easier IDE type discovery. Perhaps more subjectively, I think that it also aids DevEx as it becomes easier to see which types are being imported at a glance since related types are necessarily grouped together.Therefore, we should update the rest of the
typesimports across Lighthouse to do the following:use types::*;types/src/lib.rsSimilarly, we should remove any re-exports of the
typescrate in other crates (eth2,beacon_chain) which I don't believe is worth it. There are a plethora of situations where the same file will export types from multiple versions of the same crate through different re-exports. We should avoid situations like this and import only fromtypesdirectly.External Crate Re-Exports
We are re-exporting several crates inside
types.My opinion is that we should eventually remove all these re-exports (or at least most of them).
Feature Gates
I think this refactor enables potential feature gating of certain groups of types. The most obvious one being
light_client. The module is entirely distinct and not relied upon by any other module (outside of certain consts which could easily be moved).