Skip to content

Refactor consensus/types#7827

Merged
mergify[bot] merged 14 commits into
sigp:unstablefrom
macladson:modular-types
Dec 4, 2025
Merged

Refactor consensus/types#7827
mergify[bot] merged 14 commits into
sigp:unstablefrom
macladson:modular-types

Conversation

@macladson

@macladson macladson commented Aug 5, 2025

Copy link
Copy Markdown
Member

Proposed Changes

Organize and categorize consensus/types into modules based on their relation to key consensus structures/concepts.
This is a precursor to a sensible public interface.

While this refactor is very opinionated, I am open to suggestions on module names, or type groupings if my current ones are inappropriate.

Reviewer Guide

While this is a 2000+ line diff, I don't believe the review should be difficult as the vast majority of the changes are from the following:

  1. Restructuring internal file imports (usually from something like use crate::* to a more explicit and verbose import. The imports will take the order std, external, internal with a new line between each category of import. For example:
use std::collections::HashSet;

use serde::{Decode, Encode};

use crate::block::BeaconBlockBody;

This format is one of the available options in nightly rustfmt: https://rust-lang.github.io/rustfmt/?version=latest&search=#group_imports and is my personal preference.

  1. Removing generic Error names in favour of more specific error names e.g. BeaconStateError.
  2. Reorganizing exports into various mod.rs files. lib.rs can then just import each module directly.
  3. Various temporary re-exports and facade modules to maintain backwards compatibility (see below).

Future Work

Import Structure

To (mostly) keep backwards compatibility with the rest of Lighthouse (and to stop this PR from exploding in size and scope), we export all public types at the root level. For example, you can do use types::BeaconBlock. There is also various facade modules to keep the original import structure. These can be removed 1-by-1 over time in small discreet PRs.

For us to transition the types crate to a public library, it is my belief that we should require imports be at the module level (except for very core types such as Hash256, EthSpec, Slot, etc). So we should instead require use types::block::BeaconBlock. In my view, this helps keep the namespaces and docs cleaner and enables easier IDE type discovery. Perhaps more subjectively, I think that it also aids DevEx as it becomes easier to see which types are being imported at a glance since related types are necessarily grouped together.

Therefore, we should update the rest of the types imports across Lighthouse to do the following:

  • Remove all instances of use types::*;
  • Swap all root level imports to use their module variants if applicable
  • Remove the root-level re-exports from types/src/lib.rs

Similarly, we should remove any re-exports of the types crate in other crates (eth2, beacon_chain) which I don't believe is worth it. There are a plethora of situations where the same file will export types from multiple versions of the same crate through different re-exports. We should avoid situations like this and import only from types directly.

External Crate Re-Exports

We are re-exporting several crates inside types.

// Temporary re-exports to maintain backwards compatibility for Lighthouse.
pub use bls::{
    get_withdrawal_credentials, AggregatePublicKey, AggregateSignature, Error as BlsError, Keypair,
    PublicKey, PublicKeyBytes, SecretKey, Signature, SignatureBytes, PUBLIC_KEY_BYTES_LEN,
    SIGNATURE_BYTES_LEN,
};
pub use context_deserialize::ContextDeserialize;
pub use context_deserialize_derive::context_deserialize;
pub use fixed_bytes::FixedBytesExtended;
pub use milhouse::{self, List, Vector};
pub use ssz_types::{typenum, typenum::Unsigned, BitList, BitVector, FixedVector, VariableList};
pub use superstruct::superstruct;

My opinion is that we should eventually remove all these re-exports (or at least most of them).

Feature Gates

I think this refactor enables potential feature gating of certain groups of types. The most obvious one being light_client. The module is entirely distinct and not relied upon by any other module (outside of certain consts which could easily be moved).

@macladson macladson added work-in-progress PR is a work-in-progress code-quality labels Aug 5, 2025
@macladson macladson force-pushed the modular-types branch 3 times, most recently from 6bcaa2a to 9da67f9 Compare August 5, 2025 16:23
@macladson macladson added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels Aug 19, 2025
@mergify

mergify Bot commented Aug 20, 2025

Copy link
Copy Markdown

This pull request has merge conflicts. Could you please resolve them @macladson? 🙏

@mergify mergify Bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Aug 20, 2025
@macladson macladson mentioned this pull request Aug 21, 2025
@macladson macladson added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Sep 8, 2025
@macladson

Copy link
Copy Markdown
Member Author

There seems to be some issue with the genesis sync tests but I don't believe they are specific to this PR

@mergify

mergify Bot commented Sep 8, 2025

Copy link
Copy Markdown

Some required checks have failed. Could you please take a look @macladson? 🙏

@mergify mergify Bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Sep 8, 2025
@mergify mergify Bot added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Sep 11, 2025
@mergify

mergify Bot commented Sep 18, 2025

Copy link
Copy Markdown

This pull request has merge conflicts. Could you please resolve them @macladson? 🙏

@mergify mergify Bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Sep 18, 2025
Comment on lines -69 to -79
#[derive(Debug)]
pub enum Error {
ArithError(ArithError),
InvalidCustodySubnetCount(u64),
}

impl From<ArithError> for Error {
fn from(e: ArithError) -> Self {
Error::ArithError(e)
}
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this error type since from what I could tell it was unused

@macladson macladson added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Oct 28, 2025
@mergify

mergify Bot commented Nov 2, 2025

Copy link
Copy Markdown

This pull request has merge conflicts. Could you please resolve them @macladson? 🙏

@mergify mergify Bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Nov 2, 2025

@pawanjay176 pawanjay176 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking great 🎉 Thanks a lot for leading this effort and sorry for the delay in reviewing.

Just a couple minor nits and a suggestion to adhere to module_inception lint. Lmk what you think

Comment thread consensus/types/src/beacon_response.rs
Comment thread common/eth2/src/types.rs Outdated
Comment thread consensus/types/Cargo.toml Outdated
Comment thread consensus/types/src/attestation/indexed_attestation.rs Outdated
Comment thread consensus/types/src/block/beacon_block.rs Outdated
Comment thread consensus/types/src/execution/execution_block_header.rs Outdated
Comment thread consensus/types/src/fork/fork_context.rs Outdated
Comment thread consensus/types/src/light_client/error.rs
Comment thread consensus/types/src/state/historical_batch.rs Outdated
Comment thread consensus/types/Cargo.toml
@pawanjay176 pawanjay176 added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Dec 3, 2025

@eserilev eserilev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, I can tell that there was a lot of thought put into these changes, its very much appreciated. I'm on board with the changes, we could always tweak things in subsequent PR's if needed

LGTM!

@pawanjay176 pawanjay176 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LFG🚀 🚀

@michaelsproul michaelsproul left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to merge too, thanks Mac for sticking with this, and Pawan for the thorough review as well!

@pawanjay176 pawanjay176 added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Dec 4, 2025
@mergify

mergify Bot commented Dec 4, 2025

Copy link
Copy Markdown

Merge Queue Status Beta

✅ The pull request has been merged

This pull request spent 39 minutes 38 seconds in the queue, including 37 minutes 2 seconds running CI.
The checks were run on draft #8534.

Required conditions to merge
  • check-success=local-testnet-success
  • check-success=test-suite-success

@mergify mergify Bot added the queued label Dec 4, 2025
mergify Bot added a commit that referenced this pull request Dec 4, 2025
@mergify mergify Bot merged commit 4e958a9 into sigp:unstable Dec 4, 2025
36 checks passed
@mergify mergify Bot removed the queued label Dec 4, 2025
@macladson macladson deleted the modular-types branch December 4, 2025 10:05
@kevaundray

Copy link
Copy Markdown
Contributor

My favourite type of PRs!

kevaundray added a commit to eth-act/lighthouse that referenced this pull request Dec 13, 2025
* Update local testnet scripts for the fulu fork (sigp#8489)

* Remove `fulu-devnet-3` testing on CI
* Delete `scripts/local_testnet/network_params_das.yaml` and consolidate it into the main `network_params.yaml` file we use on CI
* Delete enclave before building image, so it doesn't cause slow image building.


  


Co-Authored-By: Jimmy Chen <[email protected]>

* Fix data columns sorting when reconstructing blobs (sigp#8510)

Closes sigp#8509


  


Co-Authored-By: Antoine James <[email protected]>

* Instrument attestation signing. (sigp#8508)

We noticed attestation signing taking 2+ seconds on some of our hoodi nodes and the current traces doesn't provide enough details. This PR adds a few more spans to the `attestation_duty_cycle` code path in the VC.

Before:
<img width="842" height="154" alt="image" src="https://github.com/user-attachments/assets/cfc5c8c0-e6f2-4f56-a8e4-65001af4a005" />

After:
<img width="496" height="217" alt="image" src="https://github.com/user-attachments/assets/c91cfa88-af1b-456e-8c64-625809eb6881" />


  


Co-Authored-By: Jimmy Chen <[email protected]>

* Always use committee index 0 when getting attestation data (sigp#8171)

* sigp#8046


  Split the function `publish_attestations_and_aggregates` into `publish_attestations` and `handle_aggregates`, so that for attestations, only 1 task is spawned.


Co-Authored-By: Tan Chee Keong <[email protected]>

Co-Authored-By: chonghe <[email protected]>

Co-Authored-By: Michael Sproul <[email protected]>

Co-Authored-By: Michael Sproul <[email protected]>

* Move deposit contract artifacts to /target (sigp#8518)

Alternative to:

- sigp#8488


  Refactors deposit_contract crate to comply with Rust build conventions by placing generated artifacts in the build output directory.


Co-Authored-By: Michael Sproul <[email protected]>

* Move beacon state endpoints to a separate module. (sigp#8529)

Part of the http api refactor to move endpoint handlers to separate modules.

This should improve code maintainability, incremental compilation time and rust analyzer performance.


  


Co-Authored-By: Jimmy Chen <[email protected]>

* Refactor `consensus/types` (sigp#7827)

Organize and categorize `consensus/types` into modules based on their relation to key consensus structures/concepts.
This is a precursor to a sensible public interface.

While this refactor is very opinionated, I am open to suggestions on module names, or type groupings if my current ones are inappropriate.


Co-Authored-By: Mac L <[email protected]>

* Move validator http endpoints to a separate module (sigp#8536)

Continuation of:
* sigp#8529

Moving `/validator` endpoints out of `http_api` to a separation module. This should improve code maintainability, incremental compilation time and rust analyzer performance.

This is a tedious but straight forward change, so we're going with a pair & insta-merge approach to avoid painful & slow async review. @michaelsproul and I paired on the first commit - I believe we are almost done, will pair with @pawanjay176 tomorrow to wrap it up and merge tomorrow. (cc @macladson )


  


Co-Authored-By: Jimmy Chen <[email protected]>

* Move beacon pool http api to its own separate module (sigp#8543)

Continuation of:
* sigp#8536

Moving `/beacon/pool` endpoints out of `http_api` to a separation module. This should improve code maintainability, incremental compilation time and rust analyzer performance.

This is a tedious but straight forward change, so we're going with a pair & insta-merge approach to avoid painful & slow async review


  


Co-Authored-By: Jimmy Chen <[email protected]>

* Reduce `eth2` dependency space  (sigp#8524)

Remove certain dependencies from `eth2`, and feature-gate others which are only used by certain endpoints.

| Removed | Optional | Dev only |
| -------- | -------- | -------- |
| `either` `enr` `libp2p-identity` `multiaddr` | `protoarray` `eth2_keystore` `eip_3076` `zeroize` `reqwest-eventsource` `futures` `futures-util` | `rand` `test_random_derive` |

This is done by adding an `events` feature which enables the events endpoint and its associated dependencies.
The `lighthouse` feature also enables its associated dependencies making them optional.

The networking-adjacent dependencies were removed by just having certain fields use a `String` instead of an explicit network type. This means the user should handle conversion at the call site instead. This is a bit spicy, but I believe `PeerId`, `Enr` and `Multiaddr` are easily converted to and from `String`s so I think it's fine and reduces our dependency space by a lot. The alternative is to feature gate these types behind a `network` feature instead.


Co-Authored-By: Mac L <[email protected]>

* Clarify `alloy` dependencies (sigp#8550)

Previously, we had a pinned version of `alloy` to fix some crate compatibility issues we encountered during the migration away from `ethers`. Now that the migration is complete we should remove the pin. This also updates alloy crates to their latest versions.


Co-Authored-By: Mac L <[email protected]>

* Remove `consensus/types` re-exports (sigp#8540)

There are certain crates which we re-export within `types` which creates a fragmented DevEx, where there are various ways to import the same crates.

```rust
// consensus/types/src/lib.rs
pub use bls::{
AggregatePublicKey, AggregateSignature, Error as BlsError, Keypair, PUBLIC_KEY_BYTES_LEN,
PublicKey, PublicKeyBytes, SIGNATURE_BYTES_LEN, SecretKey, Signature, SignatureBytes,
get_withdrawal_credentials,
};
pub use context_deserialize::{ContextDeserialize, context_deserialize};
pub use fixed_bytes::FixedBytesExtended;
pub use milhouse::{self, List, Vector};
pub use ssz_types::{BitList, BitVector, FixedVector, VariableList, typenum, typenum::Unsigned};
pub use superstruct::superstruct;
```

This PR removes these re-exports and makes it explicit that these types are imported from a non-`consensus/types` crate.


Co-Authored-By: Mac L <[email protected]>

* Fix testnet script (sigp#8557)

Fix an issue where a kurtosis testnet script was failing because no supernodes were provided


```
There was an error interpreting Starlark code
Evaluation error: fail: Fulu fork is enabled (epoch: 0) but no supernodes are configured, no nodes have 128 or more validators, and perfect_peerdas_enabled is not enabled. Either configure a supernode, ensure at least one node has 128+ validators, or enable perfect_peerdas_enabled in network_params with 16 participants.
at [github.com/ethpandaops/ethereum-package/main.star:83:57]: run
at [github.com/ethpandaops/ethereum-package/src/package_io/input_parser.star:377:17]: input_parser
at [0:0]: fail
```


  


Co-Authored-By: Eitan Seri-Levi <[email protected]>

Co-Authored-By: Pawan Dhananjay <[email protected]>

* Do not request attestation data when attestation duty is empty (sigp#8559)

Co-Authored-By: Tan Chee Keong <[email protected]>

* Rust 1.92 lints (sigp#8567)

Co-Authored-By: Eitan Seri-Levi <[email protected]>

---------

Co-authored-by: Jimmy Chen <[email protected]>
Co-authored-by: Jimmy Chen <[email protected]>
Co-authored-by: 0xMushow <[email protected]>
Co-authored-by: Antoine James <[email protected]>
Co-authored-by: chonghe <[email protected]>
Co-authored-by: Michael Sproul <[email protected]>
Co-authored-by: Michael Sproul <[email protected]>
Co-authored-by: Mac L <[email protected]>
Co-authored-by: Eitan Seri-Levi <[email protected]>
Co-authored-by: Pawan Dhananjay <[email protected]>
Co-authored-by: Tan Chee Keong <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

code-quality ready-for-merge This PR is ready to merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants