Simple DVT: Obol Testnet Results

February 06, 2024 in Ethereum, Node Operator by KimonSh (Will)

The Lido x Obol Simple DVT Testnet has officially ended, with the overall Lido x Obol validator set surpassing all of the minimum requirements to move forward to mainnet!

This was the 3rd testnet allowing Node Operators to run validators utilizing the Lido protocol with Obol’s DVT solution, following the first testnet in 2022 and second testnet in 2023, and the first DVT testnet by any staking protocol to be held on the Holesky testnet.

As described in the Simple DVT Module Proposal, participation in a Simple DVT testnet is a requirement for Node Operators seeking to participate in running validators using the Lido protocol via the Simple DVT Module on mainnet.

Operators (including solo stakers) interested in participating in the next Simple DVT Testnet can sign up here.

What is the Simple DVT Module?

The Simple DVT Module (SDVTM) is expected to be the second mainnet Lido protocol module, slated (pending DAO vote) to be added in Q1’24. The module is intended to battle test Obol and SSV Network based DVT on mainnet, while adding the first opportunity for solo and community stakers to run validators using the protocol.

According to the proposal, the module would be initially capped at 0.5% of total Lido stake (with the option to be increased via DAO vote) and is expected to allow for the addition of over 250 net-new Node Operators to the protocol in the first six months post-launch.

Importantly, the SDVTM is intended to be wound down within 3 years, during which time more scalable DVT modules with permissionless elements are expected to be added to the protocol.

The “Simple DVT Module Committee” will be responsible for creating and executing Easy Track motions specifically for Simple DVT that can create new clusters, activate and deactivate existing clusters, raise and lower cluster key limits, and change cluster manager and reward addresses. These processes will be executed via the Easy Track optimistic governance process with a 72 hour window for LDO holders to veto any motions made.

Mainnet participants and clusters will be proposed to the DAO by the Lido Node Operator Subgovernance Group (LNOSG), a sub-committee of the DAO made up of Lido on Ethereum and Lido on Polygon Node Operators, in addition to independent members of the staking community. The proposed participants and clusters will be posted to the Lido Research forums; if after a week there are no significant disagreements, the Simple DVT Module Committee will launch Easy Track motions to add the Node Operators to the Simple DVT Operator registry.

To learn more about the SDVTM, read the full proposal here.

Obol Testing

The Obol testing began in November 2023 on Holesky testnet, with a total of 214 Node Operator participants consisting of solo stakers, community stakers, and professional node operators. Participants were split into 32 clusters of 7 members (with a 5/7 threshold configuration) that in most cases attempted to match geographic regions and achieve diversity of underlying infrastructure type (e.g. home machines, bare metal in a data center, and public cloud).

Each cluster also contained at least one Node Operator from the Lido on Ethereum Curated set to ensure one participant experienced with the processes required to run validators through the protocol was included in each cluster. Eight of the clusters were formed (or filled with a replacement) to test greater geographic diversity, with nodes run by participants from a combination of either Europe and the Americas or the Asia Pacific Region and Europe. Overall, participants ran nodes from 39 different countries, including participants in North & South America, Europe, the Middle East, Africa, Asia, and Australia (fingers crossed one day we will include Antarctica!).

Each cluster started the process by choosing a “Cluster Coordinator”, a participant that would be responsible for creating a Safe multi-sig and initializing the Distributed Key Generation (DKG) ceremony for each cluster. Each cluster’s SAFE also consisted of a 5/7 threshold, and represented their cluster in the Lido Node Operator registry on Holesky.

Next, cluster participants each submitted and verified an “Individual Manager Address”, used for signing messages in the Safe and Obol cluster, and an optional “Individual Reward Address” that participants could choose to receive validator rewards to in lieu of their Individual Manager Address.

Once completed, Cluster Coordinators created their respective cluster Safes and participants configured their nodes, generated 100 validator keys via DKG, and submitted the keys to the Simple DVT Module on Holesky. Due to the early-stage nature of the Holesky testnet, MEV-Boost was not initially supported, requiring participants to directly connect to relays via their CL client. The relays used included the Eden Network relay, Flashbots relay, and Titan relay. During this period, some participants were replaced due to technical difficulties or lack of activity, leading to the final number of 195 participants that would complete the trial.

The testnet Simple DVT Module Committee raised key limits for each of the clusters over the course of a week to 5, and then began the initial monitoring period. After a week, attestation Effectiveness and Uptime results were promising (due to the large number of validators on Holesky, no trend was yet observed in the low number of block proposals) and validator key limits were raised to 50 for all clusters.

During the next monitoring period, it was observed that while Attestation Effectiveness and Uptime continued to perform well above the benchmarks, the aggregate Block Proposal Success Rate was well below the required 70% minimum threshold, at ~ 54%. While some clusters had successfully submitted all possible block proposals, a significant number of clusters had missed the vast majority of slots, and in some cases missed all of them.

At this point, the Obol team’s analysis of the missed proposals identified four main issues:

General NO misconfigurations (generally related to their beacon nodes);
Cluster latency;
Issues with Holesky MEV Boost relays, and;
Lack of a stable version for the MEV-Boost sidecar client on Holesky.

Soon after this analysis, Flashbots added MEV-Boost support to Holesky, the Obol team deployed an upgraded Charon version, v0.18.0, and also began additional troubleshooting with the relay teams. A rapid improvement was observed in the aggregate Block Proposal Success Rate, with an improvement to 63% in less than a week. As this improvement was being observed, cluster key limits were raised from 50 to 100, for 30 of the 32 clusters.

Two of the clusters, “Crimson Coyote” and “Glacial Gull” were among those with a 0% Block Proposal Success Rate. These clusters had both lost members due to inactivity during the course of testing, but were kept online to test potential performance impacts. Over the next weeks, the Glacial Gull cluster lost an additional member while another participant was having hardware issues, resulting in validator downtime. It was then determined that the validators should be exited by the remaining members and replacements filled to restart the cluster with a full 7 members. While the Crimson Coyote remained running in a 6/7 format, multiple participants had intermittent hardware issues, leading to sustained poor performance. For these reasons, these clusters were limited to a maximum of 50 active validators for the course of the trial.

It is a clear takeaway that for the mainnet implementation, when a cluster member goes inactive, before any serious issues can occur the cluster’s validators should be exited and a replacement participant will take the place in a new cluster.

Over the weeks following the Charon v0.18.0 upgrade the Block Proposal Success Rate continued to improve, and for the remaining 45 days of the test, the average aggregate Block Proposal Success Rate improved to 71.2%, surpassing the trial’s minimum requirement of 70%. As noted in the recent Simple DVT update post on January 17, the Block Proposal Success Rate for the 7 days preceding the end of testing remained above 85% for 6/7 days.

To conclude the testing, all of the validators were exited on January 22nd, and participants successfully completed claiming of rewards via the Simple DVT Reward Distribution process, as outlined in the January 17 update post.

Performance Results & Learnings for Mainnet

Aggregate Results

As seen in the image above, the aggregate metrics for the Lido x Obol testnet surpassed all of the Minimum Testnet Success Characteristics outlined in the Simple DVT Module Proposal, with 97.93% Uptime, 84.53% Attestation Effectiveness, and a 71.17% Block Proposal Success Rate (all metrics per Rated). In addition, outside of Block Proposals, Obol cluster performance also surpassed the tracked aggregate average for the entire Holesky network. As a result, the Obol trial passed all requirements for Obol based DVT to move forward to the mainnet SDVTM.

Cluster Results

At the cluster level, 18 of the 32 clusters surpassed the Minimum Block Proposal Success Rate benchmark of 70%, 9 finished with between 50% - 70%, and 5 finished with Block Proposal Success Rate lower than 50%. 29 of the clusters surpassed the minimum Uptime requirement, and 31 of the clusters surpassed the Avg. Attestation Effectiveness requirement.

Upon further investigation by the Obol team, four key factors were highlighted as reasons certain clusters underperformed: 1. Latency, 2. Hardware, 3. Beacon Node, and 4. MEV-Boost and Relays.

Latency

While most clusters were formed to maintain a non-extreme geographic distance (and therefore latency) between cluster participants, eight clusters contained a mixture of participants across continents in order to either test the impact on cluster performance or as a result of replacing a participant. Of these 8 clusters, 5 fell below at least one of the performance benchmarks.

Additionally, of the 6 clusters containing only participants from the Asia Pacific region, only two surpassed the benchmark. Issues with latency were clearly observed in these clusters, likely due to the significant size of the region. For example, while the distance from Lisbon to Moscow is approximately 3,900 km, the distance between Seoul and Sydney is over 8,300 km.

The results of this testnet have made clear that when Simple DVT is deployed on mainnet, an even greater focus must be placed on geographically tuning clusters.

Hardware & Beacon Node

In certain clusters with a higher number of solo and community staker participants, persistent hardware problems occurred throughout the trial. While there are a multitude of potential issues related to hardware, follow up surveys clearly show that many of those participants with issues were using low-performance VPS offerings, not running hardware with a sufficient number of CPUs, and in some cases using HDD storage.

This caused numerous issues related to Beacon Nodes falling out of sync, especially as the number of active validators per cluster increased.

As a result, participants moving forward to mainnet will have to confirm that their hardware specifications meet the minimum requirements recommended by Obol for mainnet.

MEV-Boost & Relays

Throughout the testnet, issues related to the early-stage of community infrastructure on Holesky were a challenge. Before the start of the testnet, MEV-Boost and publicly available MEV-relays from any of the major providers were not available on Holesky. Lido DAO and Obol contributors requested relay providers to stand up infrastructure for Holesky, and the teams from Eden Network, Flashbots, and Titan all graciously agreed to help support the testnet.

While an alpha version of MEV-Boost was made available during the validator setup phase of the trial, it was determined to wait for a more stable release before connecting to the sidecar. As a result, participants directly configured relays to their beacon nodes for the first weeks that validators were online.

While overall relay performance was suitable to start, certain issues expected in the normal course of testing were encountered. Exacerbating this, issues such as overbidding and suboptimal geographical relay placement for some clusters impacted performance.

Following the Charon v0.18.0 upgrade, MEV-Boost was added to all of the participant’s nodes. The introduction of MEV-Boost added another set of challenges for some clusters that were already experiencing latency issues. Per Obol’s analysis of the testing, in 25% of cases, MEV-Boost related latency added over 2 seconds to the process while each node fetched the block header (vs. the ~ 4 second requirement for the entire process to be concluded for chain inclusion).

Later in the trial, a mixed approach was taken where some clusters continued to use MEV-Boost and others went back to connecting to a single relay via the beacon node. While those directly connected generally showed superior performance, the benefits also must be weighed against the optionality MEV-Boost provides in receiving bids from multiple relays.

While some of these issues are expected to be less of an issue on mainnet given the maturity and higher hardware investments by relay providers, a mixed approach between clusters utilizing MEV-Boost and relays connected to the beacon node will likely be taken as performance results are analyzed. Additionally, the Obol team was able to take advantage of the testing to come away with additional insights into the relationship between Obol DV nodes and MEV-Boost and relays, where continuing improvements to the Charon client will likely drive higher performance over time.

Among other takeaways from the trial, it is clear that additional focus will be paid to improving the alerting resources for SDVTM participants. While Lido contributors have already made the Ethereum-Validators-Monitoring dashboards open source, additional work is underway to help improve the alerting resources available to SDVTM participants.

Participant Performance

While specific participant performance will not be discussed in this blog post, there is an important point to note: participants in clusters that did not reach the minimum performance benchmarks are not automatically excluded from moving forward to mainnet.

In many cases, these clusters contained a majority of Node Operators who were highly cooperative with their clusters, responsive to updates and testing requirements and whose nodes were performant.

The LNOSG will meet in the coming weeks to examine the quantitative performance results of the trial at the aggregate, cluster, and participant level and will also have the opportunity to examine the qualitative metrics obtained via survey and notes from the trial from the Lido DAO and Obol contributors.

Following their meeting, a forum post will be made on the Lido research forums to propose the clusters and participants to move forward to mainnet in the first stage (and potentially for the second stage). Each participant will receive an email with individual feedback noting their status and inviting them to participate in the next testnet if they so choose.

Path to Mainnet

Following the posting of the LNOSG suggested list to the forums, the DAO will have one week to discuss the proposal and state any objections. If no objections arise, clusters will begin the coordination process and the Simple DVT Module Committee will commence Easy Track motions (which can be rejected by LDO token holders) to register the clusters on the Lido Simple DVT Module registry.

In parallel, the SSV Network trial is currently underway and expected to be complete by the end of February or early March. Upon completion, a similar blog post will be shared with the community and an additional LNOSG process will follow for those participants.

When the steps outlined above are completed (pending success of the SSV trial) and SSV clusters are added to the registry, the Simple DVT Module Committee will raise key limits for both Obol and SSV clusters to 5, and a 30 day monitoring period will begin.

Following this monitoring period, performance will be shared with the DAO. If the results show strong performance, cluster key limits will be raised again and additional clusters will be added to the module.

Future Simple DVT Testnets

The next round of Simple DVT testnets are expected to commence in late March or early April. The requirements to participate are provable experience running an Obol node or SSV Operator. All solo stakers, community stakers, and professional node operators are invited to apply.

Please fill out this form if you are interested in participating.

Depending on the number of participants that move forward from these two testnets, it is possible the module will reach stake capacity. If that is the case, a discussion and vote can be held for the DAO to consider raising the 0.5% of Lido stake limit.

Other Notes

A huge huge thank you is due to all of the community and ecosystem members that made this testnet possible.

Without the support of the relay and MEV infrastructure teams from Eden Network, Flashbots, and Titan, the testing and takeaways from this testnet would not have been possible.

Next, thanks go to the Protofire team, who with the support of LEGO, deployed and maintained an instance of Safe on Holesky that was pivotal to the trial, and to Rated, for setting up Holesky monitoring earlier than planned.

Also, thank you to the Obol Labs team, who have continued to drive significant improvements to the Obol technology suite and for all of their diligent work assisting participants during the testnet. The results of this testing show that Obol based DVT is not only feasible, but will drive further decentralization across infrastructure, software, and geographies for Ethereum based validators. See Obol’s analysis of the testnet here.

Finally, and most importantly, to all testnet participants: thank you for keeping your nodes running, going through multiple rounds of upgrades and changes, and especially for your enthusiasm in helping to drive the decentralization of the Ethereum network.

Obol Testnet #3 Participants

Cluster 1: dgnatiuk, demrwr, HashKey Cloud, Nodera, Raccoon Nodes, Serenita, SuperJax
Cluster 2: Chainnodes, kobzar3830, PowerStaking, sodiumstar, Stakely, stellar_the_one, systemd
Cluster 3: _bara_kuda, Dappnode, Eridian, Piconbello, RockawayX, TRUPROCRYPTO
Cluster 4: Chainode Tech, DVStakersSpacesider, Kukis Global, kvqd777, MGTeam, minivipers, Valakas
Cluster 5: Cryptoria, Liquify LTD, narko2t1, natalia3647, Node3.Tech, RockLogic GmbH, SECARD
Cluster 6: 01node, Deutsche Telekom, Simply Staking, SpaceX, Thenop.io, TrustedAdvizer, vladislav7137
Cluster 7: A41, Blockblaz(G11 tech labs pvt. ltd.), DVStakersSpacesider, Kunyoung Kim - IT Times.com, Lefey, StakeWithUs
Cluster 8: anvel, H2O Nodes, Metanull, ramza107, SenseiNode, Sub7 Security, Web3DAO
Cluster 9: Barracuda, Everstake, GraphOps, nodeproxyz, Power Intelligence, Republic Crypto, yura_zp
Cluster 10: Blockscape, Cosmostation, dimsome, Everlasting Global, nodeADDICT, testnet.cn, thucnguyen#8149
Cluster 11: Allnodes, amarkelov, Dappnode, knightsemplar, natalia_256, TdrSys, Yutu
Cluster 12: BeeHive, Cryptology, Eridian, jayjay, LinkPool, Nethermind, VanGogh
Cluster 13: CryptoManufaktur, farukyasar, F5 Nodes, Las01, LIVE.NODE, maxim_101, Node Guardians
Cluster 14: Cryptology, Finoa, Nokey, nodeskuge, noxuspace, Stakin, TRUPROCRYPTO
Cluster 15: BeeHive, Chainlayer, deNodes, katesizova, Tesla, testovich, val4n17
Cluster 16: Chainlayer, ContributionDAO, Lavender.Five Nodes, Pier Two, RockX, Starnodes, Validation Cloud
Cluster 17: DSRV, goldstream777, Infinite Lux Staking, MGTeam, Power Intelligence, rrrmmmmm, Spectrum Staking
Cluster 18: Launchnodes, mahof, Metanull, Piconbello, SenseiNode, Staking4All, Steaking Frens, UniqNodes, Web3DAO
Cluster 19: alkadelta, Konstantin#1194, P2P.org, PhiNodes, Starnodes, Светлана1969#2890, yellowbee#7307
Cluster 20: 🅰🅻🅴🆇ⒾⓉ, antonduzhenko, Infstones, Nodes.Guru, Range, systemd, Weaitonamazerid
Cluster 21: AntNodes, Conqueror, Highnok, LIVE.NODE, Mahof, P-OPS Team, Staking Facilities
Cluster 22: Ebunker, GoldenTrust, Imperator.co, KingSuper, RockX, Validation Cloud, Youngha Kim - IT Times.com
Cluster 23: 01Node, Cryptofisher, Investernco, Kukis Global, Liquify LTD., Pacobits, Polkachu
Cluster 24: HashKey Cloud, HellmanResearch, Investernco, kobya4evo, Luck#7063, NodeInfra
Cluster 25: GlobalStake, Investernco, irina#7966, lcofjurn, NakoTurk, RockawayX, StakingCabin
Cluster 26: Applepai, ChainOps, Coinstamp, CryptoManufaktur, CVJoint, H2O Nodes, rodion007#5553
Cluster 27: Astronodes, Colinka | BeeHive,DMITRY | SCANDALIST, DragonStake, lesya, Simply Staking, Swiss Staking
Cluster 28: Blockpower, Crouton Digital, daniilkir, Eridian, iicc1, SNC.xyz, Stakin
Cluster 29: A41, Cat6, Chainbase, Forbole, Luganodes, StakeWithUs
Cluster 30: Anonstake, archimedes0159, cryptozab, D-Stake, guglez, igorzp60, P2P.org
Cluster 31: 1to, Andrei0707#1159, Blackb0x, Bware Labs, Cosmostation, DSRV, OranG3cluB
Cluster 32: Di-nodes, goooodnes, Metanull, SenseiNode, SpaceX, tungnguyen.zk