Exploring Distributed Validator Technology with SafeStake
This blog post covers the results of an initial pilot testnet integration with SafeStake, a Distributed Validator Technology (DVT) provider. Conducted in a similar manner to the initial pilot testnets with Obol and SSV pre-Simple DVT, this testnet was aimed at assessing whether SafeStake DVT infrastructure could be utilized by Node Operators to run Distributed Validators (DVs) using the Lido protocol and to examine cluster performance.
Over the past two months, 17 node operators (NOs) have participated in the SafeStake testnet on Lido on Holesky. For the duration of the testnet, attestation performance was generally found to be quite performant, but the majority of clusters faced issues with block proposals. Additionally there was a slashing incident which occurred as a result of a faulty SafeStake software upgrade towards the end of the testnet.
Lido on Holesky SafeStake Pilot
Participant Distribution
The Pilot Lido x SafeStake Testnet started in late April 2024 on the Holesky testnet, with 17 total participants including 2 solo stakers & community stakers, 14 professional node operators, and SafeStake.
Participants were split into 5 clusters of 4, with an effort to minimize geography-related latency and achieve a diversity of infrastructure types (e.g. bare metal servers, home machines, and public cloud). One of the clusters was a SafeStake-only cluster and used as a benchmark to compare against other clusters.
Similarly to the other pilot testnets, each cluster included at least one member of the Lido on Ethereum Curated Node Operator set. In addition, an effort was made to include participants that had indicated significant experience with SafeStake based DVT during their applications.
At the aggregate level, participants from 13 countries across North & South America, Europe, Asia, and Australia came together to run nodes.
The Process
Each cluster started the process by choosing a “Cluster Coordinator”, a participant that would be responsible for creating a Safe multi-sig.
Each cluster, and the corresponding Safe multisig consisted of a 3/4 threshold, and represented their cluster in the Lido Node Operator registry on Holesky. Each participant was responsible for setting up their SafeStake operator and sharing their operator ID with the rest of the cluster participants.
Once completed, a single Node Operator (NO) from each cluster (usually the coordinator) generated 100 validator keys and then manually split the keys before submitting them to the Simple DVT Module on Holesky.
It’s important to note that no Distributed Key Generation (DKG) was used in this process, as the key splitting was performed by the NO itself. While the Simple DVT (SDVT) module was used, this pilot was not considered an SDVT trial, as it was the first time SafeStake was being piloted.
The participants started off by using MEV Boost with Aestus, Flashbots, Titan, and Ultrasound as relays. Due to a current limitation of SafeStake’s implementation, the validators were not able to set the appropriate Execution Layer Rewards Vault address as the specified fee address recipient as part of this testnet.
After the validators were activated, low block proposal success rates were observed, despite high validator uptime. After analysis by the SafeStake team, they determined that this was due to incorrect MEV-boost configuration as part of the participants’ operator setups.
While the SafeStake cluster was able to submit some block proposals successfully, the other clusters were unable to successfully propose any blocks.
After the rollout of a software update intended to address block proposal failures during the Lido x SafeStake Testnet, an issue arose where validator slashing occurred due to inconsistent attestation duties.
The root cause was traced to running two conflicting versions of the SafeStake software simultaneously within the Clever Coyote cluster, which created a scenario where two leaders handled attestation duties differently, leading to double attestations for the same slot.
While the primary focus was on the validators within Lido's SafeStake pilot on Holesky, other SafeStake validators outside of the Lido pilot were also impacted, leading to a slashing event for 15 validators in the Clever Coyote cluster and additional slashing across non-Lido SafeStake clusters.
The issue stemmed from a lack of version management between the old and new SafeStake software versions (Post-Mortem).
The older version allowed two leaders to propose attestation duties for the same slot, while the updated version enforced a single-leader model. The conflict between the two software versions resulted in different attestations being submitted for the same block, triggering the slashing.
To mitigate the issue, validators were halted across affected clusters, and a new version of the software was deployed to resolve the issue. The operators were instructed to upgrade to the fixed version before restarting their validators. This process ensured that validators resumed operation without further slashing incidents.
Soon after, the testnet was concluded. All validators were exited on September 7th, and participants successfully completed claiming of rewards via the SafeStake Reward Distribution process.
Performance Results
SafeStake Cluster Aggregate Results
As seen in the image above, the aggregate metrics for this Lido x SafeStake testnet completed with 91.86% Uptime, 71.56% Attestation Effectiveness, and a 10.59% Block Proposal Success Rate (all metrics per Rated).
Cluster Results
Next Steps
While this SafeStake testnet has demonstrated proof of concept, there are still a number of changes and improvements that need to be made and further testing that will need to be undergone before another testnet on Holesky with SafeStake takes place via the Lido protocol.
Some of the enhancements necessary include further support for block proposals, distributed key and deposit data generation, and custom fee address recipient address. Among these features, the addition of DKG will be a critical element to reduce the trust requirements currently necessary for creating and submitting validators.
DVT continues to be a promising method to enable solo operators and permissionless entry into the node operator set while further decentralizing validator infrastructure using the Lido protocol.
Appendix: SafeStake Pilot Participants
Cluster 1: DSRV, Ebunker, Hashkey, Pier Two
Cluster 2: ChainSafe, Chainup, Enti, InfStones
Cluster 3: Blockscape, Launchnodes, Openbitlab, RockLogic
Cluster 4: Allnodes, CryptoManufaktur, Kukis Global, Stakely
Cluster 5: SafeStake