Post Mortem: Disrupted rewards distribution due to missed oracle reports

On November 9 and 11, 2022, Lido Oracle failed to report validator balance updates causing the rewards distribution to occur every other day instead of daily. Neither user funds nor accrued rewards were at risk or otherwise affected during the incident. The issue was resolved on November 13, and the rebase cadence was restored to daily updates.

What happened?

Due to the extreme market conditions on the day of the incident, the protocol saw an unexpectedly high amount of execution-layer (EL) rewards. As a result, Oracle was unable to finalize reports because of an outdated security threshold for rewards. When the rewards exceeded this threshold, Oracle’s security check was triggered. Suspicious of the unusually high daily yield, the protocol put distribution on hold. Due to the intricacies of the check, balance updates were still delivered every other day. With the Lido DAO’s approval, the team resolved the issue by raising the security threshold with plans for a complete overhaul in the future.

Why did it happen?

Before the Merge, the only source of earnings was the consensus layer (CL), and the maximum APR across Ethereum was 10%. As such, Lido set up a security cap on the daily positive rebase at 10/365%.

With the Merge hard fork, the protocol started generating additional rewards from the execution layer through priority gas fees and MEV and accumulating them on a special vault. These rewards are then re-staked with a daily limit of 0.02% of the total stETH supply (around 940 ETH at the time of the incident) to prevent sandwich attacks. Thus, EL rewards provide an additional APR of up to 7.3%. This made it possible for the protocol to exceed the 10% APR threshold and fail the security check—precisely what happened on November 9, a particularly profitable day for the protocol with the EL reward vault exceeding the 940 ETH limit by a large margin.

The rebase still occurred every other day because the rewards spread over the previous two days. With the EL APR halved, the overall APR was able to slip under the 10% security threshold, and the user balances were updated.

How did we fix it?

In the few days following the incident, the team launched an Aragon vote to raise the security threshold to 17.5%, i.e., 10% plus 7.3% for CL and EL APR, respectively, as well as an additional 0.2% padding. The Lido DAO approved the change, and the threshold was updated.

We apologize for any inconvenience caused by the missed updates. We are committed to preventing similar incidents in the future and ensuring timely reward distribution to users. We have thoroughly reviewed our processes and procedures and will be giving double attention to harmonizing security checks in the upcoming Oracle overhaul.

Helpful links

• Merge-ready protocol service pack
• Increasing max APR sanity check for Oracle Lido report
• Vote 143 page
• About beacon-chain oracles
• Execution-layer rewards vault