Over the past weekend a portion of our users experienced a temporary issue with un performed duties; more significantly, missed block proposals.
Incident Summary:
Blox upgrade (at epoch 44559) included a release of a new ‘validator center’ version. Validator center is the component in Blox backend responsible for processing validator duties.
The version contained optimization to duty processing as part of an ongoing effort to improve and stabilize our effectiveness rate (current avg.98-99%).
Impact:
53 missed block proposals across over 1400 validators Blox manages.
block proposers experienced an opportunity loss on proposal rewards.
On Average of 0.0046 ETH for proposed reward.
Root Causes:
Validator center fetches duties from the beacon chain for a given epoch and stores them in cache. It is in charge of mapping public key activities, slot, and duty type.
In the last release, a major refactor introduced a bug where proposal duties were overridden with the attestation slot duty, causing them not to be executed.
Resolution:
Correctly cache duties by their type fixed the problem.
Lessons Learned
What went well
Our extensive logging system, reflecting in detail the whole lifecycle of processing & execution of epoch duties helped us to detect the issue and deploy a fix within less than an hour.
What went wrong
We monitor and validate attestations and not proposals, there was no alert leading to over 24 hours without knowing we missed proposals – we are currently developing a fix.
