BloxStaking - Missed Block Proposals

BloxStaking – Missed Block Proposals
Share this post
Share on facebook
Share on linkedin
Share on twitter
Share on telegram
Share on whatsapp
Share on email

Over the past weekend a portion of our users experienced a temporary issue with un performed duties; more significantly, missed block proposals.

Incident Summary:

Blox upgrade (at epoch 44559) included a release of a new ‘validator center’ version. Validator center is the component in Blox backend responsible for processing validator duties.

The version contained optimization to duty processing as part of an ongoing effort to improve and stabilize our effectiveness rate (current avg.98-99%).

Impact:

53 missed block proposals across over 1400 validators Blox manages.

block proposers experienced an opportunity loss on proposal rewards.

On Average of 0.0046 ETH for proposed reward.

Root Causes:

Validator center fetches duties from the beacon chain for a given epoch and stores them in cache. It is in charge of mapping public key activities, slot, and duty type.

In the last release, a major refactor introduced a bug where proposal duties were overridden with the attestation slot duty, causing them not to be executed.

Resolution:

Correctly cache duties by their type fixed the problem.

Lessons Learned

What went well

Our extensive logging system, reflecting in detail the whole lifecycle of processing & execution of epoch duties helped us to detect the issue and deploy a fix within less than an hour.

What went wrong

We monitor and validate attestations and not proposals, there was no alert leading to over 24 hours without knowing we missed proposals – we are currently developing a fix.