Polygon’s zkEVM rollup entered an emergency state on March 23 after the chain stopped processing blocks. Final week, Polygon launched an evaluation of the outage that took its zkEVM down.
Polygon reported late Saturday that its zkEVM sequencer had failed, attributing the trigger to a reorg of the Ethereum mainnet. The postmortem states that the synchronizer for Polygon zkEVM mishandled the reorg, and the chain began processing batches of transactions with incorrect timestamps.
A blockchain reorganization, or reorg, occurs when a community discards blocks from a earlier model and adopts a brand new chain of blocks because the correct historical past of transactions. This may happen when completely different elements of the community briefly disagree on current transaction order or validity. The community ultimately converges on a single model of the reality, selecting the chain of blocks with essentially the most computational work (or, in some consensus mechanisms, essentially the most stake backing it) as the proper sequence. Reorgs assist to make sure that the community stays safe and constant, although it may possibly result in short-term uncertainties in transaction histories.
Learn extra: Polygon zkEVM prover reaches Type 1 status
Polygon zkEVM’s sequencer, which assembles and broadcasts batches of transactions to the community nodes, performs a vital function within the ecosystem. It isn’t solely liable for ordering transactions but in addition assigns them timestamps, vital for sustaining a coherent historical past of transactions on the community. Nevertheless, through the incident, the sequencer didn’t detect this reorg on the Ethereum mainnet.
This led to the processing of transactions based mostly on outdated info, including an incorrect world exit route — Polygon zkEVMs cryptographic methodology to make sure Ethereum layer-1 and Polygon’s layer-2 settlement — to the subsequent transaction block. The resultant mismatch induced inaccuracies, disrupting the community’s potential to take care of a constant report of the community’s ledger.
In the course of the confusion, numerous transactions had been processed by the community as no-operation, with the community appearing “as in the event that they weren’t there,” the autopsy stated. Synchronizing these empty transactions was a part of what made the community’s downtime prolong so long as it did.
All-told, it took round 14 hours for the rollup to reopen for withdrawals, and round 4,000 transactions might have been affected by the outage, Polygon stated.
The outage solely affected Polygon zkEVM, which is in so-called mainnet beta and subsequently could have bugs and availability points, in line with Polygon’s assist web page. The Polygon PoS layer-1 and chains utilizing Polygon’s chain growth equipment (CDK) weren’t impacted.
Polygon’s zkEVM, brief for zero-knowledge Ethereum Digital Machine, is a layer-2 that makes use of zero-knowledge proofs to confirm transactions with the Polygon-operated sequencer earlier than settling batches of transactions on Ethereum. StarkWare and zkSync are constructing comparable merchandise. That is all in distinction to standard layer-2s like Arbitrum and Optimism, which perform as optimistic rollups and don’t use zero-knowledge proofs.
Learn extra: StarkWare’s zero-knowledge prover Stwo comes out of stealth
Shortly after the outage started, Polygon’s zkEVM went into an emergency state that halted the community and let an upgrade to the community’s prover and verifier be pushed by way of alongside a repair to maintain the sequencer from malfunctioning throughout a reorg.
Upon finishing this upgrade, the emergency state was deactivated, that means any additional updates to the system are topic to a 10-day timelock earlier than taking impact.
This occasion marked the primary time Polygon’s zkEVMs emergency state perform was used, and doing so required approval from the community’s Safety Council. When activated, this state halts sure operations, together with the updating of transactions and the functioning of the blockchain bridge, to guard customers’ property.
The council operates as a 6/8 multisig. Polygon says two of the council members are from Polygon Labs, however the identities of the Safety Council members aren’t publicly disclosed.
A 2023 weblog publish said the group is “made up of extremely respected members of the Ethereum neighborhood” and might be dismantled as soon as Polygon zkEVM is able to ditch its “coaching wheels.”
The outage and its remediation factors to lingering centralization points throughout Ethereum layer-2 networks, according to Jarrod Watts, a developer relations engineer at Polygon Labs:
“The worst case situation is an upgrade is carried out that steals person funds,” he said on X. “This type of malicious upgrade is one thing that’s attainable on nearly each [layer-2] at the moment.”
However Watts added, “over time, IMO the chance of malicious upgrades to steal person funds on [layer-2s] might be zero or very near zero!”
Polygon zkEVM has struggled to realize whole quantity locked (TVL) and utilization, and has but to profit from Ethereum’s Dencun hard fork. The chain is slated so as to add assist for the “blobspace” of EIP-4844 in a couple of month’s time, following its “Feijoa” upgrade.
Don’t miss the subsequent large story – be a part of our free daily newsletter.