Optimism outage unrelated to scheduled maintenance

The Optimism Mainnet blockchain experienced a series of outages and instabilities on Feb. 15, 2024, beginning around 6:00 am ET. The team started investigating reports of an “unsafe head stall,” meaning the sequencer stopped producing blocks.

Shortly after at 6:10 am ET, developers identified the cause of the issue and applied a fix that briefly restored operations.

However, the resolution was short-lived; by 6:26 am ET, reports of additional instability called into question the efficacy of the patch, according to the network’s status page.

The Optimism sequencer remains centralized, meaning OP Labs is responsible for identifying and troubleshooting problems.

After finding the suspected root cause, the network resumed around 8:00 am ET when the status page noted the incident was unrelated to the planned pause of OP Mainnet withdrawals — a scheduled maintenance window began at 9:01 am ET and ended one hour later.

Loading Tweet..

Blockworks has contacted Optimism for details.

Arbitrum, the largest Ethereum rollup by total value locked (TVL), suffered a similar outage in December 2023.

Read more: Arbitrum suffers ‘partial outage’ amid traffic influx

Although the team stated that no action was required from users or node operators following the resolution, community members in Discord dispute that.

“I can also confirm that a full node restart of op-node and op-geth was required to get unstuck,” wrote one member hazim-j.eth, co-founder of Stackup, who works on account abstraction tooling.

“If the status page had advised a restart we might have been able to get back online a lot sooner,” he said.

During the outage, node operators observed single-block reorganizations — or reorg — of the unsafe chain.

A single-block reorg, not uncommon in blockchain operations, occurs when the chain diverges into two paths due to discrepancies in block validation and then reconverges on a single, agreed-upon path.

In the case of Optimism, an “unsafe” head reorg refers to blocks produced by the sequencer, but not yet posted in batches to Ethereum mainnet, according to pseudonymous developer support engineer at OP Labs known as Soyboy Vegan.

According to Optimism’s documentation, “it is still possible for [unsafe transactions] to be excluded from the final blockchain if the Sequencer fails to publish the block to Ethereum in a timely manner.”

The planned withdrawal pause scheduled for 9:00 to 10:00 am ET was intended to “test Optimism’s incident response capabilities,” according to an OP Labs X post.

Updated on Feb. 15, 2024, at 11:05 am ET, with additional context.

Get the news in your inbox. Explore Blockworks newsletters:

Blockworks Daily: The newsletter that helps thousands of investors understand crypto and the markets, by Byron Gilliam.
Empire: Start your morning with the top news and analysis to inform your day in crypto.
Forward Guidance: Reporting and analysis on the growing intersection of crypto and macroeconomics, policy and finance.
0xResearch: Alpha directly in your inbox. Market highlights, data, degen trade ideas, governance updates, token performance and more.
Lightspeed: Built for Solana investors, developers and community members. The latest from one of crypto’s hottest networks.
The Drop: For crypto collectors and traders, covering apps, games, memes and more.
Supply Shock: Tracking Bitcoin’s rise from internet plaything worth less than a penny to global phenomenon disrupting money as we know it.