Cardano recovers from an outage on 50% relay and block-producing nodes

Published at: Jan. 23, 2023

Cardano stake pool operators (SPO) and users have been reporting at least half of the nodes for the network briefly went offline over the weekend.

According to a Jan. 22 post shared on the Telegram SPO for Input Output Global, the engineering and research in fintech behind the Cardano blockchain, an anomaly caused 50% of Cardano nodes to disconnect and restart.

Last night during the anomaly on the #Cardano network, the entire network did not go down. There was a brief period of degradation. Most nodes impacted had gracefully recovered. No network restart was required. pic.twitter.com/FupQXk2otV

— Rick McCracken DIGI (@RichardMcCrackn) January 22, 2023

"This appears to have been triggered by a transient anomaly causing two reactions in the node, some disconnected from a peer, others threw an exception and restarted," the post read while explaining the sudden disruption.

Despite a momentary degradation, the Cardano network recovered without external intervention. As explained in the post, "such transient issues" were considered in the node design and consensus and "the systems behaved exactly as expected."

During the anomaly, which happened between block 8300569 and 8300570, block production reportedly kept going but was slowed for a few minutes and the "impact was low, akin to the delays that occur during normal operations."

"Most nodes automatically recovered, depending on the SPO of choice."

At the time of writing, the root cause of the anomaly and resulting node disconnections and restarts are still under investigation.

“We’re now investigating the root cause for this anomalous behavior and implementing further logging measures alongside our regular monitoring procedures,” stated the official announcement.

Tom Stokes, the co-founder of Node Shark and a Cardano SPO, reported in a Jan. 22 post that more than half of the listed nodes were affected.

He also shared a chart showing where the network sync fell from 100% to slightly above 40% for over 300 reporting nodes.

A few hours ago over half of all #Cardano nodes went offline. This is why decentralization matters. pic.twitter.com/NXDVyKy8ep

— Tom Stokes (@eUTxO_pro) January 22, 2023

According to Stokes chart, the network sync recovered back to around 87% after the drop but did not immediately return to its previous level at 100%.

Another SPO reported similar issues to Stoke in a Jan. 22 post, but declared "Some SPOs saw no impact."

"Others had relays and BPs restart. SPOs, Devs, and IOG are in Discord debugging atm. No root cause yet," they said.