The Solana network encountered a major disruption on February 6th at 10:22 am UTC, resulting in a block production halt that lasted over five hours before the network was restarted by validators. This is not the first time Solana has experienced such an issue. Since January 2022, there have been approximately six significant outages and 15 partial or major outage days on the Solana network.
These incidents varied in severity, ranging from partial to total network outages. The causes of these outages included software vulnerabilities, denial-of-service attacks, and resource exhaustion attacks on the network.
The most recent outage was attributed to a bug that caused an infinite loop error, resulting in validators getting stuck on a specific block. An infinite loop error occurs when a program gets trapped in an endless cycle. In the case of Solana, this prevented validators from verifying transactions as they were unable to process the same block.
A postmortem report on the recent outage was released by Anza, a Solana-focused software development firm, on February 9th. The report detailed the cause of the outage and how validators were able to restart the network. According to the report, the bug occurred in Solana’s Just-in-Time (JIT) compilation cache, which compiles all programs before executing a transaction.
Previously, the JIT cache used a program called “ExecutorsCache.” However, with the v1.16 release, this was replaced by a new implementation called “LoadedPrograms.” The new loader program encountered a legacy loader issue, causing the JIT cache to enter a loop with older programs and repeatedly recompile them instead of efficiently executing their instructions. This glitch resulted in a network stall, preventing transactions from being processed.
Austin Federa, head of strategy at the Solana Foundation, explained that there was a process in place to replace the old loader system with a new one, but it was scheduled to be disabled with an updated version.
Solana engineers quickly identified and fixed the problem, updated the system’s processes, and addressed the issue with older programs to prevent future recompilation cycles. This solution involved reducing the deployment of problematic legacy loaders to ensure accurate and efficient access to all program instructions.
Some critics argue that a lack of client diversity is a significant factor contributing to Solana’s outages, as it leads to potential central points of failure. However, Federa argues that Solana has a more accurate architecture compared to other networks. He explained that only Ethereum and Solana have independent validator clients, and currently, Solana has one major validator client built on one functional codebase.
Despite its outages, Solana has made significant progress in the decentralized finance (DeFi) ecosystem since its launch in March 2020. It is currently the fifth-largest DeFi chain in terms of total value locked, with over $1.7 billion allocated to different on-chain protocols. The native Solana (SOL) token also has a market valuation of around $49 billion, making it the fifth-largest cryptocurrency.
Although Solana is still referred to as being in mainnet beta, it has been open to everyone, with numerous decentralized applications and a vibrant nonfungible token (NFT) ecosystem. The network processes more transactions than most other chains, despite ongoing code changes.
The beta status has attracted criticism, especially in light of the network’s outages. However, Federa argues that most blockchain systems are currently in beta, and only Bitcoin and Ethereum can be considered complete mainnets.
While Solana’s downtime has been seen as a hurdle to its goal of competing with Ethereum, developers have been working to address the bugs that caused past outages to ensure the network’s stability in the future. Despite its history of downtime, Solana’s high speed and low cost continue to attract new projects to the platform.