Carlos Sibrian
← Todas las entradasCiencia

The memory wall: why AI chips spend so much time waiting

Part 1 of 3. Before asking what fungi could teach computers, we need the problem they would be solving: the von Neumann bottleneck, the memory wall and the new shape of Moore’s law.

Por Carlos Sibrian · 23 oct 2026 · 10 min de lectura

Serie · Fungi, brains and machines

Esta entrada está en inglés.

An engraved paper cut-out of an antique hourglass on a black background, with its glass cut away. The upper bulb is full of glowing amber grains, a thin trickle falls through the neck, and an engraved chip lies at the bottom, catching the few grains that arrive. Golden mycelium threads climb both sides of the hourglass.
Memory above, the chip below. A collage made for this series: an engraved hourglass whose upper bulb holds a vast memory of data, which reaches the processor waiting below only through the narrow neck. Outside the glass, a mycelium grown by an algorithm climbs from one bulb to the other without passing through the neck.
In shortArtificial intelligence runs on machines that keep memory and computation apart, so much of their energy goes into moving data rather than computing with it. At the same time, Moore’s law still adds transistors but no longer makes each one faster or more efficient for free. The next gains have to come from how computers are organized. This first part of a three-part series lays out that problem; the next two ask what the networks of the brain and of fungi could contribute.

Artificial intelligence is running into a problem that has little to do with algorithms. Models grow faster than hardware can feed them data, and the electricity used by data centers keeps rising. The International Energy Agency estimates that data centers used about 415 TWh in 2024, around 1.5% of the world’s electricity, and projects about 945 TWh by 2030 in its base case [1].

This series asks a strange question with a serious purpose: what could the networks of fungi, which grow under every forest floor with no brain and no plan, teach the people who design AI hardware? Before any mushroom appears, we need to understand the problem it would be solving, and that is what this first part is about. Part 2 compares the brain, the mycelium and the chip. Part 3 pushes the idea to its limit: human neurons wired the way fungi wire a forest.

  • In today’s chips, fetching a number from memory costs hundreds to thousands of times more energy than computing with it. AI workloads suffer from this more than most.
  • Moore’s law still adds transistors, but since the mid-2000s it no longer brings automatic gains in speed or efficiency. The big gains now come from architecture.
  • The brain shows another way to organize computing: memory and processing in the same place, and activity only where it is needed.
A close-up illustration of a chip package on a dark background: a large central processor flanked by eight stacked memory chips and joined to them by fine gold traces, with points of light crowding the traces where they reach the processor.
Data waiting at the door. An illustration generated with AI for this series and retouched with open-source tools, not a photograph of a real product. The processor sits beside stacks of high-bandwidth memory, yet every number still has to travel between them; the points of light, added afterwards, stand for data queuing at the processor’s edge.

1 · The von Neumann bottleneck: moving data costs more than computing

In 1945 John von Neumann described a computer split into two parts: a unit that calculates and a memory that holds both instructions and data [2]. Almost every computer since has followed that plan. In 1977, accepting the Turing Award, John Backus named its central weakness. The processor and the memory talk through a narrow channel, and every piece of data must pass through it on the way in and on the way out. “I propose to call this tube the von Neumann bottleneck,” he wrote [3].

For decades the bottleneck was tolerable. Today it dominates the cost of computing. In a 45-nanometer chip, adding two 32-bit integers costs about 0.1 picojoules, while reading 64 bits from main memory costs 1,300 to 2,600 picojoules [4]. Fetching the data takes thousands of times more energy than using it.

Horizontal bar chart on a log scale. Computing: adding two integers 0.1 picojoules, adding two decimals 0.9, multiplying two decimals 3.7. Reading memory: an 8 KB cache 10, a 1 MB cache 100, main memory 1,300 to 2,600 picojoules.
The cost is in the commute. Energy per operation in a 45 nm chip. Source: M. Horowitz, IEEE ISSCC 2014.

The gap keeps widening. Amir Gholami and colleagues at Berkeley looked at twenty years of server hardware. Peak computing power grew 3.0 times every two years, the bandwidth of DRAM memory only 1.6 times, and the links between chips 1.4 times [5]. Over two decades that adds up to roughly 60,000 times more compute against about 100 times more memory bandwidth. William Wulf and Sally McKee called this mismatch the memory wall back in 1995 [6].

Line chart on a log scale over 20 years. Peak compute reaches about 60,000 times its starting value, memory bandwidth about 100 times and chip-to-chip links about 30 times, leaving a gap of about 600 times.
Compute has outrun its supply line. Growth of peak compute, memory bandwidth and chip-to-chip bandwidth over 20 years, modeled from the average rates in A. Gholami et al., IEEE Micro 2024.

Large language models feel this more than most software. When a model writes text, every new token requires reading all of its weights from memory, and each weight is used for only a couple of arithmetic operations. That is roughly one operation per byte read. An NVIDIA H100 offers about 989 trillion dense 16-bit operations per second and 3.35 TB/s of memory bandwidth [7], so it needs around 295 operations per byte to keep its arithmetic units busy (my arithmetic, a standard “roofline” estimate). Serving many users at once raises the ratio, but the basic picture holds: an expensive chip that spends much of its time waiting for data.

Phones and laptops are not exempt. A 2018 study of Google workloads on consumer devices found that, on average, 62.7% of total system energy went into moving data between memory and the processor [8].

2 · Moore’s law did not die. It changed shape

In 1965 Gordon Moore observed that the number of components on a chip was doubling every year [9]. In 1975 he predicted that the pace would slow to a doubling every two years [10]. The forecast held up remarkably well. Intel’s 4004 of 1971 had 2,300 transistors. NVIDIA’s B200, announced in 2024, has 208 billion, split across two pieces of silicon [11]. That is about 26 doublings in 53 years, close to one every two years.

Two charts. Left, transistors per chip on a log scale, from 2,300 in the Intel 4004 to 208 billion in the NVIDIA B200, with a dashed line in the mid-2000s marking the end of Dennard scaling. Right, bars of yearly single-thread speedup: 22% before 1986, 52% from 1986 to 2003, 23% to 2011, 12% to 2015 and 3.5% since.
The transistors kept coming; the free speed did not. Transistors per chip from 1971 to 2024 and the yearly growth of single-thread performance. Sources: Intel and NVIDIA; Hennessy and Patterson, Computer Architecture, 6th ed.

What ended was a side effect. For thirty years, shrinking transistors also made them faster and more efficient without heating the chip more. This effect, called Dennard scaling after a 1974 paper by Robert Dennard and colleagues, broke down in the mid-2000s [12]. Clock speeds stalled at a few gigahertz, and the industry turned to more cores, more parallelism and specialized chips.

The performance of a single processor core tells the story. It grew about 52% a year between 1986 and 2003 and has grown about 3.5% a year since 2015 [13]. Meanwhile the cost per transistor stopped falling at around the 28-nanometer node, according to an industry slide presented by Google in 2023 [14]. Power per chip keeps climbing: an H100 can draw 700 W and a B200 1,000 W [7, 11].

Four small charts. Data centers used 415 TWh in 2024 and are projected to use 945 TWh in 2030. A CPU rack draws about 12 kW, an air-cooled H100 rack about 40 kW and a GB200 NVL72 rack 120 to 132 kW. A grid of 419 dots shows the unexpected interruptions in 54 days of Llama 3 training, 78% of them hardware-related. The memory cache of one chat with a 70-billion-parameter model grows from 2.7 GB at 8 thousand tokens to 43 GB at 128 thousand, against 80 GB in one H100.
The bill has already arrived. Four pressures on AI hardware: world data-center electricity (IEA), rack power (SemiAnalysis, Supermicro), interruptions during Llama 3 training (Meta) and the memory cache of one long conversation (author’s calculation).

The figure above puts numbers on the pressure. Electricity is one limit. Heat is another: a rack of the newest AI servers draws 120 to 132 kW, ten times a conventional rack, which makes liquid cooling mandatory [15]. Reliability is a third. During a 54-day window of training Meta’s Llama 3 405B model on more than 16,000 GPUs, the job was interrupted unexpectedly 419 times, about eight times a day, and roughly 78% of those interruptions were traced to hardware [16]. Memory is the fourth. While it writes, a model keeps a “KV cache”, a working memory of every token in the conversation so far. For a 70-billion-parameter model and a single 128,000-token conversation, that cache takes about 43 GB, more than half of an H100’s memory, on top of roughly 140 GB for the model’s weights, which already have to be spread over at least two GPUs (my calculation from the model’s published architecture [17]).

In 2020, Charles Leiserson and colleagues summed up the consequence in Science. When the silicon stops handing out improvements, they have to come from “the top”: software, algorithms and hardware architecture [18].

3 · The brain’s answer: 86 billion neurons on about 20 watts

There is one machine that already runs intelligence on a tiny power budget. The human brain contains about 86 billion neurons [19] and, in the neocortex alone, on the order of 150 trillion synapses [20]. It accounts for about 2% of body weight and about 20% of the body’s oxygen consumption at rest [21], roughly 20 W [22]. A single H100 draws as much power as about 35 brains. Training GPT-3 was estimated by Google researchers at 1,287 MWh [23]; a brain running at 20 W uses about 3.5 MWh in twenty years. The comparison is only an illustration, but it conveys the distance.

Neurons are slow. An electrical spike lasts about a millisecond and travels from about half a meter to more than a hundred meters per second, depending on whether the fiber is insulated with myelin [24]. A chip completes billions of cycles per second. The brain’s advantage comes from how it is organized:

  • Memory and computation in the same place. A synapse stores the strength of a connection and also processes the signal that crosses it. There is no memory bus.
  • Sparse, event-driven activity. The energy cost of a spike is so high that probably fewer than 1% of cortical neurons can be substantially active at the same time [25]. Whatever is quiet costs almost nothing.
  • Local learning. A synapse strengthens or weakens depending on the relative timing of the two neurons it connects, within a window of about 20 milliseconds [26]. Artificial networks learn by backpropagation, a global error signal that the brain does not seem to use in the same form [27].

Engineers already copy these ideas. IBM’s TrueNorth chip (2014) packed a million artificial spiking neurons into a chip that drew 63 milliwatts in a real-time video task [28]. Intel’s Hala Point (2024) combines 1,152 Loihi 2 chips with up to 1.15 billion neurons and a maximum power of 2,600 W [29]. IBM’s NorthPole (2023) places memory next to every core and, on an image-recognition benchmark, processed about 25 times more frames per joule than a GPU built on a comparable 12-nanometer process [30]. An IBM analog chip performs matrix multiplications inside the memory itself, with an energy efficiency that IBM estimates at up to 14 times that of comparable digital hardware [31].

Two caveats. An artificial neuron is not a biological one, so counting them side by side is only an illustration. And neuromorphic chips win on specific tasks: on large problems, spiking networks still trail conventional deep networks and are harder to train.

4 · Two claims to retire

  • “Moore’s law is dead.” Transistor counts keep growing. What ended were the automatic gains in speed and efficiency that used to come with each new generation [13].
  • “A brain equals an exaflop,” or “neurons are a million times more efficient.” Spikes and floating-point operations are not comparable units. There is not even a reliable figure for how much a brain computes: Joe Carlsmith’s careful review judges anything from about 10¹³ to 10¹⁷ operations per second plausible for matching the brain’s performance, with 10¹⁵ more likely than not to be enough [32].

What comes next

If the next improvement in AI hardware comes from organization rather than from smaller transistors, it is worth studying every network that has solved the problem of moving resources cheaply. The brain is the obvious one. The other grows under our feet. It has no neurons at all, yet it builds transport networks that are cheap, robust and able to repair themselves.

Two diagrams. Left, a processor with four cores and a memory block connected by one narrow channel where data waits in line. Right, a network of nodes, each half memory and half compute, joined by short links; the busiest routes are thick golden cords, idle links are thin, and teal dots mark signals between neighbors.
Two ways to organize computing. Left, the von Neumann architecture with processor and memory joined by one narrow channel. Right, a conceptual mycelium-like network of nodes that store and compute; it is not an existing chip.

That network is the mycelium of fungi, and it is the subject of Part 2: Thinking like a mycelium.

Notes on methods

The cover is a collage made by computer for this series, not a photograph: an hourglass and a chip modeled in 3D with Blender and engraved by a program, and a mycelium network grown by an algorithm. The charts were drawn from the sources cited in each caption. Two figures are my own arithmetic: the “roofline” estimate for the H100 (989 dense TFLOPS divided by 3.35 TB/s) and the cache size of Llama 3.1 70B (2 × 80 layers × 8 key-value heads × 128 dimensions × 2 bytes ≈ 0.33 MB per token). Figures supplied by manufacturers are presented as such.

References

  1. International Energy Agency. Energy and AI. April 2025. iea.org
  2. von Neumann, J. “First Draft of a Report on the EDVAC” (1945). Reprinted in IEEE Annals of the History of Computing 15(4), 1993. DOI
  3. Backus, J. “Can Programming Be Liberated from the von Neumann Style? A Functional Style and Its Algebra of Programs.” Communications of the ACM 21(8): 613–641, 1978. DOI
  4. Horowitz, M. “Computing’s energy problem (and what we can do about it).” IEEE International Solid-State Circuits Conference (ISSCC), 2014. DOI
  5. Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W. and Keutzer, K. “AI and Memory Wall.” IEEE Micro, 2024. DOI
  6. Wulf, W. A. and McKee, S. A. “Hitting the Memory Wall: Implications of the Obvious.” ACM SIGARCH Computer Architecture News 23(1): 20–24, 1995. DOI
  7. NVIDIA. H100 Tensor Core GPU, datasheet. nvidia.com
  8. Boroumand, A. et al. “Google Workloads for Consumer Devices: Mitigating Data Movement Bottlenecks.” ASPLOS, 2018. DOI
  9. Moore, G. E. “Cramming more components onto integrated circuits.” Electronics 38(8), 1965. Reprinted in Proceedings of the IEEE 86(1), 1998. DOI
  10. Moore, G. E. “Progress in digital integrated electronics.” IEEE International Electron Devices Meeting, 1975. Context at the Computer History Museum. computerhistory.org
  11. NVIDIA. “NVIDIA Blackwell Platform Arrives to Power a New Era of Computing.” Press release, 18 March 2024. nvidianews.nvidia.com
  12. Dennard, R. H. et al. “Design of ion-implanted MOSFET’s with very small physical dimensions.” IEEE Journal of Solid-State Circuits 9(5): 256–268, 1974. DOI
  13. Hennessy, J. L. and Patterson, D. A. Computer Architecture: A Quantitative Approach, 6th ed., 2019, fig. 1.1. See also “A New Golden Age for Computer Architecture,” Communications of the ACM 62(2), 2019. DOI
  14. Tom’s Hardware. “Chips aren’t getting cheaper: the cost per transistor stopped dropping a decade ago at 28nm.” December 2023, reporting a Google slide shown at IEDM 2023. tomshardware.com
  15. Supermicro. SRS-GB200-NVL72 rack specifications (about 132 kW); SemiAnalysis estimates about 120 kW for the same rack. supermicro.com
  16. Llama Team, AI at Meta. “The Llama 3 Herd of Models.” arXiv:2407.21783, 2024 (interruption counts in Table 5). arXiv
  17. Meta. Llama 3.1 model card, 2024. github.com
  18. Leiserson, C. E., Thompson, N. C., Emer, J. S. et al. “There’s plenty of room at the Top: What will drive computer performance after Moore’s law?” Science 368: eaam9744, 2020. DOI
  19. Azevedo, F. A. C. et al. “Equal numbers of neuronal and nonneuronal cells make the human brain an isometrically scaled-up primate brain.” Journal of Comparative Neurology 513(5): 532–541, 2009. DOI
  20. Pakkenberg, B. et al. “Aging and the human neocortex.” Experimental Gerontology 38: 95–99, 2003. DOI
  21. Raichle, M. E. and Gusnard, D. A. “Appraising the brain’s energy budget.” PNAS 99(16): 10237–10239, 2002. DOI
  22. Balasubramanian, V. “Brain power.” PNAS 118(32): e2107022118, 2021. DOI
  23. Patterson, D. et al. “Carbon Emissions and Large Neural Network Training.” arXiv:2104.10350, 2021. arXiv
  24. Purves, D. et al. Neuroscience, 2nd ed. “Increased Conduction Velocity as a Result of Myelination.” NCBI Bookshelf. ncbi.nlm.nih.gov
  25. Lennie, P. “The cost of cortical computation.” Current Biology 13(6): 493–497, 2003. DOI
  26. Bi, G. and Poo, M. “Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type.” Journal of Neuroscience 18(24): 10464–10472, 1998. DOI
  27. Lillicrap, T. P., Santoro, A., Marris, L., Akerman, C. J. and Hinton, G. “Backpropagation and the brain.” Nature Reviews Neuroscience 21: 335–346, 2020. DOI
  28. Merolla, P. A. et al. “A million spiking-neuron integrated circuit with a scalable communication network and interface.” Science 345(6197): 668–673, 2014. DOI
  29. Intel. “Intel Builds World’s Largest Neuromorphic System to Enable More Sustainable AI.” Press release, 17 April 2024. newsroom.intel.com
  30. Modha, D. S. et al. “Neural inference at the frontier of energy, space, and time.” Science 382(6668): 329–335, 2023. DOI
  31. Ambrogio, S. et al. “An analog-AI chip for energy-efficient speech recognition and transcription.” Nature 620: 768–775, 2023. DOI
  32. Carlsmith, J. How Much Computational Power Does It Take to Match the Human Brain? Open Philanthropy, 2020. openphilanthropy.org

Comentarios0