Computing Directly in Memory: Why In-Memory Computing Could Speed Up AI and Cut Energy Use
Artificial intelligence is usually associated with ever more powerful processors, but the processor is only part of the performance problem. Modern AI systems constantly move enormous quantities of information between memory and computing units, and that movement consumes time and electricity even before useful calculations are completed. As models grow, particularly large language models, this memory traffic becomes increasingly important. In-Memory Computing, often shortened to IMC, approaches the problem from a different direction: instead of repeatedly carrying data from memory to a separate processor, selected calculations are performed where the data is stored. Research published through 2026 shows that this idea is moving beyond small laboratory demonstrations towards more complete AI accelerators. It could improve inference speed and energy efficiency in data centres, mobile devices, industrial equipment and other systems, although important limits concerning precision, manufacturing, software and scalability mean that conventional CPUs and GPUs are not about to disappear.
Why Moving Data Has Become One of AI’s Biggest Hardware Problems
Most conventional computers separate memory from processing. Information is stored in memory, transferred to a CPU, GPU or another accelerator when a calculation is required, and the result may then be sent back to memory. This arrangement has worked extremely well for decades because processors could become faster while memory systems also improved. The difficulty is that moving information across a chip or between separate components costs energy. It also takes time. A processor may theoretically be capable of performing a vast number of calculations every second, yet it cannot use that capability efficiently if it is regularly waiting for the data required for the next operation.
AI makes this weakness more visible because neural networks repeatedly perform calculations using large collections of numerical values known as weights. A model may need to read the same weights many times while processing images, speech or text. Large language models increase the pressure further because they can contain billions of parameters and require substantial memory capacity and bandwidth during inference. IBM Research described this issue particularly clearly in a 2026 study of analogue in-memory computing for language-model inference: for modern LLMs, memory capacity and data movement can become more important restrictions than the availability of raw arithmetic capability itself. Faster processors alone therefore do not solve every performance problem.
The energy cost matters just as much as speed. Every transfer may be small in isolation, but AI accelerators carry out an enormous number of transfers. At data-centre scale, the accumulated electricity demand affects operating costs, cooling requirements and the amount of computing that can be installed within a given power budget. On battery-powered devices, the same issue appears in a different form: unnecessary memory traffic can shorten operating time and create extra heat. Reducing data movement is therefore attractive even when the underlying processor is already efficient. This is the problem that In-Memory Computing is designed to address.
What Changes When Calculations Take Place Inside Memory
In-Memory Computing changes the usual journey of data. Rather than reading every stored value, sending it to a separate processor and then carrying the result elsewhere, a memory array can participate directly in selected calculations. The approach is particularly useful for matrix and vector operations, which appear constantly in neural-network workloads. A simplified way to think about it is that conventional hardware repeatedly takes numbers out of storage to calculate with them, whereas an IMC design can use the physical arrangement of stored values to help perform the calculation in the same location. Less travelling means fewer transfers, lower latency and potentially less electricity used for each completed AI task.
There is no single type of In-Memory Computing hardware. Some designs use familiar static random-access memory, or SRAM, and perform digital calculations close to or within its memory cells. Others rely on non-volatile technologies such as phase-change memory, resistive RAM, flash memory or magnetic RAM. Analogue designs can represent values using physical properties such as electrical conductance, allowing many multiplication and addition operations to occur in parallel. Digital designs generally sacrifice some of the extreme efficiency potential in exchange for more predictable numerical behaviour. Researchers are also combining memory types so that different parts of an AI workload can use the method that suits them best.
This distinction matters because IMC should not be understood as a complete computer that happens to be made from memory. Only particular operations benefit strongly from being moved into memory. Other work still requires conventional digital logic, including control tasks, data preparation, activation functions, communication and operations that need higher numerical precision. Successful designs therefore contain a mixture of memory-based computation and ordinary processing circuitry. The practical objective is not to eliminate processors but to stop using expensive processor-memory transfers for calculations that can be carried out more efficiently elsewhere.
Where In-Memory Computing Can Make AI Faster and More Efficient
AI inference is one of the strongest candidates for In-Memory Computing because a trained model repeatedly applies stored weights to new input data. If those weights can remain inside a memory array that also performs the necessary multiplication and accumulation, the system avoids fetching the same information over and over again. This can reduce latency while allowing many operations to run in parallel. The potential benefit is especially significant when an AI workload is limited by memory bandwidth rather than by the mathematical capability of the processor. Image recognition, speech processing, sensor analysis and parts of language-model inference all contain operations that fit this pattern.
Real hardware demonstrates that the idea is more than a theoretical energy-saving technique. An IBM research chip reported in Nature used 35 million phase-change memory devices arranged across 34 analogue computing tiles and achieved up to 12.4 trillion operations per second per watt in its measured configuration. The researchers demonstrated speech-recognition and transcription workloads, including a network with tens of millions of weights distributed across several chips. More recent IBM work has continued towards 64-core designs and architectures intended to support larger neural networks. These experiments do not mean that an analogue memory chip can simply replace a modern GPU, but they show that substantial neural-network workloads can be executed using the approach.
Research in 2026 has also broadened the range of memory technologies involved. One published 55-nanometre system-on-chip used eight megabits of embedded NOR flash for compute-in-memory operations aimed at edge AI. It delivered 0.16 TOPS while consuming 21.4 milliwatts, corresponding to 7.48 TOPS per watt, and the researchers reported at least a fourfold reduction in end-to-end inference latency compared with their CPU baselines. At ISSCC 2026, another digital compute-in-memory accelerator fabricated using Intel 18A technology demonstrated peak measured efficiency of 147 TOPS per watt under a specific low-voltage, 25% input-activity test condition. Such figures show the rapid pace of development, although they should never be treated as direct comparisons between unrelated chips.
Why Neural Networks Are Particularly Well Suited to the Approach
A major reason IMC works well for AI is the repetitive structure of neural-network mathematics. Much of the processing consists of multiplying many input values by stored weights and then adding the results together. Conventional accelerators already perform these operations in parallel, but they still need a constant supply of weights and activations from memory. A compute-in-memory array can keep large groups of weights physically close to the calculation. That allows hundreds or thousands of operations to be initiated together instead of treating every multiplication as a separate trip between memory and processing logic.
Large language models make this advantage particularly interesting, but also reveal why the technology is not a simple solution to every AI bottleneck. During generation, model weights must be accessed repeatedly as each new token is produced. This makes some stages of LLM inference heavily dependent on memory bandwidth, so storing weights inside computational memory could reduce a significant amount of traffic. However, transformers also use attention mechanisms, intermediate activations and key-value caches whose size can increase as a conversation or document becomes longer. Those data are more dynamic than fixed model weights. A practical LLM accelerator therefore needs conventional high-speed memory and digital processing alongside any compute-in-memory components.
The clearest early benefits may appear in edge AI, where power and cooling budgets are much smaller than in a data centre. A security camera performing local object recognition, an industrial sensor detecting abnormal machinery behaviour or a wearable device processing signals does not necessarily need the flexibility of a large GPU. It needs to perform a defined set of AI operations quickly while consuming as little energy as possible. Non-volatile memory can offer an additional advantage because model weights remain stored when power is removed. A device can therefore wake, process information and return to a low-power state without repeatedly loading the entire model from external storage.
What Still Prevents In-Memory Computing from Replacing Today’s AI Hardware
The largest technical challenge for analogue In-Memory Computing is that real memory devices are not mathematically perfect. Electrical noise, manufacturing differences between cells, changes over time and temperature effects can cause the physical value stored in a device to deviate slightly from the intended number. Phase-change memory, for example, can experience resistance drift, while other resistive memories have their own sources of variability. Neural networks can tolerate a certain amount of approximation, but errors still need to remain within acceptable limits. A 2026 review in npj Unconventional Computing highlighted techniques such as mixed-precision processing and error-correction methods that can improve accuracy, but these measures add circuitry or additional calculations.
There is also an energy cost outside the memory array itself. An analogue calculation is useful to ordinary software only after its inputs and outputs have been represented in a form the rest of the chip can understand. Digital-to-analogue and analogue-to-digital converters can consume meaningful power and silicon area. Communication between several memory arrays can also reduce the advantage obtained inside an individual array. This is why impressive laboratory measurements for a small compute-in-memory macro cannot automatically be compared with the efficiency of a complete GPU or AI accelerator. Precision, workload, memory capacity, input activity, chip area and the parts of the system included in the measurement all influence reported TOPS-per-watt figures.
Digital compute-in-memory designs avoid some analogue accuracy problems by keeping calculations in conventional binary form. Research published in Nature Electronics in 2025 demonstrated a spintronic digital compute-in-memory macro based on magnetic memory that supported several numerical precision levels and reached software-equivalent inference accuracy in tested workloads. Depending on precision and operating conditions, the reported efficiency ranged from 7.02 to 112.3 TOPS per watt. The wide range itself illustrates an important point: energy efficiency is not a fixed property of an architecture. Increasing precision, handling different workloads or moving from a small memory macro to a complete usable system changes the result considerably.
What a Realistic Adoption Path Looks Like After 2026
The most realistic future is therefore heterogeneous computing, where several kinds of hardware work together rather than one design replacing everything else. CPUs are good at general control and irregular tasks. GPUs and dedicated digital accelerators provide programmability, mature software support and high performance across many AI workloads. Compute-in-memory blocks can be added for the operations where moving weights back and forth is particularly wasteful. Research processors already follow this pattern by combining analogue or digital memory arrays with conventional logic. A system can then direct each part of a neural network towards the hardware that handles it efficiently.
Edge devices may adopt these ideas differently from large data centres. In a small device, the priority can be milliwatt-level power consumption, instant response and the ability to keep a compact model stored locally. The 2026 NOR-flash demonstration shows how compute-in-memory can be integrated with supporting circuitry and software for this type of workload. Data centres have a harder problem because the largest models require enormous memory capacity, sophisticated networking and frequent software changes. Here, IMC is more likely to appear initially as a specialised accelerator or memory-centric component working beside established AI processors rather than as a standalone replacement for existing server hardware.
By 2026, In-Memory Computing has advanced far enough to demonstrate a credible answer to one of AI hardware’s fundamental inefficiencies: moving enormous quantities of data merely so that calculations can take place somewhere else. Research chips using phase-change memory, SRAM, magnetic memory, memristors and flash have shown that useful neural-network operations can be performed closer to stored data with substantial gains in energy efficiency under the right conditions. Yet the remaining issues are equally important: accuracy, converters, memory capacity, software compatibility, manufacturing consistency and system-level communication all affect real performance. The strongest case for IMC is therefore not that memory will replace the processor, but that future AI hardware can waste far less energy by deciding more carefully where each calculation should happen.
