← Back to feed

Sakana AI published PC-ALM, an augmented Lagrangian variant of predictive coding that eliminates backpropagation’s global synchronization requirement and matches backprop performance on 1,000-layer networks while using only layer-local updates.

The paper, released at pub.sakana.ai/pc-alm, addresses a fundamental constraint in how modern neural networks are trained. Backpropagation is a global algorithm: a complete forward pass must finish before any backward pass begins, and the backward pass must complete before weights are updated. Every layer waits on every other layer. That sequential dependency caps the depth at which networks can be practically trained and rules out asynchronous or distributed layer-level optimization.

The Predictive Coding Alternative

Predictive coding reframes learning as a local process. Instead of passing error signals from output back through the entire network in one sweep, each layer maintains its own prediction of what the layer above expects and minimizes local prediction error. Layers can update in parallel because no layer needs the full network’s backward pass to know its own gradient.

The approach is biologically motivated — brains show no known mechanism for the precise weight symmetry and phase locking that backpropagation requires across billions of synapses. Predictive coding offers a plausible model for how local Hebbian-style updates could produce network-wide learning.

Prior PC implementations struggled to match backprop in practice, particularly in large-scale settings. The augmented Lagrangian method is the key addition in PC-ALM. It enforces consistency constraints between layer predictions using a penalty-based optimization framework that provably converges without requiring end-to-end signal coordination. The result is a training algorithm that is fully layer-local, differentiable, and compatible with existing deep learning infrastructure.

Why 1,000 Layers Matters

Current frontier models do not use 1,000-layer feedforward networks. The paper tests PC-ALM against standard backpropagation across a matched depth-width grid on MNIST and Fashion-MNIST (depths 8 to 128), plus a targeted 1,000-layer test at width 32 on MNIST. At 1,000 layers, PC-ALM comes within roughly two percentage points of backprop performance using only local layer updates, with no global backward pass.

The number is significant as a scaling proof rather than a deployment target. The key finding is not that backprop fails at depth but that PC-ALM competes with it without the global synchronization requirement. In settings where modularity, asynchronous training, or biological plausibility are design goals, that independence from the backward pass is the relevant property.

Sakana’s Research Direction

Sakana AI, the Tokyo-based lab founded in 2023 by former Google Brain researchers, has focused on biologically inspired and evolutionary approaches to AI. Its Fugu series of models reached SWE-bench Pro scores in the 73% range using collective intelligence methods. PC-ALM fits the lab’s pattern of publishing foundational alternatives to standard deep learning assumptions rather than racing on benchmark tables.

The team positions the work as a proof-of-concept for the optimizer class. Comparisons are included against backprop at equivalent scale using MNIST and Fashion-MNIST. The code is open-sourced alongside the paper. Broader efficiency profiling on larger-scale benchmarks and production-class architectures is expected in follow-on work.