Parcae is a stable looped transformer architecture designed to improve model quality without increasing parameter count. By utilizing a prelude-recurrent-coda structure and addressing training instabilities like residual state explosion, it achieves performance equivalent to models twice its size. The research also establishes the first scaling laws for looped architectures, emphasizing the need to increase recurrence and data capacity in tandem.
Key points
Looped transformer architectures can achieve parity with fixed-depth models at significantly lower parameter counts.
Effective training of recurrent blocks requires careful mitigation of residual state explosion and loss spikes.
Compute-optimal training for looped models follows specific scaling laws where loop count and training data must be scaled together.
Input injection (via addition or projection) is essential to maintain input influence across recursive passes.