AIUpdated November 4

Ai2’s direct-training tests find Gated DeltaNet and hybrid models generalizing farther than transformers

Ai2’s architecture comparison used a different setup: the team trained small, 275-million-parameter models from scratch on StateBench, with sequences up to 256 assignments, then tested them on sequences as long as 64,000—250 times the maximum training length. In those tests, Gated DeltaNet and hybrid models stayed near-perfect on R-trivial, Aperiodic, and Soluble, while transformers weakened as sequences grew and never learned General finite. On that hardest level, the flexible Gated DeltaNet variant stayed near-perfect to about 10 times its longest training sequence. Ai2 reported these results; they have not been independently verified.

1 publisher1 report

Coverage started November 4