Recently, we talked to Dan Fu and Tri Dao – authors of “Hungry Hungry Hippos” (aka “H3”) – on our Deep Papers podcast. H3 is a proposed language modeling architecture that performs comparably to ...

For a while now, we’ve been talking about transformers, frontier neural network logic models, as a transformative technology, no pun intended. But now, these attention mechanisms have other competing ...

Researchers have introduced a new approach to sequence modeling called linear oscillatory state-space (LinOSS) models, designed for efficient learning on long sequences. Drawing inspiration from ...

Every state space model has an ARMA representation, and conversely every ARMA model has a state space representation. This section discusses this equivalence. The following material is adapted from ...