First-hand information for everyone
Main discovery METTA decomposes the fast adaptation phase into two temporal scales. A causal neural memory performs fine-grained within-episode adaptation. Compact episode-level summaries support long-range, cross-episode task inference without attending over the full interaction history. (Supported by f1)
How it works (method and architecture) Within an episode, METTA uses Test‑Time Training (TTT) layers as the neural memory. Each TTT layer holds fast weights that are initialized at episode start and updated online via self-supervised gradient steps while processing the episode. This implements causal, per-step adaptation during interaction. (f2) At the end of each episode, METTA stores the final adapted within-episode representation as a compact episode summary. These summaries are causally available to later episodes. (f12) Cross-episode aggregation is performed in two ways. One is a simple exponential moving average (EMA) of stored summaries. The other is a learnable attention layer that retrieves from stored episode summaries and fuses the retrieved vector with the current within-episode representation via layer normalization. (f3) METTA is modular: while TTT is the default within-episode memory, the framework can use other causal neural-memory modules. (f13) All modules (within-episode memory, cross-episode aggregation, actor, critic) are trained end-to-end with PPO. To save compute when training adaptive sequence models, PPO updates are applied sequentially over trajectory segments (each episode is one segment). (f9)
Concrete results On unseen held-out Meta‑World ML10/ML45 test environments, METTA reaches a success rate over 50% while using a low training budget and short wall-clock time. (f6) On the long‑context, sparse‑reward PointMaze at 1.5M environment steps, METTA shows more efficient exploration than transformer baselines. Table 1 (mean ± SEM) reports: - METTA: Coverage 0.447 ± 0.036; Episode Jaccard 0.792 ± 0.022; Steps 194.8 ± 30.1. (f7) - ECET: Coverage 0.250 ± 0.087; Episode Jaccard 0.935 ± 0.028; Steps 427.3 ± 44.5. (f7) - TrMRL: Coverage 0.330 ± 0.031; Episode Jaccard 1.000 ± 0.000; Steps 373.4 ± 16.1. (f7) Across locomotion, manipulation, and navigation benchmarks, METTA improves fast adaptation in both standard and long‑context meta‑RL settings. The relative gains are especially notable when generalizing to unseen, held-out Meta‑World task families not encountered during meta‑training (task‑family‑level out‑of‑distribution generalization). (f8) Learned context representations show task‑discriminative structure. t‑SNE visualizations and pairwise cosine similarities of final episode representations on ML10 organize contexts by task families and align test tasks with related training tasks. This suggests generalization rather than mere memorization. (f14)
Computation and scaling TTT yields a constant per-step inner‑update cost that does not grow with timestep history, preserving linear sequence complexity for the within‑episode memory. Over a complete trial, METTA requires O(E^2 T) computation with attention‑based cross‑episode aggregation and O(E T) with EMA. By contrast, transformer baselines incur higher costs (e.g., O(E^2 T^2) for TrMRL and O(E T^3 + E^3 T) for ECET). (f10)
Supported caveats and limitations Disabling the online within‑episode TTT updates causes a large drop in test performance on PointMaze, but has little effect on ML10. This shows that within‑episode online adaptation is particularly important when task‑relevant information is sparse and must be accumulated through interaction. (f11) The evaluations are in a fixed‑task trial setting. Extending METTA to continual or nonstationary environments (for example by adding temporal or recency information into cross‑episode memory) is noted as future work. The current work does not demonstrate continual nonstationary task changes. (f15)
Benchmarks and baselines Experiments used standard meta‑RL benchmarks including MuJoCo HalfCheetahVel (locomotion), Meta‑World ML1/ML10/ML45 (robotic manipulation), and sparse‑reward PointMaze (navigation). (f4) METTA was compared against recurrent, gradient‑based, latent‑context, and transformer‑based baselines (e.g., RL^2, MAML, PEARL, VariBAD, TrMRL, ECET, AMAGO). All baselines used their default implementations and hyperparameters under the same evaluation protocol; reported results are averaged over five seeds with shaded regions showing ±1 SEM. (f5)
Overall significance METTA shows that combining online within‑episode neural memory with compact, causal episode summaries can improve fast adaptation and long‑context inference in meta‑RL. It achieves favorable empirical gains and a practical computational trade‑off, while leaving extensions to continual nonstationary settings for future work. (f1, f2, f3, f6, f7, f8, f10, f11, f15)