First-hand information for everyone
A new study found that a wide range of faults that occur when an environment and RL trainer communicate can go undetected. Researchers made the boundary a wire protocol and recorded every step on both sides in a step ledger. In four simulated, state‑based environments – Pendulum, HalfCheetah, Hopper and a 7‑joint Pusher arm – they injected as many as 23 faults into the data that crosses the wire and 2 into the trainer’s own log. The faults covered rewards, actions, final observations, episode endings, logs and even hidden variables such as shared module returns.
| kind | where | what was wrong |
|---|---|---|
| reward | RLlib [15] | an agent with no last observation got reward 0, not the environment’s |
| Tianshou 0.4.2 [16] |
| rewards earned while the opponent moved were dropped |
| action | PettingZoo 1.22 [17] | one-dimensional continuous actions were cast to integers |
| final obs. | SuperSuit [18] | the reset’s info overwrote the terminal observation |
| episode end | MAPPO’s SMACv2 wrapper [19] | dead agents’ padded steps were trained on as real |
| log | Gymnasium 1.3 (E51) | under SAME_STEP, every episode after an env’s first lost its first reward and a step (fixed in 1.4 [20]) |
| CleanRL [10] (E51) | on Gymnasium 1.3, not the 0.29 it pins, no episode was logged |
| RLlib 2.53 [21] | a shared module logged one agent’s return, not the sum |
| our specification (E44) | for a client written from it, the server logged a return of 0 |
The key metric was a learning‑curve threshold that compares each run’s mean return against a gain threshold set 500 points above the median of fault‑free seeds. When only a few percent of the transmitted values were altered, the threshold missed most faults, catching at most one run in ten. Only when every transmitted value was wrong did the threshold flag about 35–48 % of runs, and most of those failures were obvious because learning broke or logs were empty.
A second comparison looked at how many seed runs passed a statistical spread test. That test raised very few false alarms – 0 to 1 of ten fault‑free seeds – but still caught at most 21 % of faults when less than all values were wrong. This shows that learning‑curve and seed‑spread checks are weak guards against silent boundary faults.
| exp. | trainer | environments ran | result |
| E50 | SB3, PPO | in-process vs. bridge | same weights 6/6 |
| E53 | SB3, CNN | Atari frames, as E50 | same weights 3/3 |
| E52 | SB3, PPO | proxy, 0–25 ms each way | same weights 12/12 |
| E43 | PlugRL, FPO [41] | same machine | passes 6/6 |
| (HalfCheetah [42]) | laptop, Wi-Fi + Tailscale | passes 6/6 | |
| E44 | PlugRL, PPO [43] | Python env client | passes 3/3 |
| (Pendulum) | C++, no third-party library | passes 3/3 | |
| E48 | RLinf, PPO | same machine | passes 3/3 |
| (HalfCheetah) | laptop, Tailscale | passes 3/3 | |
| E49 | RLinf, SB3 (PPO) | C++ Pendulum | passes 6/6 |
| SB3, CleanRL (PPO) | HalfCheetah | passes 6/6 |
These findings highlight that even when trainers and environments run inside the same process, on the same machine, or even across the network, faults at the communication boundary can slip through without raising an error. The study also documents that deployments relying on such checks may overlook subtle faults that leave policies intact and only become apparent when evaluation policy quality is examined.
For details on the fault list and the precise injection methodology, see Table I. For the performance of the learning‑curve and seed‑spread checks, see Table III.