Hugging Face's TRL library has introduced LoRA adapter support to its AsyncGRPOTrainer, enabling reinforcement learning practitioners to run model training and inference on separate machines without the overhead of full-weight model synchronization. The update, shipping in TRL v1.14, leverages Storage Buckets as a shared filesystem and routes adapter updates through a proxy server, eliminating the need for NCCL communication across separate Job instances.
In a real-world deployment, the architecture runs AsyncGRPOTrainer on one Hugging Face Job, with vLLM inference replicas on separate compute instances. A Storage Bucket mounted across all instances distributes LoRA adapters—which are only a few megabytes compared to the full 3 GB model. A proxy server handles routing each rollout to the replica holding its KV cache prefix and broadcasts adapter updates to all inference replicas.
The infrastructure achieved significant performance gains. A benchmark running 500 training steps completed in 53 minutes, down from 3 hours 27 minutes in previous configurations—a 4x speedup. The approach is particularly suited for RL training because LoRA rank-1 adapters can match full fine-tuning performance while dramatically reducing synchronization overhead between distributed components. Checkpoints and final adapters persist to Storage Buckets, enabling preempted trainers to resume without data loss.
Key Points
TRL v1.14 adds LoRA adapter training to AsyncGRPOTrainer, enabling full separation of training and inference across distributed Hugging Face Jobs
LoRA rank-1 adapters (few MB) sync via Storage Buckets instead of full 3 GB models, eliminating NCCL network requirements
Proxy server intelligently routes rollouts to KV-cache-holding replicas and broadcasts adapter updates to all vLLM instances
Real-world benchmark shows 4x speedup: 500-step training reduced from 3 hours 27 minutes to 53 minutes