ServiceNow CoreAI has introduced AutoSynthData, a system for automatically generating training data to improve AI agents in enterprise environments. The technology addresses a persistent challenge: while large language models are broadly capable, they often struggle with domain-specific workflows, tool integration, and organizational constraints. AutoSynthData converts agent failures into structured training tasks through a curriculum learning approach, allowing enterprises to systematically improve agents for their specific use cases.
The system works by identifying where a target model underperforms compared to a stronger teacher model, then generating new training tasks grounded in the actual environment. Each generated task includes a system specification defining constraints and context, a user prompt specifying what the agent should accomplish, and a verifier establishing success criteria. AutoSynthData ensures these tasks are feasible, realistic, and appropriately challenging—exposing genuine capability gaps without requiring manual annotation.
ServiceNow demonstrated the approach using EnterpriseOps Gym, an open-source environment for enterprise AI agent testing, and released associated datasets. The curriculum-based pipeline continuously adapts to focus on remaining capability gaps as models improve, potentially reducing manual effort required to prepare domain-specific training data for enterprise deployments.
Key Points
AutoSynthData automatically generates training data by analyzing model failures and learning from stronger teacher models to identify capability gaps
Uses curriculum learning to dynamically shift training focus toward remaining weaknesses as the agent improves
Generated tasks are rigorously validated for feasibility, realism, and appropriate difficulty to ensure meaningful training signals
ServiceNow released the system alongside EnterpriseOps Gym, an open-source enterprise agent testing environment with public datasets