Researchers at Dharma-AI analysed 272 supervised fine-tuning jobs run on a shared cluster of NVIDIA H200 GPUs, totalling about 7,382 GPU-hours. They found that the standard FLOP-based estimate, which divides training compute by peak GPU throughput at an assumed utilisation rate, explained little of the observed run times, with an R² of 0.31 and a mean absolute error of 12.41 hours. A CatBoost regression model that uses only information available before a job is dispatched reached an R² of 0.98, with a mean absolute error of 1.71 hours and a median error of 0.51 hours in five-fold cross-validation.
The gap, according to the authors, comes from overheads that a closed-form formula cannot capture. Environment setup, model download, tokenisation, periodic evaluations, checkpoint uploads and final model export all add time, and on a multi-tenant cluster they compete with other jobs for network and storage bandwidth. The model uses features such as evaluation volume, checkpoint frequency, per-GPU token load and fine-tuning regime, and learns the nonlinear interactions between them from past runs. Durations in the dataset ranged from 1.6 to 172.4 hours, with a mean of 23.43 hours and a median of 15.29 hours. The jobs covered models up to 15 billion parameters, both LoRA and full fine-tuning, and runs on one, two or four GPUs. The results are a preview of a forthcoming arXiv paper and come from a single corporate production environment.
For Australian businesses and professionals, the practical impact is limited to teams that run their own shared GPU clusters or pay for dedicated fine-tuning capacity. Better duration estimates could reduce idle GPU time and wasted cloud spend, but the approach needs a few hundred historical runs from your own environment before it becomes reliable, and the findings are specific to one H200 pool. The method works on job configuration rather than personal data, so the Privacy Act is not directly engaged by it, although any training datasets remain subject to the Act. Teams fine-tuning through a hosted API will see no change.
Key Points
FLOP-based estimates explained only about 31% of variance in run times across 272 H200 fine-tuning jobs (R² 0.31).
A CatBoost model using only pre-dispatch configuration data reached R² 0.98 and a 1.71-hour mean absolute error.
Evaluation schedules, checkpoint I/O and cluster contention account for much of the gap between estimate and actual runtime.
Observed durations ranged from 1.6 to 172.4 hours, so single-number compute estimates can miss by days.
The results come from one production environment, and the full arXiv paper is not yet published.