
Real-time artificial intelligence development and industry coverage.
Back to Articles a]:hidden"> Why your fine-tuning job takes longer than the FLOPs say Team Article Published November 18, 2026 Upvote 8 +2 Bruno Duarte brunoduarte01 Follow Dharma-AI Gabriel Pimenta de Freitas Cardoso GabrielPimenta99 Follow Dharma-AI Isis F Carvalho isisfc Follow Dharma-AI Breno de Almeida Beleza BrenoBeleza Follow Dharma-AI Felipe Lima limafelipe Follow Dharma-AI :last-child]:mb-0"> The estimate everyone uses is compute-only What we built The data Results What the predictor relies on How wrong is it, and where What it is good for Limitations What's next Further reading Predicting end-to-end fine-tuning time on shared GPU clusters, from 272 production runs.
A preview of our upcoming paper, Predicting End-to-End Fine-Tuning Time for Generative Language Models. The full version will be on arXiv soon.
GPUs are among the most expensive and scarcest resources in AI today. Anyone running a shared cluster has to decide, every day, which jobs go on which GPUs and in what order. That plan is only as good as the estimate of how long each job will take. When a job finishes hours early, GPUs sit idle until the next one is dispatched. When it runs hours late, everything queued behind it slips, experiments wait and delivery dates move.
Better duration estimates turn this into planning instead of guesswork: what runs, when, and on how many GPUs, with less idle capacity in between. Yet the estimate most teams use is a back-of-the-envelope FLOP calculation. It says nine hours, and the job finishes the next afternoon.
We wanted to know how far off that math really is, and whether we could do better before a job is even dispatched. So we took 272 real supervised fine-tuning (SFT) jobs from our production pool of NVIDIA H200 GPUs, about 7,382 GPU-hours in total, and compared analytical estimates against learned predictors.
The short version: the usual FLOP-based estimate explains little of what we observed (R² = 0.31, MAE 12.41 h). A CatBoost regressor that only sees information available before dispatch reaches R² = 0.98, with a mean absolute error of 1.71 h and a median error of 0.51 h in 5-fold cross-validation. The gap comes from two things a closed-form estimate cannot do. The regressor sees more training-specific signals, such as evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and it learns the nonlinear interactions between them from past runs.
The estimate everyone uses is compute-only For a dense Transformer, training costs roughly 6·P·D floating-point operations, where P is the parameter count and D the number of training tokens. Divide that by what your GPUs can deliver and you get a training-loop time:
T_theoretical = (6 · P · D) / (N_GPU · FLOPS_peak · MFU) Here N_GPU is the number of GPUs, FLOPS_peak is the peak throughput of one GPU from its spec sheet, and MFU (Model FLOPs Utilization) is the fraction of that peak the training actually achieves. Real jobs never run at peak because of memory traffic, communication and kernel inefficiencies, so MFU typically sits somewhere between 30% and 60%, and the right value for a given job is not known in advance.
This is a good description of the optimization loop. A job is more than its loop, though. Before the first step there is environment setup, model download and loading, and tokenization. During training there are periodic evaluations and checkpoint uploads. At the end, the final model goes to object storage. On a multi-tenant cluster, all of it competes with other jobs for network and storage I/O.
These overheads depend on configuration, and they do not scale neatly with workload or GPU count. Two jobs with the same nominal compute can finish hours apart.
To keep the comparison fair, the FLOP baseline we test counts the scheduled evaluation tokens as well as the training tokens, and we sweep MFU over a wide range.
What we built We treat end-to-end duration as a supervised regression problem. Given a feature vector built from the job configuration and model metadata, predict the wall-clock time between the moment the job gets its GPUs and the moment it is done. Queue time is excluded.
We compared Linear Regression, Ridge, XGBoost and CatBoost. Feature selection combined correlation filtering, mutual information and permutation importance across 18 candidate feature sets, followed by a grid search with MAE as the criterion. The final CatBoost model uses 24 features.
The data 272 SFT runs from the operational logs of a corporate production environment, covering several model families up to 15 billion parameters, both LoRA and full fine-tuning, and a wide range of dataset sizes and hyperparameters. Shared infrastructure. Jobs ran on 1, 2 or 4 GPUs drawn from a pool of 8 to 32 H200s across 1 to 4 nodes, alongside other training and inference workloads. The measured times include real contention. Durations from 1.6 to 172.4 hours, with a mean of 23.43 h and a median of 15.29 h. Stack: Hugging Face Transformers and Unsloth. We model the stack as a categorical feature and make no assumption about a fixed speedup between the two.
Distribution of end-to-end duration across the 272 runs. Mean 23.43 h, median 15.29 h.
Hugging Face
Original Reporting Source
Explore features, ratings, and alternatives from our 19,000+ AI tools catalog.
AssistLoop's AI chatbots slash response times by 70% and boost satisfaction, scaling support for e-commerce and SaaS without extra hires.
Design professional websites effortlessly with AI.
Hypertxt.ai is an AI-powered SEO content writing tool that helps create high-quality, optimized content for improved search engine rankings. It offers various features to streamline the content creation process, from keyword research to content optimization and plagiarism checks.