AiAnyTool - AI Tools Directory LogoAiAnyTool - AI Tools Directory Logo
AIAnyToolAI Tools Directory
Full ListAI CategoriesAI News Today
Loading theme toggle
Submit an AISign up
AiAnyTool - AI Tools Directory LogoAiAnyTool - AI Tools Directory Logo
AiAnyTool

Artificial intelligence for everyone. Discover, compare, and leverage over 19,000 verified AI tools.

Updated daily with manual verification

Resources

  • AI Deals & Discounts
  • AI News Today
  • Tutorials & Guides
  • MCP Servers Directory

Directory

  • Full AI List
  • AI Categories
  • Top 10 Best AI
  • New AI Tools
  • Submit an AI Tool →

About & Legal

  • About AiAnyTool
  • Contact Support
  • Privacy Policy
  • Terms of Service

© 2026 AiAnyTool.com — All rights reserved.

The definitive artificial intelligence directory and search engine.

HomeAI NewsOpen Source

Why your fine-tuning job takes longer than the FLOPs say

Hugging FaceOpen SourceNovember 18, 2026(2d ago)5 min readBREAKING
Share:
Why your fine-tuning job takes longer than the FLOPs say
Story Summary

Real-time artificial intelligence development and industry coverage.

Full Coverage Details

Hugging Face Blog

Back to Articles a]:hidden"> Why your fine-tuning job takes longer than the FLOPs say Team Article Published November 18, 2026 Upvote 8 +2 Bruno Duarte brunoduarte01 Follow Dharma-AI Gabriel Pimenta de Freitas Cardoso GabrielPimenta99 Follow Dharma-AI Isis F Carvalho isisfc Follow Dharma-AI Breno de Almeida Beleza BrenoBeleza Follow Dharma-AI Felipe Lima limafelipe Follow Dharma-AI :last-child]:mb-0"> The estimate everyone uses is compute-only What we built The data Results What the predictor relies on How wrong is it, and where What it is good for Limitations What's next Further reading Predicting end-to-end fine-tuning time on shared GPU clusters, from 272 production runs.

A preview of our upcoming paper, Predicting End-to-End Fine-Tuning Time for Generative Language Models. The full version will be on arXiv soon.

GPUs are among the most expensive and scarcest resources in AI today. Anyone running a shared cluster has to decide, every day, which jobs go on which GPUs and in what order. That plan is only as good as the estimate of how long each job will take. When a job finishes hours early, GPUs sit idle until the next one is dispatched. When it runs hours late, everything queued behind it slips, experiments wait and delivery dates move.

Better duration estimates turn this into planning instead of guesswork: what runs, when, and on how many GPUs, with less idle capacity in between. Yet the estimate most teams use is a back-of-the-envelope FLOP calculation. It says nine hours, and the job finishes the next afternoon.

We wanted to know how far off that math really is, and whether we could do better before a job is even dispatched. So we took 272 real supervised fine-tuning (SFT) jobs from our production pool of NVIDIA H200 GPUs, about 7,382 GPU-hours in total, and compared analytical estimates against learned predictors.

The short version: the usual FLOP-based estimate explains little of what we observed (R² = 0.31, MAE 12.41 h). A CatBoost regressor that only sees information available before dispatch reaches R² = 0.98, with a mean absolute error of 1.71 h and a median error of 0.51 h in 5-fold cross-validation. The gap comes from two things a closed-form estimate cannot do. The regressor sees more training-specific signals, such as evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and it learns the nonlinear interactions between them from past runs.

The estimate everyone uses is compute-only For a dense Transformer, training costs roughly 6·P·D floating-point operations, where P is the parameter count and D the number of training tokens. Divide that by what your GPUs can deliver and you get a training-loop time:

T_theoretical = (6 · P · D) / (N_GPU · FLOPS_peak · MFU) Here N_GPU is the number of GPUs, FLOPS_peak is the peak throughput of one GPU from its spec sheet, and MFU (Model FLOPs Utilization) is the fraction of that peak the training actually achieves. Real jobs never run at peak because of memory traffic, communication and kernel inefficiencies, so MFU typically sits somewhere between 30% and 60%, and the right value for a given job is not known in advance.

This is a good description of the optimization loop. A job is more than its loop, though. Before the first step there is environment setup, model download and loading, and tokenization. During training there are periodic evaluations and checkpoint uploads. At the end, the final model goes to object storage. On a multi-tenant cluster, all of it competes with other jobs for network and storage I/O.

These overheads depend on configuration, and they do not scale neatly with workload or GPU count. Two jobs with the same nominal compute can finish hours apart.

To keep the comparison fair, the FLOP baseline we test counts the scheduled evaluation tokens as well as the training tokens, and we sweep MFU over a wide range.

What we built We treat end-to-end duration as a supervised regression problem. Given a feature vector built from the job configuration and model metadata, predict the wall-clock time between the moment the job gets its GPUs and the moment it is done. Queue time is excluded.

We compared Linear Regression, Ridge, XGBoost and CatBoost. Feature selection combined correlation filtering, mutual information and permutation importance across 18 candidate feature sets, followed by a grid search with MAE as the criterion. The final CatBoost model uses 24 features.

The data 272 SFT runs from the operational logs of a corporate production environment, covering several model families up to 15 billion parameters, both LoRA and full fine-tuning, and a wide range of dataset sizes and hyperparameters. Shared infrastructure. Jobs ran on 1, 2 or 4 GPUs drawn from a pool of 8 to 32 H200s across 1 to 4 nodes, alongside other training and inference workloads. The measured times include real contention. Durations from 1.6 to 172.4 hours, with a mean of 23.43 h and a median of 15.29 h. Stack: Hugging Face Transformers and Unsloth. We model the stack as a categorical feature and make no assumption about a fixed speedup between the two.

Distribution of end-to-end duration across the 272 runs. Mean 23.43 h, median 15.29 h.

H

Hugging Face

Original Reporting Source

Read Full Article at Source

AI Tools Mentioned in this Story

Explore features, ratings, and alternatives from our 19,000+ AI tools catalog.

Explore Directory

AssistLoop

Free plan with 150 message credits and basic features, Basic at $25/month for 3,000 credits and lead collection, Pro at $109/month for 15,000 credits and advanced actions, Business with custom pricing for 50,000+ credits and unlimited agents.

AssistLoop's AI chatbots slash response times by 70% and boost satisfaction, scaling support for e-commerce and SaaS without extra hires.

AlternativesView Tool

GetResponse AI Website Builder

Free + from $10.23/mo

Design professional websites effortlessly with AI.

AlternativesView Tool

HyperTxt

Lite plan at $15/month with 20 credits, Pro at $39/month (most popular) with 100 credits and priority support, Pro+ at $125/month with 500 credits and advanced features, free trial available on all paid plans.

Hypertxt.ai is an AI-powered SEO content writing tool that helps create high-quality, optimized content for improved search engine rankings. It offers various features to streamline the content creation process, from keyword research to content optimization and plagiarism checks.

AlternativesView Tool
← Back to AI News RadarExplore Trending Stories
Trending AI Stories
View All
#1

Signal blocks screenshots on iOS 27 and Android to stop AI agents from reading private chats

The Decoder • 3h ago
#2

Google's Gemini can now run Mac apps and finish desktop chores in the background while you keep working

The Decoder • 4h ago
#3

Anthropic mathematician claims to have solved prestigious Millennium Prize problem with Claude

The Verge AI • 6h ago
#4

Apple is hiring engineers to build its own AI agent

The Verge AI • 7h ago
#5

Apple joins the race for AI agents that control your device while raising new barriers for competitors

The Decoder • 8h ago

More in Open Source

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

2d ago
180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

2d ago
Hugging Face and OpenRouter join forces to make open-weight models easy to run

Hugging Face and OpenRouter join forces to make open-weight models easy to run

4d ago
Build a Robot Pick-and-Place Simulation with SimReady Assets

Build a Robot Pick-and-Place Simulation with SimReady Assets

Nov 13