Articles » Benchmark guide

How to Benchmark a Cloud GPU Before a Long Training Job

A low hourly price is not enough. Slow storage, weak CPU allocation, network bottlenecks, throttling, or an unstable host can cost more over the complete job.

Record the environment

nvidia-smi
lscpu | sed -n '1,25p'
free -h
df -h
python --version

Confirm the model and VRAM match the offer. Save driver and CUDA information with your results.

Check idle health

nvidia-smi --query-gpu=name,memory.total,temperature.gpu,power.limit,clocks.sm --format=csv

Unexpected memory use can signal a configuration issue. Temperature and clocks are most meaningful under sustained load.

Measure framework compute

python -c "import torch,time; a=torch.randn(8192,8192,device='cuda',dtype=torch.float16); b=a@a; torch.cuda.synchronize(); t=time.time(); b=a@a; torch.cuda.synchronize(); print(time.time()-t)"

This is a comparison test, not a universal training score. Use the same image, shapes, dtype, and commands across offers.

Test the actual storage path

dd if=/dev/zero of=/workspace/io-test.bin bs=1M count=2048 conv=fdatasync
dd if=/workspace/io-test.bin of=/dev/null bs=1M
rm /workspace/io-test.bin

Cached reads can inflate results, and this test does not represent every workload, but it catches severely slow storage.

Test real data routes

Download a representative public object from the same model or dataset host your job will use. A generic speed test may not predict routing to the actual source. Measure upload performance if checkpoints must leave the instance.

Watch sustained behavior

watch -n 1 'nvidia-smi --query-gpu=utilization.gpu,temperature.gpu,power.draw,clocks.sm --format=csv,noheader'

Run a representative workload for 10–15 minutes and look for falling clocks, repeated errors, or utilization gaps caused by data loading.

Compare total economics

Estimate runtime times compute price, then add storage and bandwidth. A faster GPU can be cheaper overall if it finishes much sooner. Keep a log of offer ID, date, image, commands, results, and failures. Use our GPU comparison to choose candidates.