Articles » Ollama guide

How to Run Ollama on a Vast.ai Cloud GPU

Ollama provides a straightforward way to download and run supported language models. A rented GPU helps when a laptop lacks enough VRAM or a temporary project needs faster generation.

Choose a suitable GPU

Memory use depends on model size, quantization, context length, and concurrent requests. Start with a smaller model that leaves VRAM headroom. Compare reliability, storage, and bandwidth as well as hourly price.

Connect and verify NVIDIA access

nvidia-smi

Use a CUDA-capable Linux template with SSH. Our SSH guide covers keys and port forwarding.

Install Ollama

Follow the current Linux method in the official Ollama documentation. Review any install script before running it.

ollama --version
ollama serve

Use a process supervisor or terminal multiplexer if the server must survive an SSH disconnect.

Pull and test a model

ollama pull MODEL_NAME
ollama run MODEL_NAME

Choose a model listed by Ollama whose license and memory requirements fit your use.

Call the API

curl http://127.0.0.1:11434/api/generate -H 'Content-Type: application/json' -d '{"model":"MODEL_NAME","prompt":"Explain GPU memory briefly.","stream":false}'

Keep the endpoint private

ssh -p MAPPED_PORT -L 11434:127.0.0.1:11434 root@HOST_ADDRESS

Your local application can now use http://127.0.0.1:11434. Do not expose an unauthenticated model API publicly.

Persist model downloads and stop billing

Mount persistent storage before downloading large models, maintain a separate backup for important data, save outputs, and verify them before stopping compute. Review storage charges separately. For higher-throughput serving, compare our vLLM guide.