Tutorials
AMD Developer Cloud Tutorial: Host an LLM on an AMD GPU for AI Hackathons
AI hackathon guide: run a model with vLLM on an AMD MI300X GPU and expose an OpenAI-compatible API endpoint in under 30 minutes.
By Starfield LabsPublished Intermediate30 min

Before you start
- A computer with a terminal (Mac, Linux, or Windows with WSL)
- An SSH key pair (you'll generate one if you don't have one)
- A credit card on file (required to unlock GPU access, charges are covered by free credits)
Introduction
If you've been building AI applications but relying entirely on managed API endpoints, this tutorial is your entry point into running models on raw GPU hardware: your own endpoint, your own model, your own infrastructure.
AMD Developer Cloud gives you on-demand access to the AMD Instinct MI300X, a GPU with 192 GB of VRAM. You don't need physical hardware or a data center. For $1.99/hour (or free with credits), you get a machine that can load many open-source models.
In this tutorial you'll go from zero to a live, publicly accessible AI API endpoint in under 30 minutes:
- Sign up for the AMD AI Developer Program and claim your free credits
- Create a GPU Droplet pre-loaded with vLLM
- SSH in, launch a model, and hit the API from your laptop
This setup is practical for AI hackathons, where you need a real inference endpoint running quickly without spending days on infrastructure. Having your own AMD-hosted model gives you control over latency, model selection, and cost, which helps when you are working under hackathon time constraints. When your endpoint is ready, see the upcoming AI hackathons to put it into practice.
Why AMD Developer Cloud?
Before diving in, here's why this setup makes sense for developers building AI applications:
| Feature | AMD MI300X | NVIDIA H100 |
|---|---|---|
| VRAM | 192 GB | 80 GB |
| Memory bandwidth | 5.3 TB/s | 3.35 TB/s |
| Price (cloud) | $1.99/hr | ~$4-6/hr |
| Open-source stack | ROCm (fully open) | CUDA (proprietary) |
With 192 GB of VRAM, the MI300X can hold very large models, such as Llama 3.1 405B, on a single node. For a hackathon or a prototype, that gives you a lot of room to work with.
Prerequisites
- A computer with a terminal (Mac, Linux, or Windows with WSL)
- An SSH key pair (you'll generate one if you don't have one)
- A credit card on file (required to unlock GPU access, charges are covered by free credits)
Phase 1: Getting Access and Credits
Joining the AMD AI Developer Program
A direct way to get GPU credits is through the AMD AI Developer Program. It gives you $100 in credits (~50 hours on a single MI300X) just for signing up, no approval process.
- Go to AMD AI Developer Program
- Create a free account
- Credits are applied automatically to your AMD Developer Cloud account
As a bonus, the program also includes a private Discord with AMD engineers and a one-month DeepLearning.AI pro membership.
Accessing AMD Developer Cloud
AMD Developer Cloud is powered by DigitalOcean. Once you've signed up on the AMD AI Developer Program, proceed to sign up at AMD Developer Cloud.
Sign in with your AMD Developer account. You'll land on a standard DigitalOcean-style dashboard, but this instance gives you access to AMD Instinct GPU hardware that isn't available on the regular DigitalOcean.
Adding a Payment Method
Before you can create a GPU Droplet, the platform requires a valid payment method on file, even if your $100 credits will cover everything. Without a card, the "Create GPU Droplet" button stays grayed out.
Go to Billing in the left sidebar and add a card. Your credits will be consumed before the card is charged; the card is a safety net for overage.
Cost reference: At $1.99/hr for a single MI300X, $100 gives you approximately 50 hours of GPU time. Credits typically expire 30 days after being applied, check your expiry date under Billing.
Phase 2: Creating the GPU Droplet
Choosing Your Configuration
Navigate to GPU Droplets in the sidebar and click Create GPU Droplet.
You'll be presented with several choices. Here's what to select and why:
Region
Choose ATL1 (Atlanta) as it consistently has MI300X availability. If you see capacity issues, try the other available region.
GPU Plan: MI300X (1 GPU)
| Spec | Value |
|---|---|
| GPU | 1x AMD Instinct MI300X |
| VRAM | 192 GB HBM3 |
| vCPU | 20 |
| RAM | 240 GB |
| Boot disk | 720 GB NVMe |
| Scratch disk | 5 TB |
| Cost | $1.99/hr |
The single MI300X is the right choice for running and serving a model. The 8x MI300X option ($15.92/hr) is for large distributed training jobs or serving 70B+ parameter models at scale, more than an initial deployment needs.
Image: vLLM Quick Start Package
When selecting the image, you'll see several options:
| Image | What it is |
|---|---|
| vLLM (recommended) | Pre-configured LLM inference engine with OpenAI-compatible API |
| ROCm Software | Bare AMD GPU environment, requires manual setup |
| SGLang | Alternative inference framework |
| PyTorch | Raw PyTorch, for training not serving |
| Megatron | Large-scale distributed training only |
Select the vLLM Quick Start image. This gives you a pre-built Docker container with vLLM, ROCm, and all dependencies already installed and configured. Zero setup time.
SSH Key
GPU Droplets require SSH key authentication, no password login. If you already have a key pair, add your public key here. If not, generate one:
ssh-keygen -t ed25519 -C "your@email.com"
Your public key is at ~/.ssh/id_ed25519.pub. Copy its contents and paste it into the SSH key field.
Creating the Droplet
Click Create GPU Droplet. The droplet takes 2-4 minutes to provision. You'll see it move from Creating to Active in the dashboard. Once active, copy the Public IP address as you'll need it for the next step.
Phase 3: Connecting and Launching a Model
SSH Into the Droplet
Once the droplet is active, connect to it from your terminal:
ssh root@<your-droplet-ip>
Replace <your-droplet-ip> with the public IP from the dashboard. If your SSH key is in a non-default location, specify it with -i:
ssh -i ~/.ssh/id_ed25519 root@<your-droplet-ip>
You'll land in a root shell on Ubuntu 24.04.
Entering the vLLM Container
The vLLM Quick Start image runs a pre-built Docker container named rocm. Enter it:
docker exec -it rocm /bin/bash
You're now inside the container where vLLM is installed and the AMD ROCm stack is ready.
Launching a Model
Run the following command to start the vLLM API server with Qwen2.5-1.5B-Instruct:
python -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-1.5B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--dtype float16 \
> /tmp/vllm.log 2>&1 &
Why Qwen2.5-1.5B?
- Downloads freely from HuggingFace with no authentication (unlike Llama, which requires accepting Meta's license)
- At 1.5B parameters (~3 GB), it loads in seconds on the MI300X's 192 GB of VRAM
- Supports the OpenAI chat format (
/v1/chat/completions) out of the box - Produces quality responses for demos and prototypes
Why --dtype float16?
The model's default dtype is bfloat16, but float16 has broader compatibility with AMD ROCm at this stage. vLLM will warn you about this, casting to float16 is the correct choice here.
Why run it in the background with &?
Running in the foreground locks your terminal. The background flag (&) plus log redirect (> /tmp/vllm.log 2>&1) lets you monitor startup progress while keeping your shell free.
Monitoring Startup
Watch the logs until the server is ready:
tail -f /tmp/vllm.log
You'll see the model weights download from HuggingFace, load into GPU memory, and the ROCm graph compilation complete. The whole process takes about 25-30 seconds on the initial run. Wait for this line:
INFO: Application startup complete.
Then press Ctrl+C to stop tailing the log. Your server is live.
Phase 4: Testing the API
Test From Inside the Droplet
With the server running, run a quick test from inside the container:
curl -s http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"messages": [{"role": "user", "content": "Hello"}]
}'
You'll get a response like:
{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"choices": [{
"message": {
"role": "assistant",
"content": "Hello! How can I assist you today?"
},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": 30,
"completion_tokens": 10,
"total_tokens": 40
}
}
Test From Your Laptop
Open a new terminal on your local machine and call the public IP directly:
curl -s http://<your-droplet-ip>:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"messages": [{"role": "user", "content": "Explain ROCm in one sentence."}]
}'
Port 8000 is accessible from the public internet because the vLLM Docker container maps it directly via iptables. Docker bypasses UFW firewall rules when using port mapping, so your endpoint is live.
Calling it Like the OpenAI SDK
Because vLLM exposes an OpenAI-compatible API, you can use the standard openai Python client by pointing it at your droplet:
from openai import OpenAI
client = OpenAI(
base_url="http://<your-droplet-ip>:8000/v1",
api_key="not-required" # vLLM doesn't require auth by default
)
response = client.chat.completions.create(
model="Qwen/Qwen2.5-1.5B-Instruct",
messages=[{"role": "user", "content": "What can you do?"}]
)
print(response.choices[0].message.content)
Any code that works with OpenAI's gpt-4o works with your AMD-hosted model, just change the base_url and model name.
Testing With a Chat UI
If you'd rather test your endpoint through a proper chat interface instead of curl commands, you can use this single-file HTML chat template built specifically for vLLM and OpenAI-compatible endpoints: vllm-chat-template.
No build step, no dependencies. Clone it, open the file in a browser, and you're chatting with your AMD-hosted model.
Step 1. Clone the repo:
git clone https://github.com/Stephen-Kimoi/vllm-chat-template
cd vllm-chat-template
Step 2. Enable CORS on your vLLM server.
By default, the browser will block requests from a local HTML file to your remote server. Restart vLLM with CORS enabled:
pkill -f vllm
python -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-1.5B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--dtype float16 \
--allowed-origins '["*"]' \
> /tmp/vllm.log 2>&1 &
Step 3. Configure the template.
Open index.html and update the CONFIG block near the top of the <script> section:
const CONFIG = {
apiUrl: 'http://<your-droplet-ip>:8000/v1/chat/completions',
model: 'Qwen/Qwen2.5-1.5B-Instruct',
badge: 'vLLM',
headerTitle: 'Chat - AMD MI300X',
statusText: 'Connected',
footerInfo: 'vLLM · OpenAI-Compatible API',
specs: [
{ label: 'Endpoint', value: '<your-droplet-ip>:8000', accent: true },
{ label: 'Model', value: 'Qwen2.5-1.5B-Instruct' },
{ label: 'Hardware', value: 'AMD MI300X' },
],
};
Replace <your-droplet-ip> with your droplet's public IP, save the file, and open it directly in your browser (open index.html on Mac or double-click on Windows).
The UI shows a full chat interface on the left and the raw request/response JSON on the right, useful for seeing exactly what your endpoint is returning.
Phase 5: Loading Larger Models
Running a 7B Model
Once you've confirmed the basic setup works, try a larger model. The MI300X's 192 GB of VRAM means you have room to run models that would require multiple GPUs elsewhere:
python -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--dtype float16 \
> /tmp/vllm.log 2>&1 &
At 7B parameters (~14 GB), this model loads in about a minute and produces noticeably better outputs than the 1.5B variant.
Running Llama Models
For Llama 3.x models, you'll need to accept Meta's license on HuggingFace, then pass your token:
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.2-3B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--dtype float16 \
--hf-token <your-huggingface-token> \
> /tmp/vllm.log 2>&1 &
Checking GPU Memory Usage
To see how much VRAM your model is using:
rocm-smi
This is the AMD equivalent of nvidia-smi. You'll see GPU utilization, memory usage, and temperature.
Phase 6: Cost Management
The Key Rule
Even when the droplet is powered off, you are still billed. To stop charges completely, you must Destroy the droplet from the dashboard, not just power it off.
Quick Reference: Stop and Restart
Stop vLLM (inside the container):
pkill -f vllm
Full restart sequence from your laptop:
# 1. SSH in
ssh root@<your-droplet-ip>
# 2. Enter the container
docker exec -it rocm /bin/bash
# 3. Start vLLM
python -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-1.5B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--dtype float16 \
> /tmp/vllm.log 2>&1 &
# 4. Wait for startup (~30s on subsequent runs, weights are cached)
tail -f /tmp/vllm.log
Snapshots
If you want to save your environment and restore it later, take a snapshot before destroying:
AMD Developer Cloud > GPU Droplets > your droplet > Snapshots > Take Snapshot
Snapshots incur a small storage cost but let you restore the exact state of your droplet without re-running setup.
Cost Breakdown
| Resource | Rate | Notes |
|---|---|---|
| MI300X (1 GPU) | $1.99/hr | Covered by $100 credits |
| MI300X (8 GPU) | $15.92/hr | For large-scale training only |
| Bandwidth | Included | Standard transfer pool |
| Snapshot storage | ~$0.05/GB/month | Optional |
Summary: What You Built
| Component | Detail |
|---|---|
| Hardware | AMD Instinct MI300X, 192 GB HBM3 VRAM |
| Platform | AMD Developer Cloud (DigitalOcean) |
| Inference engine | vLLM (OpenAI-compatible) |
| Model | Qwen/Qwen2.5-1.5B-Instruct |
| API endpoint | POST /v1/chat/completions |
| Accessible from | Anywhere via public IP |
| Cost | $1.99/hr, ~50 hrs covered by free credits |
You now have a live LLM endpoint running on AMD GPU hardware. From here you can swap in any model from HuggingFace, connect it to a LangChain agent, a RAG pipeline, or any application that speaks the OpenAI API format.
Frequently asked questions
Can I use AMD Developer Cloud for an AI hackathon?
Yes. AMD Developer Cloud is well suited to AI hackathons. For $1.99/hr (or free with the $100 credit grant from the AMD AI Developer Program), you get a dedicated MI300X instance with 192 GB of VRAM that can run many open-source models. The vLLM Quick Start image means you can have a live API endpoint in under 30 minutes, which matters when you're working against a hackathon deadline.
Do I need GPU or Linux experience to follow this tutorial?
No prior GPU experience is required. The tutorial uses a pre-configured vLLM Docker image that handles the entire ROCm setup for you. Basic terminal comfort (SSH, running commands) is all you need.
How long does it take to get the endpoint running?
Under 30 minutes from scratch. Droplet provisioning takes 2-4 minutes. The initial model load (Qwen2.5-1.5B) takes about 30 seconds. After that run, model weights are cached and subsequent starts take under 10 seconds.
Resources
- AMD AI Developer Program (free credits)
- AMD Developer Cloud
- Getting Started on AMD Developer Cloud
- ROCm Documentation
- ROCm Linux Installation Guide
- ROCm on GitHub
- AMD Training Resources
How this tutorial was tested
This content is carried over from Steve's earlier tutorial and updated for Starfield Labs. It was not re-run in this build. The commands, timings and outputs above are the author's original results, and the author's video and screenshot were not carried over.
The following were not run here: signing up for the AMD AI Developer Program and AMD Developer Cloud (these need accounts and a card on file), creating the paid GPU Droplet, every command that runs on the droplet (SSH, Docker, vLLM, rocm-smi), the Llama model download (this needs a Hugging Face token), and the chat UI template.