Tutorials
Best Practices for Deploying AI Agents with the Llama Stack for AI Hackathons
AI hackathon guide to deploying AI agents locally with Meta's Llama Stack: download a model, build a server and call it from Python.
By Starfield LabsPublished Intermediate20 min

Running a language model locally can mean a lot of dependencies, configurations, and cryptic error messages. Llama Stack is Meta's toolkit that aims to reduce that complexity.
Llama Stack is Meta's toolkit for AI development that covers basic inference through to conversational systems. You can run chat completions, generate embeddings for semantic search, and use safety features through Llama Guard, all locally.
Deploying AI agents with the Llama Stack is particularly valuable for AI hackathons, where participants may want to run AI models locally without relying on external APIs or cloud services. Whether you're participating in online AI hackathons or virtual AI hackathons, understanding how to deploy and optimize AI agents locally can help you build AI hackathon projects. If you're looking for upcoming AI hackathons to apply these skills, see the upcoming AI hackathons.
Before we write any code, we need to get access to the models. Head over to Meta's download page at llama.com/llama-downloads, where you fill out a request form.
Fill out the details and select the models you want access to. For this guide, we'll use Llama3.2-8B as named in the commands below, a middle ground between performance and resource usage. Once approved, Meta will send you download URLs for each model.
Let's set up your environment and get that model downloaded:
conda create -n llama python=3.10
conda activate llama
pip install llama-stack
Now we can download the model. Replace <META_URL> with the URL you received:
llama download --source meta --model-id Llama3.2-8B --meta-url <META_URL>
The download will take a few minutes and shows a progress bar. The model gets stored in your ~/.llama directory, and checksums are printed at the end to confirm you have an intact copy.
Now that our model is downloaded, let's build our first AI server. Llama Stack uses a build-configure-run workflow.
First, let's create your distribution. This step tells Llama Stack what components you want in your server:
llama stack build
The build process will walk you through setting up your server. Name it something simple like "my-local-stack" and choose "conda" for the image type. The prompts look like this:
> Enter an unique name for identifying your Llama Stack build distribution: my-local-stack
> Enter the image type you want your distribution to be built with: conda
Llama Stack is composed of several APIs working together. Let's configure the providers:
> Enter the API provider for the inference API: meta-reference
> Enter the API provider for the safety API: meta-reference
> Enter the API provider for the agents API: meta-reference
> Enter the API provider for the memory API: meta-reference
> Enter the API provider for the telemetry API: meta-reference
The defaults (meta-reference) are what we use here. They are Meta's reference implementations that handle running the model and managing conversations.
Next, configure your server:
llama stack configure my-local-stack
This is where we tell our server how to run. Start with the inference settings.
Keep the defaults for most settings, but make sure to point it to our Llama3.2-8B model. A sequence length of 4096 leaves room for context, and a batch size of 1 keeps things simple while getting started.
The safety settings are optional. You can skip them for initial testing, but they are worth exploring later if you build something for production.
Now that we've configured everything, let's start our server:
llama stack run my-local-stack
When you run this command, your Llama Stack server initializes its core components. Note that the client examples below connect to http://localhost:5000; keep the server bound to your own machine and do not expose it to the internet without access control.
Understanding the Server Architecture
The server exposes several key endpoints:
/inference/chat_completionfor text generation and conversations/inference/embeddingsfor vector representations/memory_banks/*for conversation state management/agentic_system/*for complex reasoning tasks
Let's explore how to interact with these endpoints using the Llama Stack Client:
from llama_stack_client import LlamaStackClient
# Initialize the client
client = LlamaStackClient(base_url="http://localhost:5000")
# Basic inference example
response = client.inference.chat_completion(
messages=[{
"role": "user",
"content": "Explain how neural networks process data."
}],
model="Llama3.2-8B"
)
print(response.choices[0].message.content)
Working with Conversation Memory
The Memory API enables contextual awareness across multiple interactions:
# Create a conversation session
conversation_id = client.memory.create_conversation()
# Initial context
response1 = client.inference.chat_completion(
messages=[{
"role": "user",
"content": "Let's discuss machine learning algorithms."
}],
conversation_id=conversation_id
)
# Follow-up with context
response2 = client.inference.chat_completion(
messages=[{
"role": "user",
"content": "What are their practical applications?"
}],
conversation_id=conversation_id
)
Implementing Safety Controls
For production deployments, enable content filtering and security features:
# Configure safety settings
client.safety.configure(
content_filtering=True,
prompt_injection_detection=True
)
response = client.inference.chat_completion_safe(
messages=[{
"role": "user",
"content": "Explain database security."
}],
safety_level="medium"
)
Parallel Processing
Handle multiple requests efficiently with batch processing:
responses = client.inference.batch_completion(
prompts=[
"Explain REST APIs.",
"Describe microservices.",
"Define cloud computing."
],
batch_size=3
)
for response in responses:
print(response.choices[0].message.content)
Error Management
Implement error handling for production stability:
from llama_stack_client.exceptions import ModelOverloadError, TokenLimitError
try:
response = client.inference.chat_completion(
messages=[{
"role": "user",
"content": extensive_text
}]
)
except TokenLimitError:
# Handle token limit exceeded
truncated_content = truncate_text(extensive_text)
response = retry_with_truncated_content(truncated_content)
except ModelOverloadError:
# Implement backoff strategy
response = retry_with_backoff()
Performance Monitoring
Enable logging and metrics collection:
# Configure monitoring
client.configure_logging(
log_level="DEBUG",
log_file="llama_stack.log"
)
client.metrics.track_performance(
track_latency=True,
track_token_usage=True,
track_memory_usage=True
)
On startup, the server logs show how Llama Stack orchestrates its components.
The server sets up multiple processing pipelines to handle concurrent requests. The logs show it initializing model parallelism, distributed data processing (DDP), and the inference pipeline.
Once the model loads, the server exposes its API endpoints.
Each endpoint covers a capability:
/inference/chat_completionhandles conversational AI/inference/embeddingsgenerates vector representations for semantic search/memory_banks/*endpoints manage conversation history and context/agentic_system/*enables complex multi-step reasoning
In the server logs you can see how requests flow through the system. 200 OK responses mean successful requests. Each request gets processed through the inference pipeline we configured, with the model generating responses in real-time.
This setup is flexible. Whether you're building a chatbot, a document analysis system, or a complex AI agent, these endpoints give you building blocks.
Now that we have our server running, let’s explore how to interact with it using the Llama Stack Client Python library. This library provides a convenient way to access the Llama Stack REST API from any Python application, making it easy to integrate AI capabilities into your projects.
Installing the Client
To get started, you’ll need to install the Llama Stack Client. You can do this easily with pip:
pip install llama-stack-client
Basic Usage
Once installed, you can use the client to interact with your Llama Stack server. Here’s a simple example to get you started:
from llama_stack_client import LlamaStackClient
from llama_stack_client.types import UserMessage
client = LlamaStackClient(
base_url="http://localhost:5000",
)
response = client.inference.chat_completion(
messages=[
UserMessage(
content="Hello world, write me a 2 sentence poem about the moon.",
role="user",
),
],
model="Llama3.2-8B",
stream=False,
)
print(response)
This code snippet initializes the client and sends a message to the chat completion endpoint. The response will contain the model's generated text, which you can then use in your application.
Asynchronous Usage
If you prefer asynchronous programming, the library also supports it. Simply import AsyncLlamaStackClient and use await for your API calls:
import asyncio
from llama_stack_client import AsyncLlamaStackClient
client = AsyncLlamaStackClient()
async def main():
response = await client.inference.chat_completion(
messages=[
UserMessage(
content="Tell me a joke.",
role="user",
),
],
model="Llama3.2-8B",
)
print(response)
asyncio.run(main())
Error Handling
The library includes error handling. If the client cannot connect to the API or if the API returns an error, you can catch these exceptions easily:
from llama_stack_client import LlamaStackClient, APIConnectionError, APIStatusError
client = LlamaStackClient()
try:
response = client.inference.chat_completion(
messages=[UserMessage(content="What's the weather like?", role="user")],
model="Llama3.2-8B",
)
except APIConnectionError:
print("Could not connect to the server.")
except APIStatusError as e:
print(f"API error: {e.status_code} - {e.response}")
Advanced Features
The client also supports advanced features like logging, retries, and timeouts. You can enable logging by setting the environment variable:
export LLAMA_STACK_CLIENT_LOG=debug
For requests that may take longer, you can configure timeouts:
client = LlamaStackClient(timeout=20.0) # 20 seconds timeout
Conclusion
With the Llama Stack Client, you've seen the fundamentals of running AI models locally. This tutorial covered the basics. For more detail on the APIs covered here, see the official documentation.
Use it at an AI hackathon
Running a model locally means your project does not depend on an external API during a short event, which can help in an AI hackathon. Local chatbots, semantic search and document analysis are realistic ideas to build on this setup. To find an event, see the upcoming AI hackathons.
Frequently asked questions
How can I use Llama Stack for AI agents in an AI hackathon?
You can run AI models locally without depending on external APIs, and build projects such as local chatbots, semantic search engines, document analysis systems, or multi-modal AI assistants.
Is it suitable for beginners?
It requires some technical knowledge of Python, command-line tools, and AI model concepts. The tutorial walks through the process step by step and suits those with intermediate programming skills.
Are there limitations in time-limited hackathons?
Yes. Downloading large model files can take significant time and bandwidth, local setup needs hardware with enough RAM and storage, and debugging local deployment issues can be time-consuming.
How this tutorial was tested
This content was carried over from an earlier tutorial and updated for Starfield Labs. It was not re-run in this build, and no step has been confirmed to work as written. To follow it you would need a Meta model-access request and download URL (accepted licence terms), conda, Python 3.10, enough disk, RAM and likely a GPU to run the model locally, and the llama-stack and llama-stack-client packages. No account, download or model was used here.