Skip to content
Starfield Labs

Tutorials

Best Practices for Deploying AI Agents with the Llama Stack for AI Hackathons

AI hackathon guide to deploying AI agents locally with Meta's Llama Stack: download a model, build a server and call it from Python.

By Starfield LabsPublished Intermediate20 min

Cover card for the tutorial "Best Practices for Deploying AI Agents with the Llama Stack for AI Hackathons"

Running a language model locally can mean a lot of dependencies, configurations, and cryptic error messages. Llama Stack is Meta's toolkit that aims to reduce that complexity.

Llama Stack is Meta's toolkit for AI development that covers basic inference through to conversational systems. You can run chat completions, generate embeddings for semantic search, and use safety features through Llama Guard, all locally.

Deploying AI agents with the Llama Stack is particularly valuable for AI hackathons, where participants may want to run AI models locally without relying on external APIs or cloud services. Whether you're participating in online AI hackathons or virtual AI hackathons, understanding how to deploy and optimize AI agents locally can help you build AI hackathon projects. If you're looking for upcoming AI hackathons to apply these skills, see the upcoming AI hackathons.

Before we write any code, we need to get access to the models. Head over to Meta's download page at llama.com/llama-downloads, where you fill out a request form.

Fill out the details and select the models you want access to. For this guide, we'll use Llama3.2-8B as named in the commands below, a middle ground between performance and resource usage. Once approved, Meta will send you download URLs for each model.

Let's set up your environment and get that model downloaded:

conda create -n llama python=3.10
conda activate llama
pip install llama-stack

Now we can download the model. Replace <META_URL> with the URL you received:

llama download --source meta --model-id Llama3.2-8B --meta-url <META_URL>

The download will take a few minutes and shows a progress bar. The model gets stored in your ~/.llama directory, and checksums are printed at the end to confirm you have an intact copy.

Now that our model is downloaded, let's build our first AI server. Llama Stack uses a build-configure-run workflow.

First, let's create your distribution. This step tells Llama Stack what components you want in your server:

llama stack build

The build process will walk you through setting up your server. Name it something simple like "my-local-stack" and choose "conda" for the image type. The prompts look like this:

> Enter an unique name for identifying your Llama Stack build distribution: my-local-stack
> Enter the image type you want your distribution to be built with: conda

Llama Stack is composed of several APIs working together. Let's configure the providers:
> Enter the API provider for the inference API: meta-reference
> Enter the API provider for the safety API: meta-reference
> Enter the API provider for the agents API: meta-reference
> Enter the API provider for the memory API: meta-reference
> Enter the API provider for the telemetry API: meta-reference

The defaults (meta-reference) are what we use here. They are Meta's reference implementations that handle running the model and managing conversations.

Next, configure your server:

llama stack configure my-local-stack

This is where we tell our server how to run. Start with the inference settings.

Keep the defaults for most settings, but make sure to point it to our Llama3.2-8B model. A sequence length of 4096 leaves room for context, and a batch size of 1 keeps things simple while getting started.

The safety settings are optional. You can skip them for initial testing, but they are worth exploring later if you build something for production.

Now that we've configured everything, let's start our server:

llama stack run my-local-stack

When you run this command, your Llama Stack server initializes its core components. Note that the client examples below connect to http://localhost:5000; keep the server bound to your own machine and do not expose it to the internet without access control.

Understanding the Server Architecture

The server exposes several key endpoints:

  • /inference/chat_completion for text generation and conversations
  • /inference/embeddings for vector representations
  • /memory_banks/* for conversation state management
  • /agentic_system/* for complex reasoning tasks

Let's explore how to interact with these endpoints using the Llama Stack Client:

from llama_stack_client import LlamaStackClient

# Initialize the client
client = LlamaStackClient(base_url="http://localhost:5000")

# Basic inference example
response = client.inference.chat_completion(
    messages=[{
        "role": "user",
        "content": "Explain how neural networks process data."
    }],
    model="Llama3.2-8B"
)

print(response.choices[0].message.content)

Working with Conversation Memory

The Memory API enables contextual awareness across multiple interactions:

# Create a conversation session
conversation_id = client.memory.create_conversation()

# Initial context
response1 = client.inference.chat_completion(
    messages=[{
        "role": "user",
        "content": "Let's discuss machine learning algorithms."
    }],
    conversation_id=conversation_id
)

# Follow-up with context
response2 = client.inference.chat_completion(
    messages=[{
        "role": "user",
        "content": "What are their practical applications?"
    }],
    conversation_id=conversation_id
)

Implementing Safety Controls

For production deployments, enable content filtering and security features:

# Configure safety settings
client.safety.configure(
    content_filtering=True,
    prompt_injection_detection=True
)

response = client.inference.chat_completion_safe(
    messages=[{
        "role": "user",
        "content": "Explain database security."
    }],
    safety_level="medium"
)

Parallel Processing

Handle multiple requests efficiently with batch processing:

responses = client.inference.batch_completion(
    prompts=[
        "Explain REST APIs.",
        "Describe microservices.",
        "Define cloud computing."
    ],
    batch_size=3
)

for response in responses:
    print(response.choices[0].message.content)

Error Management

Implement error handling for production stability:

from llama_stack_client.exceptions import ModelOverloadError, TokenLimitError

try:
    response = client.inference.chat_completion(
        messages=[{
            "role": "user",
            "content": extensive_text
        }]
    )
except TokenLimitError:
    # Handle token limit exceeded
    truncated_content = truncate_text(extensive_text)
    response = retry_with_truncated_content(truncated_content)
except ModelOverloadError:
    # Implement backoff strategy
    response = retry_with_backoff()

Performance Monitoring

Enable logging and metrics collection:

# Configure monitoring
client.configure_logging(
    log_level="DEBUG",
    log_file="llama_stack.log"
)

client.metrics.track_performance(
    track_latency=True,
    track_token_usage=True,
    track_memory_usage=True
)

On startup, the server logs show how Llama Stack orchestrates its components.

The server sets up multiple processing pipelines to handle concurrent requests. The logs show it initializing model parallelism, distributed data processing (DDP), and the inference pipeline.

Once the model loads, the server exposes its API endpoints.

Each endpoint covers a capability:

  • /inference/chat_completion handles conversational AI
  • /inference/embeddings generates vector representations for semantic search
  • /memory_banks/* endpoints manage conversation history and context
  • /agentic_system/* enables complex multi-step reasoning

In the server logs you can see how requests flow through the system. 200 OK responses mean successful requests. Each request gets processed through the inference pipeline we configured, with the model generating responses in real-time.

This setup is flexible. Whether you're building a chatbot, a document analysis system, or a complex AI agent, these endpoints give you building blocks.


Now that we have our server running, let’s explore how to interact with it using the Llama Stack Client Python library. This library provides a convenient way to access the Llama Stack REST API from any Python application, making it easy to integrate AI capabilities into your projects.

Installing the Client

To get started, you’ll need to install the Llama Stack Client. You can do this easily with pip:

pip install llama-stack-client

Basic Usage

Once installed, you can use the client to interact with your Llama Stack server. Here’s a simple example to get you started:

from llama_stack_client import LlamaStackClient
from llama_stack_client.types import UserMessage

client = LlamaStackClient(
    base_url="http://localhost:5000",
)

response = client.inference.chat_completion(
    messages=[
        UserMessage(
            content="Hello world, write me a 2 sentence poem about the moon.",
            role="user",
        ),
    ],
    model="Llama3.2-8B",
    stream=False,
)

print(response)

This code snippet initializes the client and sends a message to the chat completion endpoint. The response will contain the model's generated text, which you can then use in your application.

Asynchronous Usage

If you prefer asynchronous programming, the library also supports it. Simply import AsyncLlamaStackClient and use await for your API calls:

import asyncio
from llama_stack_client import AsyncLlamaStackClient

client = AsyncLlamaStackClient()

async def main():
    response = await client.inference.chat_completion(
        messages=[
            UserMessage(
                content="Tell me a joke.",
                role="user",
            ),
        ],
        model="Llama3.2-8B",
    )
    print(response)

asyncio.run(main())

Error Handling

The library includes error handling. If the client cannot connect to the API or if the API returns an error, you can catch these exceptions easily:

from llama_stack_client import LlamaStackClient, APIConnectionError, APIStatusError

client = LlamaStackClient()

try:
    response = client.inference.chat_completion(
        messages=[UserMessage(content="What's the weather like?", role="user")],
        model="Llama3.2-8B",
    )
except APIConnectionError:
    print("Could not connect to the server.")
except APIStatusError as e:
    print(f"API error: {e.status_code} - {e.response}")

Advanced Features

The client also supports advanced features like logging, retries, and timeouts. You can enable logging by setting the environment variable:

export LLAMA_STACK_CLIENT_LOG=debug

For requests that may take longer, you can configure timeouts:

client = LlamaStackClient(timeout=20.0)  # 20 seconds timeout

Conclusion

With the Llama Stack Client, you've seen the fundamentals of running AI models locally. This tutorial covered the basics. For more detail on the APIs covered here, see the official documentation.

Use it at an AI hackathon

Running a model locally means your project does not depend on an external API during a short event, which can help in an AI hackathon. Local chatbots, semantic search and document analysis are realistic ideas to build on this setup. To find an event, see the upcoming AI hackathons.

Frequently asked questions

How can I use Llama Stack for AI agents in an AI hackathon?

You can run AI models locally without depending on external APIs, and build projects such as local chatbots, semantic search engines, document analysis systems, or multi-modal AI assistants.

Is it suitable for beginners?

It requires some technical knowledge of Python, command-line tools, and AI model concepts. The tutorial walks through the process step by step and suits those with intermediate programming skills.

Are there limitations in time-limited hackathons?

Yes. Downloading large model files can take significant time and bandwidth, local setup needs hardware with enough RAM and storage, and debugging local deployment issues can be time-consuming.

How this tutorial was tested

This content was carried over from an earlier tutorial and updated for Starfield Labs. It was not re-run in this build, and no step has been confirmed to work as written. To follow it you would need a Meta model-access request and download URL (accepted licence terms), conda, Python 3.10, enough disk, RAM and likely a GPU to run the model locally, and the llama-stack and llama-stack-client packages. No account, download or model was used here.