Building a Multi-Agent Restaurant System with AutoGen

Building a Multi-Agent Restaurant System with AutoGen

Imagine ordering pizza by simply chatting with an AI assistant. No more scrolling through endless menus or navigating complex websites. With the power of multi-agent systems and Large Language Models (LLMs), this is becoming a reality. In this article, we’ll dive into AutoGen’s AgentChat message system and demonstrate its capabilities by creating a fully functional pizza ordering system.

Understanding AutoGen Framework

AutoGen is a comprehensive ecosystem for building AI agents and applications, particularly suited for multi-agent workflows. At its core, AutoGen uses a layered and extensible design that lets you work at different levels of abstraction.

Key Components:

• Core API: The foundation, providing message passing and event-driven agents
• AgentChat API: A higher-level API for building conversational agents
• Extensions API: For integrating with services like Azure OpenAI
• Developer Tools:
• AutoGen Studio: Visual development environment
• AutoGen Bench: Performance evaluation tools

Want to explore more? Check out the AutoGen documentation and join their Discord community for updates and support.

Why AgentChat for Our Restaurant System?

For our restaurant ordering system, we’ll use AgentChat, AutoGen’s high-level API designed specifically for building conversational multi-agent applications. Here’s why:

• Simplified Development: Perfect for building conversational flows
• Pre-built Agent Templates: Ready-to-use agent types for different roles
• Built-in Message Types: Supports text and multimodal communication
• Team Patterns: Easy to implement agent collaboration

Understanding AgentChat Messages

Think of AgentChat messages as the language our AI agents use to communicate. Just like in a real restaurant where staff members need different ways to communicate (talking, showing pictures of dishes, updating order status), our digital agents need various message types to work effectively together.

Types of Messages

At a high level, messages in AgentChat can be categorized into two types:

1. Agent-to-Agent Messages

TextMessage: The Basic Conversation

from autogen_agentchat.messages import TextMessage
# Customer placing an order
order = TextMessage(
    content="I would like to order a Margherita pizza",
    source="customer"
)

This is the simplest form of communication — plain text with a source identifier. It’s perfect for:

• Customer orders
• Order confirmations
• Status updates
• Simple notifications

MultiModalMessage: Show and Tell

from autogen_agentchat.messages import MultiModalMessage
from autogen_core import Image as AGImage
# Menu showing a pizza with description
menu_item = MultiModalMessage(
    content=[
        "Here's our Margherita Pizza with fresh basil",
        pizza_image  # AGImage object
    ],
    source="menu_agent"
)

MultiModalMessage combines text and visual elements, ideal for:

• Displaying menu items with images
• Showing order confirmations with visuals
• Providing rich status updates
• Enhanced user experience

2. Internal Events: Behind the Scenes

# Kitchen process events
events = [
    ("ToolCallRequestEvent", "Order received"),
    ("ToolCallExecutionEvent", "Preparation started"),
    ("ToolCallExecutionEvent", "Cooking in progress")
]

Building Our Restaurant System

Now that we understand how agents communicate, let’s build our restaurant ordering system. We’ll create two main agents: a MenuAgent to handle orders and a CookAgent to manage food preparation.

Disclaimer : This article demonstrates a basic multi-agent system specifically designed to explain AutoGen AgentChat’s message capabilities. For simplicity, we’ve limited our menu to just pizzas and simplified many real-world restaurant operations. The focus is on understanding how different agents communicate using various message types in AutoGen, rather than building a complete restaurant management system.

System Design

Before diving into the implementation, let’s understand how our agents will interact with each other. We’ll use two diagrams to visualize our system:

High-Level Architecture

The following block diagram shows how different components of our system interact:

Our system consists of:

• MenuAgent: Handles menu display and order taking
• CookAgent: Manages food preparation and status updates
• LLM: Provides natural language understanding

Order Flow

The sequence diagram below details the step-by-step interaction between agents:

Now that we understand our system architecture, let’s implement each component.

Implementation

Let’s build our restaurant system step by step. We’ll implement each component following our system design.

Step 1: Environment Setup

First, let’s install required packages and set up our environment:

# Install required packages
pip install -U "autogen-agentchat" "autogen-ext[openai,azure]"
pip install python-dotenv Pillow

Create a .env file with your Azure OpenAI credentials:

AZURE_OPENAI_API_KEY=your_api_key
AZURE_OPENAI_ENDPOINT=your_endpoint
AZURE_CHATGPT_MODEL=your_model_name
AZURE_OPENAI_API_VERSION=your_api_version

Step 2: Base Configuration

Set up our basic configuration and imports:

import os
from dotenv import load_dotenv
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.messages import TextMessage, MultiModalMessage
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_core import Image as AGImage
from PIL import Image
from io import BytesIO
import requests
import random
from typing import List
from IPython.display import display


# Load environment variables
load_dotenv()

# Create the OpenAI model client
model_client = OpenAIChatCompletionClient(
    model=os.getenv("AZURE_CHATGPT_MODEL"),
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),
    api_base=os.getenv("AZURE_OPENAI_ENDPOINT"),
    api_type="azure",
    api_version=os.getenv("AZURE_OPENAI_API_VERSION")
)

Step 3: Menu Data Structure

Define our menu items:

# Menu Data with Image URLs
MENU_DATA = {
    "menu_items": {
        "margherita": {
            "id": "pizza_001",
            "name": "Margherita Pizza",
            "description": "Classic pizza with tomato sauce, mozzarella, and basil",
            "price": 12.99,
            "image_url": "https://images.unsplash.com/photo-1513104890138-7c749659a591"
        },
        "pepperoni": {
            "id": "pizza_002",
            "name": "Pepperoni Pizza",
            "description": "Traditional pizza topped with pepperoni slices",
            "price": 14.99,
            "image_url": "https://images.unsplash.com/photo-1601924582970-9238bcb495d9"
        }
    }
}

Step 4: MenuAgent Implementation

Create our MenuAgent to handle menu interactions:

# Define the MenuAgent class
class MenuAgent(AssistantAgent):
    def __init__(self, name: str):
        super().__init__(
            name=name,
            model_client=model_client,
            system_message="You are a helpful menu assistant who provides details about menu items and helps customers with their food orders."
        )
        self.menu_data = MENU_DATA

    def is_valid_item(self, item_name: str) -> bool:
        """Check if the item exists in the menu."""
        return item_name.lower() in self.menu_data["menu_items"]

    def get_available_items(self) -> str:
        """Return a comma-separated list of available menu items."""
        return ", ".join(self.menu_data["menu_items"].keys())

    async def get_menu_item(self, item_name: str) -> MultiModalMessage:
        """Fetch menu item details and return a formatted response."""
        if not self.is_valid_item(item_name):
            return TextMessage(
                content=f"Sorry, {item_name} is not available. Here's our menu: {self.get_available_items()}",
                source=self.name
            )
        
        item = self.menu_data["menu_items"][item_name.lower()]
        response = requests.get(item["image_url"])
        pil_image = Image.open(BytesIO(response.content))

        # Resize the image for consistency
        resized_image = pil_image.resize((200, int(200 * (pil_image.height / pil_image.width))))
        ag_image = AGImage(resized_image)

        # Description
        description = (
            f"Here's our {item['name']}:\n"
            f"{item['description']}\n"
            f"Price: ${item['price']}\n\n"
            f"Would you like to place an order for this {item['name']}? 🍕"
        )

        # Display image inline in Jupyter
        display(resized_image)

        return MultiModalMessage(
            content=[description, ag_image],
            source=self.name
        )

    async def confirm_order(self, item_name: str) -> TextMessage:
        """Send confirmation message with estimated preparation time."""
        cooking_time = random.randint(15, 25)
        return TextMessage(
            content=f"Thank you for confirming your order for {item_name.capitalize()}! "
                   f"Your order will be ready in approximately {cooking_time} minutes. "
                   f"Our chef will start preparing it right away.",
            source=self.name
        )

Step 5: CookAgent Implementation

Create our CookAgent to handle order processing:

# Define the CookAgent class
class CookAgent(AssistantAgent):
    def __init__(self, name: str):
        super().__init__(
            name=name,
            model_client=model_client,
            system_message="You are a professional chef who provides updates about the cooking process and completes orders."
        )

    async def process_order(self, order: TextMessage) -> List[TextMessage]:
        """Simulate the cooking process and return status updates."""
        events = [
            ("ToolCallRequestEvent", "Order received and validated"),
            ("ToolCallExecutionEvent", "Starting preparation"),
            ("ToolCallExecutionEvent", "Cooking in progress"),
            ("ToolCallExecutionEvent", "Order completed")
        ]
        return [TextMessage(content=f"{event_type}: {status}", source=self.name) for event_type, status in events]

    async def send_completion_notice(self) -> TextMessage:
        """Notify the customer that the order is ready."""
        return TextMessage(
            content="🎉 Your order is ready! Please pick it up from the counter. Enjoy your meal! 🍕",
            source=self.name
        )

Each step builds upon the previous one, creating a complete system. Next, we’ll see how to run this system and handle different scenarios.

Step 6: Running the System

Now that we have our agents implemented, let’s see how they work together. We’ll create a main function that orchestrates the entire order process:

# Main Execution Flow
async def main():
    print("\n=== Multi-Agent Restaurant Order System ===\n")
    menu_agent = MenuAgent(name="menu_agent")
    cook_agent = CookAgent(name="cook_agent")

    # Step 1: Customer places an order
    order_item = "pepperoni"
    customer_request = TextMessage(content=f"I would like to order {order_item}", source="customer")
    print("📱 Customer:", customer_request.content, "\n")

    # Step 2: MenuAgent responds
    menu_response = await menu_agent.get_menu_item(order_item)
    if isinstance(menu_response, TextMessage):
        print(f"🍽️ MenuAgent: {menu_response.content}")
        return
    print(f"🍽️ MenuAgent: {menu_response.content[0]}", "\n")

    # Step 3: Customer confirms the order
    confirmation_response = await menu_agent.confirm_order(order_item)
    print("📱 Customer: Yes, I confirm my order." "\n")
    print("🍽️ MenuAgent:", confirmation_response.content, "\n")

    # Step 4: CookAgent processes the order
    print("👨‍🍳 Kitchen Updates:")
    status_updates = await cook_agent.process_order(customer_request)
    for update in status_updates:
        print(update.content)
    print()

    # Step 5: Notify order completion
    completion_notice = await cook_agent.send_completion_notice()
    print("👨‍🍳 CookAgent:", completion_notice.content)

# Use this block if you are running the script in a Jupyter Notebook
if __name__ == "__main__":
    await main()

# Use this block if you are running the script in a terminal (standalone Python environment)
# Uncomment the following lines if you're running in the terminal:
# if __name__ == "__main__":
#     import asyncio
#     asyncio.run(main())

Sample Output

When you run this code with a valid menu item (“margherita”), you’ll see output like this:

valid menu option

P.S. If you are running this code in a terminal, you will see a placeholder for the image (e.g., <PIL.Image.Image image mode=RGB size=200x299 at 0x112DED550>). However, in Jupyter Notebook, the actual image will be displayed inline, providing a richer and more visual experience.

If you try with an invalid item (e.g., “chicken”), you’ll see:

invalid menu option

Try running it with different menu items to see how the system handles various scenarios!

Real-World Applications Beyond Pizza

The message types and patterns we used in our restaurant system can be applied to various real-world scenarios:

Customer Support

• Technical Support Bot: Uses TextMessage for queries and MultiModalMessage for sharing screenshots/guides
• Automated Ticket System: Internal events track ticket status and escalation flows

E-commerce

• Shopping Assistant: Combines product images and descriptions using MultiModalMessage
• Order Management: Tracks order status using internal events, just like our kitchen updates

Healthcare

• Appointment Scheduler: Uses TextMessage for booking and confirmation
• Patient Monitoring: Internal events track patient status and follow-ups

The complete code for this article is available on our GitHub repository.

Resources

https://microsoft.github.io/autogen/stable/index.html
https://microsoft.github.io/autogen/stable/user-guide/agentchat-user-guide/quickstart.html

About the Author: I’m Jay Thakur, a Senior Software Engineer at Microsoft, exploring the transformative potential of AI Agents. With over 8 years of experience building and scaling AI solutions at Amazon, Accenture Labs, and now Microsoft, combined with my studies at Stanford GSB, I bring a unique perspective to the intersection of tech and business. I’m dedicated to making AI accessible to all — from beginners to experts — with a focus on building impactful products. As a speaker and aspiring startup advisor, I share insights on AI Agents, GenAI, LLMs, SMLs, responsible AI, and the evolving AI landscape. Connect with me on Linkedin

Did you find this article helpful? Don’t forget to clap 👏 and follow for more content like this!