Deploying Production-Grade Multi-Agent Systems with Google ADK on Vertex AI

Formare AI builds, deploys, and scales enterprise multi-agent systems on Google Cloud Platform. By integrating Google ADK with Vertex AI and Cloud Run, these orchestrated agents automate complex, multi-step workflows that are strictly GDPR and EU AI Act compliant. Production deployments reduce manual operational cycle times from hours to minutes while maintaining full audit trails.

Industry research shows up to 74% of AI initiatives fail to move from prototype to production. The reasons are almost never about model quality. They are about agent orchestration, tool reliability, observability, and cost predictability — the infrastructure problems that only surface once you leave the notebook.

Why Most Agent Prototypes Fail in Production

The most common failure pattern: a developer builds an agent using simple function calls and an in-process session store. It works perfectly in a demo. Then it hits production and fails within 48 hours. The reasons are predictable: tool calls that time out without retry logic, session state that disappears on Cloud Run restarts, no visibility into why a specific agent call returned a wrong result, and compute costs that scale quadratically with conversation length because every turn re-processes the full history.

Tool calls that time out silently — agent reports success, downstream system receives stale data
In-memory session state lost on pod restart — user conversation context wiped without warning
No distributed tracing — impossible to identify which sub-agent produced a wrong result
Token costs that compound with conversation length — a 20-turn session costs 40× more than a 1-turn session without caching
No circuit breakers on external APIs — one downstream timeout blocks the entire agent graph

The Production Architecture: Four Layers

Layer 1 — Entry & Routing
  Cloud Run (HTTPS) → FastAPI + ADK get_fast_api_app()
  ↓ Session ID extracted from header → DatabaseSessionService

Layer 2 — Orchestration
  Root Orchestrator (Gemini 2.5 Pro)
  ├── Intent classification → routes to specialist agent
  ├── Sub-task decomposition → parallel agent calls
  └── Result aggregation → structured output to caller

Layer 3 — Specialist Agents (all on ADK LlmAgent)
  ├── DataAgent      → BigQuery read/write, schema-aware SQL
  ├── ForecastAgent  → Vertex AI custom models, batch prediction
  ├── PolicyAgent    → RAG over internal documents (Vertex AI Search)
  └── ActionAgent    → External API calls (ERP, CRM, messaging)

Layer 4 — Infrastructure
  ├── Session persistence  → Cloud SQL (PostgreSQL)
  ├── Tool results cache   → Cloud Memorystore (Redis, TTL-based)
  ├── Audit log            → BigQuery (every tool call + result)
  └── Observability        → Cloud Trace + custom ADK callbacks

Persistent Sessions with Cloud SQL

The single most important production decision is session persistence. ADK's built-in DatabaseSessionService supports Cloud SQL out of the box. Never use InMemorySessionService in production — a Cloud Run pod restart (which happens on every new deployment and periodically for scaling) wipes every active conversation:

python

1import os
2from google.adk.cli.fast_api import get_fast_api_app
3from google.adk.sessions import DatabaseSessionService
4
5DATABASE_URL = os.environ["DATABASE_URL"]
6# Example: postgresql+pg8000://user:pass@/dbname?unix_sock=/cloudsql/proj:region:instance
7
8session_service = DatabaseSessionService(db_url=DATABASE_URL)
9
10app = get_fast_api_app(
11    agent=root_orchestrator,
12    session_service=session_service,
13    allow_origins=["https://your-frontend.com"],
14)
15
16# Cloud Run deployment — sessions now survive pod restarts
17# Each request attaches session_id from X-Session-ID header
18# ADK reconstructs full conversation context automatically

Tool Reliability: Retry Logic and Circuit Breakers

Every production tool that calls an external API needs three things: a timeout, exponential backoff retry logic, and a circuit breaker that opens after repeated failures. ADK FunctionTools wrap Python functions directly — add reliability at the function level so the agent never knows a retry happened:

python

1import time
2from functools import wraps
3from google.adk.tools import FunctionTool
4
5def with_retry(max_attempts: int = 3, backoff_base: float = 1.5):
6    """Decorator: exponential backoff retry with circuit breaker state."""
7    def decorator(fn):
8        fn._failure_count = 0
9        fn._circuit_open_until = 0.0
10
11        @wraps(fn)
12        def wrapper(*args, **kwargs):
13            # Circuit breaker: fail fast if recently broken
14            if time.time() < fn._circuit_open_until:
15                raise RuntimeError(f"Circuit open for {fn.__name__} — try again later")
16
17            for attempt in range(max_attempts):
18                try:
19                    result = fn(*args, **kwargs)
20                    fn._failure_count = 0  # reset on success
21                    return result
22                except Exception as e:
23                    fn._failure_count += 1
24                    if fn._failure_count >= 5:
25                        fn._circuit_open_until = time.time() + 60  # 60s open window
26                    if attempt == max_attempts - 1:
27                        raise
28                    time.sleep(backoff_base ** attempt)
29        return wrapper
30    return decorator
31
32@with_retry(max_attempts=3)
33def query_erp_system(order_id: str) -> dict:
34    """Tool: fetch order details from ERP. Retries 3× with backoff."""
35    # ... actual implementation
36    pass
37
38erp_tool = FunctionTool(query_erp_system)

Distributed Observability: Tracing Every Agent Call

When an agent returns a wrong result in production, you need to know exactly which sub-agent made which tool call, what input it received, and what it returned. ADK provides event callbacks for this. We pipe every event into BigQuery as an append-only audit log — immutable, queryable, and GDPR-compliant:

python

1from google.adk.agents import BaseAgent
2from google.adk.events import Event
3from google.cloud import bigquery
4import datetime, json
5
6BQ_TABLE = "project.audit.agent_events"
7bq_client = bigquery.Client()
8
9class AuditCallbackAgent(BaseAgent):
10    """Wraps any ADK agent to log every event to BigQuery."""
11
12    async def _run_async_impl(self, ctx):
13        async for event in self.sub_agent._run_async_impl(ctx):
14            # Log every event — tool call, tool result, model response
15            row = {
16                "timestamp":    datetime.datetime.utcnow().isoformat(),
17                "session_id":   ctx.session.id,
18                "user_id":      ctx.session.user_id,
19                "agent_name":   event.author,
20                "event_type":   event.content.parts[0].__class__.__name__,
21                "event_payload": json.dumps(
22                    event.content.parts[0].to_dict()
23                    if hasattr(event.content.parts[0], "to_dict") else {}
24                ),
25            }
26            bq_client.insert_rows_json(BQ_TABLE, [row])  # streaming insert
27            yield event  # pass event through unchanged
28
29# Wrap root orchestrator transparently
30audited_root = AuditCallbackAgent(
31    name="audited_root",
32    sub_agent=root_orchestrator,
33)

Token Cost Control: Caching and Context Windowing

A multi-agent conversation with 20 turns and 5 tools in context can easily hit 50,000 input tokens per turn. At Gemini 2.5 Pro pricing, that compounds fast. Two controls make the difference: tool result caching in Redis (identical tool calls within a session return cached results without re-invoking), and context windowing (only the last N turns are included in the model's working context):

python

1import redis, hashlib, json, os
2
3cache = redis.from_url(os.environ["REDIS_URL"])
4CACHE_TTL_SECONDS = 300  # 5-minute TTL on tool results
5
6def cached_tool(fn):
7    """Cache tool results in Redis. Same inputs → cached result for 5 min."""
8    @wraps(fn)
9    def wrapper(**kwargs):
10        cache_key = f"tool:{fn.__name__}:{hashlib.md5(json.dumps(kwargs, sort_keys=True).encode()).hexdigest()}"
11        cached = cache.get(cache_key)
12        if cached:
13            return json.loads(cached)
14        result = fn(**kwargs)
15        cache.setex(cache_key, CACHE_TTL_SECONDS, json.dumps(result))
16        return result
17    return wrapper
18
19@cached_tool
20def get_product_details(sku_id: str) -> dict:
21    """Fetches product from BigQuery. Cached 5 min per SKU."""
22    # ... BigQuery query
23    pass
24
25# Context windowing: limit history to last 10 turns
26# Configured in ADK session policy — not shown here for brevity

A2A Protocol: Inter-Agent Communication at Scale

For complex enterprise deployments where different agent systems need to collaborate — for instance, a demand forecasting agent calling a procurement planning agent run by a different team — Google's A2A (Agent-to-Agent) protocol provides a standardized, discoverable interface. Each agent exposes an Agent Card at /.well-known/agent.json describing its capabilities, and other agents discover and call it over HTTP:

python

1# agent_card.py — served at /.well-known/agent.json
2AGENT_CARD = {
3    "name": "Demand Forecasting Agent",
4    "description": "Generates 4-week demand forecasts for retail SKUs using Chronos-2 + LightGBM ensemble on Vertex AI",
5    "version": "1.0.0",
6    "url": "https://forecast-agent-xxx.run.app",
7    "capabilities": {
8        "streaming": False,
9        "pushNotifications": False,
10    },
11    "skills": [
12        {
13            "id": "forecast_sku",
14            "name": "Forecast SKU demand",
15            "description": "Returns 4-week probabilistic demand forecast for a given SKU and store combination",
16            "inputModes": ["application/json"],
17            "outputModes": ["application/json"],
18        }
19    ],
20}
21
22# Other agents discover this via HTTP GET /.well-known/agent.json
23# and call the /tasks/send endpoint using the A2A task schema

Concurrent specialist agents in production (enterprise retail): 5+
AI projects that fail the lab-to-production transition (industry avg): 74%
Median tool result served from Redis cache (vs 4.2s cold): 300ms
Agent events logged to BigQuery for audit trail: 100%

Deployment: Cloud Run Configuration for Agent Workloads

Agent workloads have different resource profiles from standard web APIs: longer request timeouts (conversations can span minutes), higher memory (model context requires significant working memory), and concurrency limits that prevent token cost explosions. Use these Cloud Run settings for ADK-based agents:

bash

1gcloud run deploy forecast-agent \
2  --image gcr.io/your-project/forecast-agent:latest \
3  --region europe-west1 \
4  --platform managed \
5  --memory 4Gi \
6  --cpu 2 \
7  --timeout 900 \          # 15-minute max for long-running conversations
8  --concurrency 10 \       # Limit parallel requests per instance
9  --max-instances 20 \     # Cap total scale for cost control
10  --set-env-vars DATABASE_URL=...,REDIS_URL=...,GOOGLE_CLOUD_PROJECT=... \
11  --set-secrets GEMINI_API_KEY=gemini-api-key:latest \
12  --service-account agent-runtime@your-project.iam.gserviceaccount.com \
13  --no-allow-unauthenticated  # Internal agents: require IAM auth

What "Production-Grade" Actually Means

A production-grade multi-agent system is not defined by the model it uses or the number of tools it has. It is defined by what happens when things go wrong. Does a tool timeout propagate gracefully or cascade into a full system failure? Does a Cloud Run restart lose an active user session? Can you answer the question "why did the agent recommend X on Tuesday at 14:32?" six months later? If the answer to any of these is no, the system is a prototype running in production — which is the most dangerous state of all.