AI Agents10 min readBy Petrică RădanReference guide
Multi-Tenant AI Architecture with Gemini, AlloyDB and Cloud Run
Learn how to isolate tenant vector embeddings in AlloyDB with Row-Level Security and enforce per-tenant Gemini token quotas on Cloud Run
Building a multi-tenant AI architecture GCP engineers can run safely in production requires pairing stateless Cloud Run compute with AlloyDB Row-Level Security (RLS) for vector retrieval and two-phase application rate limiting for Vertex AI Gemini quotas. You enforce tenant isolation by extracting authenticated tenant claims from incoming JWTs in Cloud Run, binding them to transaction-scoped PostgreSQL session variables via SET LOCAL, and executing filtered Approximate Nearest Neighbor (ANN) queries over AlloyDB ScaNN indexes while tracking per-tenant token consumption before and after LLM generation.
Without strict database-level isolation and application-layer quota governance, B2B generative AI platforms fail in two predictable ways: cross-tenant data leakage during retrieval-augmented generation (RAG), and noisy-neighbor exhaustion of project-wide Vertex AI rate limits. Relying on application developers to append WHERE clauses to every vector similarity query introduces unacceptable security risk, while trusting cloud-level quota metrics to throttle individual tenants fails due to metric sampling delays. Solving both problems requires coordinating identity propagation, connection pooling, vector index execution plans, and real-time token ledgers across your stack.
What does a production multi-tenant AI architecture on GCP look like?
A production Cloud Run multi-tenant SaaS architecture separates ingress routing, stateless AI orchestration, vector storage, and model inference across VPC-isolated boundaries, choosing between pooled multi-tenancy or a hub-and-spoke topology based on tenant compliance requirements. Most B2B SaaS platforms use a pooled model where shared Cloud Run services enforce logical tenant isolation inside a shared AlloyDB cluster and shared Vertex AI project.
For regulated enterprise workloads requiring strict network and resource isolation, Google Cloud's reference architecture for multi-tenant agentic AI systems uses a hub-and-spoke topology inside a VPC Service Controls perimeter (Multi-Tenant Agentic AI on Google Cloud: A Production Walkthrough, 2026). This pattern pairs an External Application Load Balancer and Identity-Aware Proxy at the routing hub with isolated tenant spokes holding AlloyDB for PostgreSQL and per-tenant Gemini configurations. As detailed in the Sakura Sky Blog (2026), Google Cloud published a reference architecture and Terraform module for this design where the routing hub dispatches authenticated requests via service-account-signed ID tokens to tenant Cloud Run agents backed by AlloyDB and Model Armor.
In high-density B2B SaaS where provisioning dedicated AlloyDB clusters per customer is cost-prohibitive, pooled multi-tenancy consolidates tenants onto shared infrastructure. For example, SOCRadar integrated the Gemini Enterprise Agent Platform with AlloyDB for threat detection and AI-native alert filtering, reclaiming over 45 TB of storage, reducing DBA maintenance overhead by 75%, and running analytical queries up to 20x faster than standard PostgreSQL, according to Google Cloud Blog (2026). Similarly, an enterprise 2-tier agent memory architecture documented by Google Cloud Blog (2026) combines Memorystore and AlloyDB with Gemini 3.5 Flash, enforcing multi-tenant isolation across departments and users by indexing user_id, project_id, and scope columns alongside PostgreSQL Row-Level Security (RLS) and Parameterized Secure Views (PSV).
In a pooled architecture, every request follows a deterministic execution path:
- An API gateway (such as Apigee X or Cloud Load Balancing) validates TLS and routes traffic to Cloud Run. In Apigee X, semantic vector caching intercepts repetitive LLM queries using Vector Search to deliver responses in under 50ms at zero token cost, according to Google Developer Forums (2026).
- Cloud Run middleware validates the caller's JWT, extracts the tenant UUID, and checks an application-layer token ledger (typically hosted in Memorystore for Redis) before permitting execution.
- The service opens a database transaction through AlloyDB Managed Connection Pooling, sets the transaction-local tenant context, and executes a vector similarity query to retrieve tenant-specific context chunks.
- Cloud Run invokes Vertex AI Gemini with the retrieved context, captures streaming token usage metadata, and reconciles the tenant's quota balance.
How do you inject JWT tenant context in Cloud Run without connection pool leakage?
Implementing multi-tenant AI architecture GCP Cloud Run middleware for JWT tenant context injection requires validating cryptographic token claims at the HTTP boundary and propagating the tenant ID to AlloyDB exclusively via PostgreSQL's SET LOCAL command inside an explicit database transaction. Using session-level SET commands in pooled environments leaks tenant context across concurrent requests.
Serverless containers on Cloud Run scale rapidly, opening hundreds of concurrent database connections during traffic spikes. To prevent connection exhaustion, AlloyDB provides built-in Managed Connection Pooling (based on PgBouncer in transaction pooling mode). According to Google Cloud - Medium (2026), AlloyDB's Managed Connection Pooling handles up to 5x higher connection throughput for multi-tenant workloads than standard PostgreSQL, allowing applications to pack more tenants per instance without hitting connection bloat limits, while its Columnar Engine executes tenant-wide analytical aggregations up to 100x faster without impacting disk-based transactional rows.
Never execute a session-scoped SET app.tenant_id command when connecting through transaction-mode connection pools like PgBouncer or AlloyDB Managed Connection Pooling. Because the pooler reassigns physical PostgreSQL backend connections to different clients at the end of every transaction, session variables persist and expose one tenant's vector embeddings to the next tenant's query. Always wrap queries in an explicit BEGIN...COMMIT block and use SET LOCAL.
When using SET LOCAL, PostgreSQL binds the configuration parameter strictly to the current transaction lifecycle and resets it automatically upon COMMIT or ROLLBACK, even if the application crashes mid-stream. Below is a reference Python middleware and database dependency pattern using FastAPI and psycopg (async) on Cloud Run:
1from contextlib import asynccontextmanager
2from fastapi import FastAPI, Request, HTTPException, Depends
3import jwt
4import psycopg_pool
5
6app = FastAPI()
7# Connect to AlloyDB Auth Proxy / Managed Connection Pooler
8db_pool = psycopg_pool.AsyncConnectionPool(conninfo="postgresql://app_user@127.0.0.1:6432/rag_db", open=False)
9
10async def extract_tenant_id(request: Request) -> str:
11 auth_header = request.headers.get("Authorization")
12 if not auth_header or not auth_header.startswith("Bearer "):
13 raise HTTPException(status_code=401, detail="Missing bearer token")
14 token = auth_header.split(" ")[1]
15 try:
16 # Validate signature, audience, and expiration against your IdP JWKS
17 payload = jwt.decode(token, key=PUBLIC_JWKS_KEY, algorithms=["RS256"], audience="api.example.com")
18 tenant_id = payload.get("tenant_id")
19 if not tenant_id:
20 raise HTTPException(status_code=403, detail="Token missing tenant_id claim")
21 return tenant_id
22 except jwt.PyJWTError as exc:
23 raise HTTPException(status_code=401, detail="Invalid token") from exc
24
25@asynccontextmanager
26async def tenant_db_transaction(tenant_id: str = Depends(extract_tenant_id)):
27 async with db_pool.connection() as conn:
28 # Explicit transaction block required for SET LOCAL isolation
29 async with conn.transaction():
30 await conn.execute(
31 "SELECT set_config('app.tenant_id', %s, true)",
32 (tenant_id,)
33 )
34 yield connCalling set_config('app.tenant_id', value, true) is functionally identical to SET LOCAL app.tenant_id = value, with the added advantage of supporting parameterized query binding to eliminate SQL injection vectors if a claim format changes from UUIDs to strings.
How do you enforce AlloyDB Row-Level Security (RLS) for vector embeddings?
To implement a multi-tenant AI architecture GCP AlloyDB Row-Level Security (RLS) for vector embeddings setup, define a permissive RLS policy on your document chunks table that casts current_setting('app.tenant_id', true) to UUID, and enable FORCE ROW LEVEL SECURITY so even the table owner cannot bypass tenant boundaries. Ensure the application connects via a non-superuser role without the BYPASSRLS attribute.
Enforcing tenant isolation RAG AlloyDB policies at the storage engine level guarantees that even if an engineer writes a vector search query omitting a WHERE tenant_id clause, PostgreSQL automatically rewrites the query abstract syntax tree (AST) prior to planning to filter out unauthorized rows. Passing true as the second argument to current_setting('app.tenant_id', true) prevents PostgreSQL from throwing an unhandled missing-parameter error if a background job queries the table without setting the variable; instead, it returns NULL, safely evaluating the equality check to false and returning zero rows.
1-- Create a dedicated non-superuser role for Cloud Run
2CREATE ROLE cloud_run_app NOLOGIN NOBYPASSRLS;
3GRANT USAGE ON SCHEMA public TO cloud_run_app;
4
5CREATE TABLE document_chunks (
6 chunk_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
7 tenant_id UUID NOT NULL,
8 document_id UUID NOT NULL,
9 content TEXT NOT NULL,
10 embedding vector(768) NOT NULL
11);
12
13-- B-tree index on tenant_id is mandatory for RLS + ScaNN inline filtering
14CREATE INDEX idx_chunks_tenant_id ON document_chunks USING btree (tenant_id);
15
16-- Create AlloyDB ScaNN vector index
17CREATE INDEX idx_chunks_embedding ON document_chunks
18USING alloydb_scann (embedding cosine)
19WITH (num_leaves = 1000, quantizer = 'ah');
20
21GRANT SELECT, INSERT, UPDATE, DELETE ON document_chunks TO cloud_run_app;
22
23-- Enable and force RLS so table owners are also restricted
24ALTER TABLE document_chunks ENABLE ROW LEVEL SECURITY;
25ALTER TABLE document_chunks FORCE ROW LEVEL SECURITY;
26
27CREATE POLICY tenant_isolation_policy ON document_chunks
28 AS PERMISSIVE
29 FOR ALL
30 TO cloud_run_app
31 USING (tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid)
32 WITH CHECK (tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid);Notice both USING and WITH CHECK clauses are declared. The USING clause filters existing rows during SELECT, UPDATE, and DELETE operations (protecting RAG retrieval), while WITH CHECK validates new or modified rows during INSERT and UPDATE operations, preventing a compromised Cloud Run route from writing embeddings tagged with another tenant's UUID.
How does AlloyDB ScaNN maintain recall and speed during RLS-filtered vector search?
AlloyDB row level security vector search maintains recall and low latency by combining a B-tree index on tenant_id with the alloydb_scann index's Adaptive Filtering and Bitmap-assisted Inline Filtering, dynamically choosing between pre-filtering, inline filtering, and post-filtering at runtime. Standard PostgreSQL HNSW indexes frequently suffer recall collapse or latency spikes when evaluating highly selective tenant predicates.
In naive vector databases, applying a tenant filter alongside an ANN search forces a brittle choice. Post-filtering searches the global vector index first and drops non-matching tenants afterward; if a tenant owns only 0.1% of the table, top-K vector search may return zero rows after filtering. Pre-filtering finds all rows for the tenant first and calculates exact vector distances on the subset, which degrades into a slow sequential scan for large tenants.
AlloyDB resolves this trade-off in the query optimizer. According to Google Cloud Documentation (2026), AlloyDB AI's ScaNN index provides up to 10x faster index creation, up to 4x faster standard vector search queries, and up to 10x faster filtered vector search queries compared to standard PostgreSQL HNSW indexes. Furthermore, AlloyDB's ScaNN index scales to 10 billion vectors using a four-level tree architecture and supports Asymmetric Hashing (AH) quantization, which achieves 4x greater compression compared to 8-bit Scalar Quantization (SQ8), according to Google Cloud Blog (2026) and Google Developer Forums (2025).
When an RLS policy injects tenant_id = current_setting(...) into a vector query, AlloyDB's Adaptive Filtering observes runtime statistics, filter selectivity, and data distribution to pick the optimal strategy:
| Execution Strategy | When AlloyDB Selects It | How It Executes Under RLS |
|---|---|---|
| Pre-filtering (Exact KNN) | High selectivity (small tenants holding a tiny fraction of total rows) | Uses the B-tree index on tenant_id to fetch candidate rows first, then computes exact vector distances on those rows. |
| Bitmap-assisted Inline Filtering | Moderate selectivity (mid-sized to large tenants) | Exclusive to alloydb_scann: runs a Bitmap Index Scan on the tenant_id B-tree index to build a bitmap filter, evaluating it concurrently during the Custom Scan traversal of the ScaNN index. |
| Post-filtering | Low selectivity (a dominant tenant owns the vast majority of the table) | Traverses the ScaNN vector index first and verifies the RLS tenant predicate against retrieved heap tuples. |
Concrete production workloads illustrate this optimizer behavior. In a retail search architecture on AlloyDB AI with 100,000 products where a filter matches only 0.2% of rows (200 items), the AlloyDB query planner uses a B-tree index on the metadata column to pre-filter the candidate set before running the vector similarity search with text-embedding-005 (Google Cloud Blog, 2025). Developers can also explore this behavior hands-on: a Google Cloud interactive codelab demonstrates deploying a dynamic hybrid retail search application combining Cloud Run, Gemini, and AlloyDB AI with ScaNN indexing, faceted metadata filters, adaptive filtering, and semantic reranking (Google Codelabs, 2026).
When should you add hash partitioning alongside RLS?
When your multi-tenant vector table exceeds 50 million rows, combine RLS with declarative PostgreSQL hash partitioning on tenant_id (for example, 32 or 64 hash partitions). Each partition maintains its own localized alloydb_scann index, constraining ANN scan scope via partition pruning at planning time and preventing massive enterprise tenants from degrading index traversal times for smaller cohorts.
How do you implement Gemini API token quota management per tenant?
Effective Gemini API token quota management requires a two-phase token accounting mechanism in Cloud Run: pre-flight reservation using Gemini's free CountTokens endpoint (or deterministic heuristics) against a Redis sliding-window ledger, followed by post-stream reconciliation using actual usageMetadata returned by the model. You cannot rely on Cloud Monitoring metrics to enforce real-time tenant quotas.
On Vertex AI, quotas operate at the Google Cloud project and region level, not per application tenant. In the Gemini Enterprise Agent Platform, rate quotas apply at the project and region level, allowing up to 30,000 online inference requests per minute in a single project per supported region (Gemini Enterprise Agent Platform quotas and limits - Google Cloud Documentation, 2026). For standard Pay-as-You-Go Gemini models, Google Cloud Dynamic Shared Quota (DSQ) eliminates fixed project TPM ceilings and targets a 99.5% SLO in the high-priority lane for requests within an organization's default Tokens Per Second (TPS) threshold, according to Google Cloud Blog (2026). However, an unthrottled loop or bulk-ingestion job triggered by a single noisy tenant can breach organizational TPS thresholds or exhaust Provisioned Throughput units, causing HTTP 429 Resource Exhausted errors for every customer on the platform.
Attempting to throttle tenants by querying Google Cloud Monitoring APIs fails due to telemetry lag. Cloud Monitoring quota metrics for Vertex AI (such as generate_content_input_tokens_per_minute_per_base_model) are sampled every 60 seconds and experience visibility latency of up to 150 seconds after sampling (How Vertex AI rate limits are calculated on GCP?, 2024). Within a 210-second blind spot, a runaway tenant agent can consume millions of tokens.
Phase 1: Pre-flight token estimation and reservation
Before dispatching a generateContent stream from Cloud Run, calculate the prompt token footprint and reserve a worst-case budget (input tokens plus configured max_output_tokens) in an atomic Memorystore for Redis Lua script.
- Exact pre-flight counting: For moderate request volumes, invoke the Vertex AI countTokens API. Calls to models.countTokens do not count toward a target model's inference rate limits (RPM, TPM, or RPD) and are free of charge, operating under a separate rate limit of up to 3,000 requests per minute per project (Clarification on Gemini API countTokens pricing, rate limits, and counting scope, 2026).
- Local heuristic estimation: If your multi-tenant SaaS exceeds 3,000 RPM across the project, bypass network round-trips to countTokens by running local byte-length or character-based estimation (such as dividing UTF-8 character count by 3.5 for English text) plus a static buffer for system instructions and tool schemas.
Phase 2: Streaming metadata capture and ledger reconciliation
Once the pre-flight reservation succeeds in Redis, Cloud Run streams the response from Vertex AI Gemini. When the final chunk arrives, read the usage_metadata payload (containing prompt_token_count, candidates_token_count, and cached_content_token_count) and refund the unused portion of max_output_tokens back to the tenant's Redis sliding window, while emitting an asynchronous event to BigQuery or AlloyDB for monthly billing aggregation.
1import time
2from google import genai
3from google.genai import types
4
5client = genai.Client(vertexai=True, project="prod-ai-saas", location="us-central1")
6
7RESERVE_LUA = """
8local key = KEYS[1]
9local limit = tonumber(ARGV[1])
10local reserve = tonumber(ARGV[2])
11local current = tonumber(redis.call('GET', key) or '0')
12if current + reserve > limit then
13 return 0
14end
15local updated = redis.call('INCRBY', key, reserve)
16if updated == reserve then
17 redis.call('EXPIRE', key, 60)
18end
19return 1
20"""
21
22async def generate_with_tenant_quota(redis_client, tenant_id: str, tier_tpm_limit: int, prompt: str, max_out: int = 1024):
23 minute_bucket = int(time.time() // 60)
24 quota_key = f"quota:tpm:{tenant_id}:{minute_bucket}"
25
26 # Fast local heuristic (~4 chars/token) when above 3,000 RPM countTokens project cap
27 est_input_tokens = max(1, len(prompt) // 4)
28 reserved_tokens = est_input_tokens + max_out
29
30 allowed = await redis_client.eval(RESERVE_LUA, 1, quota_key, tier_tpm_limit, reserved_tokens)
31 if not allowed:
32 raise HTTPException(status_code=429, detail="Tenant TPM quota exceeded")
33
34 actual_total = reserved_tokens
35 try:
36 response_stream = await client.aio.models.generate_content_stream(
37 model="gemini-2.5-flash",
38 contents=prompt,
39 config=types.GenerateContentConfig(max_output_tokens=max_out)
40 )
41 async for chunk in response_stream:
42 if chunk.usage_metadata and chunk.usage_metadata.total_token_count:
43 actual_total = chunk.usage_metadata.total_token_count
44 yield chunk.text
45 finally:
46 # Reconcile difference between worst-case reservation and actual usage
47 delta = reserved_tokens - actual_total
48 if delta != 0:
49 await redis_client.decrby(quota_key, delta)Wrapping the reconciliation step inside a finally block ensures that even if the client disconnects mid-stream or Vertex AI terminates early, the tenant's quota ledger accurately reflects consumed tokens rather than permanently locking the worst-case max_output_tokens reservation for the remainder of the minute window.
What are the practical next steps to implement this architecture?
Start by enforcing database-level tenant isolation and connection safety before adding vector scale optimizations or tiered token rate limiters. Execute the rollout in four verifiable engineering stages:
- Audit roles and connection pooling in AlloyDB: Enable AlloyDB Managed Connection Pooling, verify your Cloud Run service connects using a dedicated PostgreSQL role configured with NOBYPASSRLS, and refactor your data access layer to wrap every query in an explicit transaction calling set_config('app.tenant_id', tenant_uuid, true).
- Deploy RLS policies alongside composite indexing: Enable FORCE ROW LEVEL SECURITY on all vector tables, create both a standard B-tree index on tenant_id and an alloydb_scann index with Asymmetric Hashing quantization on your embedding column, and run EXPLAIN ANALYZE as the application role to confirm Bitmap-assisted Inline Filtering or Pre-filtering activates across small and large tenant IDs.
- Introduce two-phase Redis token accounting: Stand up Memorystore for Redis in your Cloud Run VPC, implement pre-flight token reservation (using models.countTokens below 3,000 RPM or character heuristics above it), and reconcile exact usageMetadata counts at stream completion.
- Plan your multi-tenancy graduation tier: Monitor table growth and enterprise compliance requirements; add declarative hash partitioning on tenant_id before vector tables cross 50 million rows, and evaluate a VPC Service Controls hub-and-spoke topology for regulated enterprise customers demanding dedicated AlloyDB spokes.
Sources
- Google Cloud, AlloyDB AI Vector Search overview - Google Cloud Documentation (2026) (opens in a new tab)
- Kumar Ramamurthy, Google Cloud - Medium (2026) (opens in a new tab)
- Google Cloud, Google Developer Forums (2025) (opens in a new tab)
- Google Cloud, Google Cloud Blog (2026) (opens in a new tab)
- Google Cloud, Google Cloud Blog (2026) (opens in a new tab)
- Google Developer Forums (2026) (opens in a new tab)
- Google AI Developer Forum, Clarification on Gemini API countTokens pricing, rate limits, and counting scope (2026) (opens in a new tab)
- Google Cloud, Gemini Enterprise Agent Platform quotas and limits - Google Cloud Documentation (2026) (opens in a new tab)
- Developers Digest, Vertex AI Quotas and Rate Limits (November 2026) (2026) (opens in a new tab)
- DEV Community, pgvector HNSW Filtering Returned 0 Rows for 52 of My 400 Queries (2026) (opens in a new tab)
- Logic42, pgvector HNSW Memory Cliffs: WAL Bloat & Iterative Scans | Logic42 Practice Brief (2026) (opens in a new tab)
- InfoQ, Google Introduces Managed Connection Pooling for AlloyDB (2026) (opens in a new tab)