All articles

Full-Stack AI12 min readBy Petrică RădanReference guide

Multi-Tenant SaaS on Cloud Run with Gemini Streaming and RBAC

Learn how to combine Identity Platform RBAC, unbuffered HTTP/2 or SSE proxies, and CPU-always-allocated Cloud Run instances for reliable per-tenant Gemini token metering

  • Cloud Run
  • Gemini API
  • Multi-Tenant SaaS
  • Token Metering
  • RBAC
  • Identity Platform
  • Server-Sent Events

To architect a multi-tenant SaaS Cloud Run application with Gemini streaming, you pair Google Cloud Identity Platform for tenant-scoped role-based access control (RBAC) with HTTP/2 or Server-Sent Events (SSE) proxies that intercept Vertex AI's final usage metadata chunks. Because streaming connections stay open for extended durations and clients can disconnect mid-generation, reliable per-tenant LLM token metering requires CPU-always-allocated Cloud Run instances that asynchronously flush token counts to Pub/Sub or Redis alongside Vertex AI request labels.

When multiple enterprise tenants share a serverless runtime and a project-wide Vertex AI quota pool, naive proxy designs fail quickly. Buffered responses break real-time UI updates, client aborts erase usage telemetry, and a single noisy tenant can exhaust regional token-per-minute thresholds. Solving these failure modes requires aligning Cloud Run container lifecycle controls, Identity Platform JWT validation, and Vertex AI's streaming protocol mechanics into a unified multi-tenant SaaS Cloud Run Multi-tenant SaaS architecture.

How do you design a Gemini streaming architecture on Cloud Run?

Designing a Gemini streaming architecture on Cloud Run requires enabling end-to-end HTTP/2 or unbuffered Server-Sent Events (SSE), extending request timeouts beyond the 5-minute default, and managing Vertex AI continuation tokens for ultra-long outputs. Without explicit protocol and timeout tuning, Google's frontend infrastructure or your container's ASGI/HTTP server will prematurely drop or buffer multi-tenant SaaS Cloud Run Gemini API streaming responses.

Protocol Negotiation: HTTP/2 Cleartext (h2c) vs. HTTP/1.1 SSE

By default, Cloud Run downgrades incoming external HTTP/2 requests to HTTP/1.1 before forwarding them to your container instance. When you explicitly configure end-to-end HTTP/2 on a Cloud Run service, Google's frontend infrastructure terminates TLS and forwards traffic to the container in HTTP/2 cleartext (h2c) format, enabling multiplexed streams over a single TCP connection. If your client architecture relies on standard browser EventSource APIs or RESTful fetch streams, you can alternatively serve SSE over HTTP/1.1 or HTTP/2 by setting Content-Type to text/event-stream, Cache-Control to no-cache, and X-Accel-Buffering to no to prevent intermediate proxy chunk buffering.

If your platform places an API gateway in front of Cloud Run, stream inspection must also operate without buffering the entire payload. For example, according to Google Cloud Documentation - Streaming server-sent events | Apigee (2026), Apigee EventFlow can proxy Vertex AI Gemini API SSE streams using content-type='text/event-stream', applying Model Armor policies with <ContinuousScan> set to true to sanitize streaming model responses in flight while routing events to Pub/Sub and Cloud Run functions for downstream processing.

Timeout Boundaries and Container Lifecycle Mechanics

Long-running reasoning chains, agentic loops, and live audio streams routinely exceed standard web request durations. According to Google Cloud Documentation - Configure request timeout for services (2026), the Cloud Run request timeout is set by default to 5 minutes (300 seconds) and can be extended up to a maximum of 60 minutes (3,600 seconds) to accommodate long-running streaming connections. Simultaneously, container initialization must stay lean; according to How to Configure Cloud Run Request Timeout (2026), Cloud Run container startup timeout requires service instances to begin listening for requests and pass startup probes within 4 minutes (240 seconds) after starting.

Engineers must handle the exact failure mechanic of a Cloud Run timeout. When a Cloud Run request reaches its configured timeout limit, Google's frontend closes the client network connection and returns an HTTP 504 error, but it does not immediately terminate the underlying container instance. If your backend does not track the remaining execution deadline and explicitly cancel the upstream Vertex AI stream, the orphaned background coroutine will continue consuming Gemini tokens that can neither be delivered to the user nor cleanly attributed.

Token Output Limits, Continuation, and Live Context Compression

Streaming directly impacts how much content a model can return in a single invocation. According to Google Cloud Documentation - Generate content with the Gemini API (2026), the single-request output limit for the Vertex AI Gemini API is 65,536 tokens for streaming requests compared to 32,768 tokens for non-streaming direct HTTP requests, with continuation tokens allowing generation up to 1,048,576 tokens. Similarly, according to Google Cloud Documentation (2026), in Gemini API streaming (streamGenerateContent), a streaming request can stop after 65,536 output tokens with a continuation token, and can continue generating up to the 1,048,576-token limit on Gemini 4 models.

Under the hood, when a long-running streaming decode hits the 65,536-token threshold, the stream terminates with finishReason set to CONTINUATION and returns an opaque continuationToken. Your Cloud Run service—or the Google Gen AI SDK via automatic_continuation in v1beta1—can pass this token back to resume generation seamlessly for the tenant. For multimodal real-time applications, context accumulation requires active bounding: according to Google Cloud Documentation (2026), during live streaming sessions with the Gemini Live API, live audio tokens accumulate at approximately 25 tokens per second of audio, and context window compression can be configured to trigger at 100,000 tokens with a 4,000-token sliding window.

How do you implement Cloud Run RBAC multi-tenant isolation?

You implement Cloud Run RBAC multi-tenant isolation by combining Google Cloud Identity Platform tenants for user-level authentication and custom JWT claims with IAM-authenticated hub-and-spoke Cloud Run topologies or database Row-Level Security (RLS). Choosing between logical isolation in a shared service and physical service-per-tenant isolation depends on enterprise compliance boundaries and blast-radius tolerances.

Tenant-Scoped Authentication with Identity Platform

Google Cloud Identity Platform supports native multi-tenancy, allowing you to provision isolated user silos under a single Google Cloud project. Each tenant silo maintains independent SAML or OIDC enterprise identity providers, user directories, and custom JWT claims (such as role and tier) alongside an immutable firebase.tenant identifier inside the issued ID token. When a request hits your Cloud Run middleware, the service cryptographically verifies the JWT signature, extracts the tenant ID and RBAC role claims, and injects a strongly typed tenant context into the request scope before any Vertex AI call is constructed.

At the persistence layer, this verified tenant context enforces strict data boundaries so one tenant's Retrieval-Augmented Generation (RAG) context or chat history never leaks to another. For example, according to How We Build Multi-Tenant SaaS Platforms on Google Cloud (2026), Dezoko architects multi-tenant SaaS platforms on Google Cloud using Cloud Run, Firebase Auth / Identity Platform, and PostgreSQL or Firestore, enforcing tenant isolation by including a tenant_id on all records backed by Postgres Row-Level Security (RLS) or custom Firestore security rules.

Logical Pool vs. Hub-and-Spoke Compute Isolation

Once authentication is established at the edge, you must decide how Cloud Run compute instances are partitioned across your customer base:

  • Shared Pool (Logical Isolation): All tenants route to a single auto-scaling Cloud Run service. Tenant isolation is enforced in application memory via middleware, scoped database sessions, and per-tenant cache prefixes. This minimizes cold starts and maximizes container utilization.
  • Hub-and-Spoke (Physical Compute Isolation): Regulated enterprise tenants receive dedicated Cloud Run services with isolated environment variables, VPC connectors, and IAM service accounts, fronted by a centralized routing tier.

Google documents this exact physical isolation pattern for high-security AI workloads. According to Multi-Tenant Agentic AI on Google Cloud: A Production Walkthrough (2026), Google Cloud Architecture Center's multi-tenant agentic AI reference architecture uses a hub-and-spoke topology on Cloud Run where a routing hub service runs as a dedicated service account (routing-hub-dispatcher) and invokes isolated tenant Cloud Run services over HTTPS using Google-signed ID tokens and roles/run.invoker. By deploying the spoke services with --no-allow-unauthenticated, even a compromised network path cannot invoke a tenant's agent runtime without IAM-verified dispatcher credentials.

How do you build accurate per-tenant LLM token metering in a streaming app?

Accurate per-tenant LLM token metering requires intercepting the final usageMetadata chunk in the Gemini stream, attaching normalized tenant labels to every Vertex AI API call, and running Cloud Run with CPU always allocated so background metering flushes survive client disconnects. Pre-request token estimation cannot account for dynamic model reasoning tokens, multi-step tool calls, or mid-stream aborts.

Intercepting Streaming Usage Metadata Across Gemini APIs

When you invoke Vertex AI Gemini streaming via streamGenerateContent (typically appended with ?alt=sse for Server-Sent Events), the API yields a sequence of incremental GenerateContentResponse JSON chunks. Intermediate chunks contain partial text deltas and empty finishReason fields, whereas the final chunk delivers both the termination reason and the aggregated usageMetadata object—exposing promptTokenCount, candidatesTokenCount, cachedContentTokenCount, thoughtsTokenCount, and totalTokenCount. If you are building stateful agents using the newer Gemini Interactions API (POST /v1beta/interactions), SSE streaming follows a symmetric step-based lifecycle (interaction.created, step.start, incremental step.delta events, step.stop, and interaction.completed), delivering final token consumption metrics inside the usage object of the interaction.completed event.

Metering logic must also account for how different models and modalities bill tokens under the hood. Reasoning tokens included in thoughtsTokenCount vary widely depending on tenant configuration; according to Coolhand Labs Documentation (2026), in Gemini 2.5 models, the numeric thinkingBudget token count parameter has model-specific valid ranges: 128 to 32,768 tokens for gemini-2.5-pro, 0 to 24,576 tokens for gemini-2.5-flash, and 512 to 24,576 tokens for gemini-2.5-flash-lite. Multimodal inputs introduce further billing nuances: according to Google Cloud Documentation (2026), for billing purposes in Gemini 3 and later models, tokens consumed by document inputs in generateContent and streamGenerateContent calls are counted as image tokens.

Handling Client Aborts and Cloud Run CPU Throttling

If a Cloud Run service uses default request-based CPU allocation, Google Cloud throttles container CPU to near zero the instant the HTTP response stream closes—freezing background token-metering coroutines and dropping usage records when clients abort mid-stream.

Client disconnects are routine in chat and agent interfaces when users click "Stop generating" or close a browser tab. If a client aborts before the final Gemini chunk arrives, your stream generator raises a cancellation exception and never receives usageMetadata. To eliminate this metering blind spot, configure your Cloud Run service with CPU always allocated (--no-cpu-throttling) and wrap the streaming loop in a try/finally block. If usageMetadata was captured, publish the exact token breakdown asynchronously to Cloud Pub/Sub or Redis Streams; if the stream aborted early, accumulate the raw character/byte deltas yielded prior to cancellation and invoke Vertex AI's local tokenizer or countTokens endpoint in the background finally block to record a fallback ledger entry.

Cloud-Native Billing Attribution via Vertex AI Labels and BigQuery

Real-time Pub/Sub metering drives live tenant quota enforcement, while Vertex AI request labels provide an auditable financial ledger in BigQuery for invoice reconciliation. According to GitHub (2026), Vertex AI request body labels used for multi-tenant billing attribution allow keys and values up to a maximum of 63 characters (Unicode codepoints) containing lowercase letters, numbers, underscores, and dashes. In practice, according to Medium (2026), Sveta Morag demonstrated extending the Google ADK Gemini model class (LabeledGemini) to attach custom tenant labels such as customer_id='acme_corp' and use_case='ticket_analysis' to every streaming and non-streaming generate_content_async request for per-tenant cost segmentation.

Once labeled requests flow through Vertex AI, downstream FinOps pipelines can reconcile costs directly from Google Cloud's billing tables without impacting request latency. According to StackSpend Blog (2026), StackSpend connects directly to Google Cloud's standard BigQuery billing export to perform agentless token metering, extracting token quantities, directions, and model names from SKU descriptions like 'Generate content input token count Gemini 2.5 Flash' and 'Generate content output token count Gemini 2.5 Pro'. Similarly, according to CloudQuell Documentation (2026), CloudQuell parses generative AI line items from BigQuery detailed usage cost exports under the Vertex AI and Gemini Enterprise Agent Platform services, mapping SKUs like 'Gemini 3.1 Flash Lite Global Text Output - Predictions' into granular input, output, cache read, and cache write token counts.

python
1import asyncio
2import re
3from fastapi import FastAPI, Request, HTTPException
4from fastapi.responses import StreamingResponse
5from google import genai
6from google.genai import types
7
8app = FastAPI()
9genai_client = genai.Client(vertexai=True, project="your-gcp-project", location="global")
10
11LABEL_PATTERN = re.compile(r"^[a-z0-9_-]{1,63}$")
12
13def sanitize_label(value: str) -> str:
14    cleaned = re.sub(r"[^a-z0-9_-]", "_", value.lower())[:63]
15    if not LABEL_PATTERN.match(cleaned):
16        raise ValueError("Invalid label format")
17    return cleaned
18
19async def flush_tenant_usage(tenant_id: str, usage_payload: dict) -> None:
20    # Publish asynchronously to Cloud Pub/Sub or Redis Streams.
21    # Requires Cloud Run --no-cpu-throttling so execution completes post-response.
22    pass
23
24@app.post("/v1/chat/stream")
25async def stream_gemini_response(request: Request, body: dict):
26    # Extracted upstream by Identity Platform JWT verification middleware
27    tenant_id = request.state.tenant_id
28    role = request.state.role
29    if role not in ("admin", "member"):
30        raise HTTPException(status_code=403, detail="Insufficient RBAC role")
31
32    config = types.GenerateContentConfig(
33        thinking_config=types.ThinkingConfig(thinking_budget=1024),
34        labels={
35            "tenant_id": sanitize_label(tenant_id),
36            "use_case": sanitize_label(body.get("use_case", "general_chat")),
37        },
38    )
39
40    async def event_generator():
41        usage_record = None
42        partial_chunks = []
43        try:
44            stream = await genai_client.aio.models.generate_content_stream(
45                model="gemini-2.5-flash",
46                contents=body["prompt"],
47                config=config,
48            )
49            async for chunk in stream:
50                if await request.is_disconnected():
51                    break
52                if chunk.text:
53                    partial_chunks.append(chunk.text)
54                    yield f"data: {chunk.text}\n\n"
55                if chunk.usage_metadata:
56                    usage_record = {
57                        "prompt_tokens": chunk.usage_metadata.prompt_token_count,
58                        "output_tokens": chunk.usage_metadata.candidates_token_count,
59                        "thought_tokens": chunk.usage_metadata.thoughts_token_count,
60                        "cached_tokens": chunk.usage_metadata.cached_content_token_count,
61                        "total_tokens": chunk.usage_metadata.total_token_count,
62                        "estimated": False,
63                    }
64        finally:
65            if usage_record is None and partial_chunks:
66                # Fallback estimation when client aborts before final usageMetadata chunk
67                usage_record = {
68                    "aborted_output_chars": sum(len(c) for c in partial_chunks),
69                    "estimated": True,
70                }
71            if usage_record:
72                asyncio.create_task(flush_tenant_usage(tenant_id, usage_record))
73
74    return StreamingResponse(
75        event_generator(),
76        media_type="text/event-stream",
77        headers={
78            "Cache-Control": "no-cache",
79            "X-Accel-Buffering": "no",
80        },
81    )

How do you prevent noisy neighbors from exhausting Vertex AI quotas?

Preventing noisy-neighbor quota exhaustion when you architect a Gemini SaaS app requires pairing Vertex AI Dynamic Shared Quota (DSQ) with low-latency application or API gateway token rate limiters scoped to each tenant's subscription tier. Relying solely on project-level Google Cloud quotas allows a single runaway tenant workload to degrade throughput for every other customer on the platform.

Understanding Vertex AI PayGo Tiers and Dynamic Shared Quota

Vertex AI manages shared capacity dynamically rather than relying purely on rigid static per-project caps, but organizational priority thresholds still govern guaranteed performance. According to Google Cloud Blog - Build a robust and cost-effective gen AI strategy (2026), in Vertex AI Pay-as-You-Go (PayGo) environments, Dynamic Shared Quota (DSQ) provides a high-priority lane based on an organization's default Tokens Per Second (TPS) threshold, targeting a 99.5% SLO before overflowing excess traffic into a best-effort lane. Furthermore, your organization's prioritized throughput scales with historical platform spend: according to Vertex AI Quotas and Rate Limits (November 2026) (2026), for Gemini Pro models on Vertex AI, Standard PayGo Tier 2 requires $250 to $2,000 in 30-day customer spend for an organization-level prioritized throughput threshold of 1,000,000 Traffic TPM (Tokens Per Minute), while Tier 4 provides 10,000,000 Traffic TPM.

If one enterprise tenant launches an unthrottled batch job against your SaaS API, they can consume your entire 1,000,000 or 10,000,000 TPM high-priority lane, pushing interactive UI streams from all other tenants into the best-effort overflow lane where HTTP 429 errors or elevated time-to-first-token (TTFT) spikes can occur during regional contention.

Enforcing Per-Tenant Token Quotas at the Edge

To keep every tenant within their contracted share of your Vertex AI pool, enforce token-aware rate limiting before forwarding requests to Gemini. You have two primary architectural enforcement points:

  • Managed API Gateway Enforcement: According to Google Cloud Documentation (2026), the Apigee LLMTokenQuota policy is supported on Apigee and Apigee hybrid version 1.16.1 and later to manage and enforce token consumption limits across minute, hour, day, week, or month intervals. Apigee extracts the tenant identifier from the verified JWT or API key and deducts actual token counts returned in Gemini response metadata.
  • Custom Redis/Memorystore Enforcement Plane: If Cloud Run sits directly behind a Global External Application Load Balancer without Apigee, implement a sliding-window token bucket in Cloud Memorystore for Redis using atomic Lua scripts. Pre-authorize requests against an estimated input budget, and reconcile the bucket asynchronously once the streaming usageMetadata arrives.

Whichever enforcement mechanism you deploy, the quota check cannot become a bottleneck on the streaming path. According to Medium (2026), in a multi-tenant API rate limiting architecture, the enforcement data plane must maintain low latency (~50ms at p95), high throughput (~30,000 requests per second per region), and 99.99% availability to avoid slowing down request traffic.

What are the architectural trade-offs across Cloud Run multi-tenant isolation models?

Choosing the right multi-tenant SaaS Cloud Run architecture comes down to balancing infrastructure density against regulatory isolation, quota governance, and operational overhead. Most B2B SaaS platforms combine a shared logical pool for standard tiers with dedicated spoke services for regulated enterprise contracts.

Architectural DimensionShared Cloud Run Service (Logical Isolation)Hub-and-Spoke Cloud Run (Service-per-Tenant)
Authentication & RBACIdentity Platform multi-tenant JWT verified in shared middlewareIdentity Platform at Hub + IAM ID tokens (roles/run.invoker) to Spokes
Data IsolationPostgres RLS or Firestore security rules keyed by tenant_idDedicated database schemas, instances, or VPC-scoped stores per Spoke
Streaming ConfigurationShared concurrency pool; up to 3,600s timeout; CPU always allocatedPer-tenant concurrency & min-instance tuning; isolated fault domains
Token MeteringStream usageMetadata interception + Vertex AI request labels (max 63 chars)Stream interception + labels, or isolated GCP projects per enterprise spoke
Noisy Neighbor ProtectionStrict Redis or Apigee LLMTokenQuota enforcement required in data planeCloud Run max-instances cap per spoke + per-tenant token rate limiting

What are the practical next steps to deploy this architecture?

Transitioning from a single-tenant prototype to a production-grade multi-tenant SaaS Cloud Run platform is best executed in sequential layers so you can validate streaming stability, auth boundaries, and billing accuracy independently.

  • Configure Cloud Run for persistent streaming: Enable end-to-end HTTP/2 or unbuffered SSE headers, raise the service request timeout from the 300-second default toward the 3,600-second ceiling based on your longest agent workflow, ensure startup probes pass well within the 240-second limit, and enable CPU always allocated (--no-cpu-throttling).
  • Stand up Identity Platform multi-tenancy: Provision tenant silos, define custom RBAC claims, and build request middleware that validates the JWT and propagates tenant_id down to Postgres RLS sessions and upstream AI clients.
  • Instrument dual-path token metering: Wrap streamGenerateContent (or Interactions API SSE streams) to capture the final usageMetadata chunk—accounting for thoughtsTokenCount, cached tokens, and continuationToken loops up to 1,048,576 tokens—while attaching sanitized 63-character tenant labels to every Vertex AI request for BigQuery billing reconciliation.
  • Deploy per-tenant token rate limiting: Place Apigee LLMTokenQuota (version 1.16.1+) or a sub-50ms p95 Memorystore Redis token bucket in front of your Vertex AI calls so no single tenant can exhaust your organization's DSQ high-priority TPM lane.
  • Test abort and timeout edge cases under load: Simulate mid-stream client disconnects and approaching 504 timeout deadlines to verify that upstream Gemini calls cancel cleanly and fallback token counts still flush to your metering ledger.

Sources

Book a discovery call

30 minutesGoogle MeetFree