Multimodal Product Search Under 300ms with Vertex AI Vector Search

Grocery search has a fundamental UX problem: customers think in terms of "the pasta with the yellow label" or "something like this photo" — not SKU names. Standard keyword search returns zero results for both. This multimodal shopping assistant solves this by understanding images, natural language, and voice equally — then retrieving the closest matching products from the catalog in under 300ms.

Three Variants Built

Variant 1 — Text-based production React app: conversational ADK agent backed by semantic vector search
Variant 2 — Multimodal image + text: customers upload a product photo, get visually similar results
Variant 3 — Live audio: real-time voice shopping via Gemini Native Audio API over WebSocket

The Search Pipeline

User Input (text | image | voice)
  ↓
Gemini multimodal embedding (multimodalembedding@001)
  ~90% of catalog has images → multimodal embedding
  ~10% text-only → text embedding fallback
  ↓
Vertex AI Vector Search (ANN, tree_ah_config / SCANN)
  approxNumNeighbors: 150 → >95% Recall@20
  P99 latency: <300ms at scale
  ↓
Feature Store enrichment (<2ms real-time lookup)
  inventory_status, price, margin_tier, click_rate
  ↓
Vertex AI Ranking API
  Populates ranking_score on each candidate
  +15–25% precision improvement over raw ANN distance
  ↓
Gemini strategy layer
  Assembles final response with reasoning

ANN Index Configuration

The index configuration is the most important performance decision. We validated against Google's published benchmarks — the same configuration used by Google Search, YouTube Recommendations, and Lowe's Visual Scout:

python

1# Vertex AI Vector Search index configuration
2# tree_ah_config (SCANN) for <300ms P99 at scale
3index = aiplatform.MatchingEngineIndex.create_tree_ah_index(
4    display_name="bringo-product-index",
5    contents_delta_uri=f"gs://{BUCKET}/embeddings/",
6    dimensions=1408,  # multimodalembedding@001 output dimension
7    approximate_neighbors_count=150,  # >95% Recall@20 threshold
8    distance_measure_type="DOT_PRODUCT_DISTANCE",
9    leaf_node_embedding_count=500,
10    leaf_nodes_to_search_percent=7,
11    description="Bringo product catalog — multimodal embeddings",
12)
13
14# Index endpoint with auto-scaling
15endpoint = aiplatform.MatchingEngineIndexEndpoint.create(
16    display_name="bringo-index-endpoint",
17    public_endpoint_enabled=True,
18)
19endpoint.deploy_index(
20    index=index,
21    deployed_index_id="bringo_product_index_v1",
22    min_replica_count=1,
23    max_replica_count=2,  # Auto-scale under load
24)

approxNumNeighbors: 150 is the key setting for >95% Recall@20. Lower values (e.g., 50) give faster queries but miss relevant products. Validated against Google's benchmark data from the Lowe's Visual Scout case study.

Embedding Generation Pipeline

python

1from google.cloud import aiplatform
2from vertexai.vision_models import MultiModalEmbeddingModel
3
4model = MultiModalEmbeddingModel.from_pretrained("multimodalembedding@001")
5
6def embed_product(product: dict) -> list[float]:
7    """Generate multimodal or text-only embedding based on image availability."""
8    if product.get("image_url"):
9        # Multimodal: image + text combined (~90% of catalog)
10        image = Image.load_from_file(download_image(product["image_url"]))
11        embeddings = model.get_embeddings(
12            image=image,
13            contextual_text=f"{product['name']} {product['category']} {product['brand']}",
14        )
15        return embeddings.image_embedding  # 1408-dimensional vector
16    else:
17        # Text-only fallback (~10% of catalog)
18        embeddings = model.get_embeddings(
19            contextual_text=f"{product['name']} {product['category']} {product['brand']}",
20        )
21        return embeddings.text_embedding
22
23# Batch pipeline: 10,000 products/hour
24# Run as Cloud Run Job, triggered by catalog update events

Ranking API Integration

Raw vector distance ranks by embedding similarity. But a product that's out of stock, or has a 2-star rating, should rank lower than its embedding similarity suggests. The Vertex AI Ranking API injects real-time signals — inventory, margin, click-through rate — on top of semantic similarity.

python

1from vertexai.preview.ranking import RankingServiceClient
2
3ranking_client = RankingServiceClient()
4
5def rerank_results(query: str, candidates: list[dict]) -> list[dict]:
6    """Re-rank ANN candidates using Vertex AI Ranking API."""
7    records = [
8        {
9            "id": p["product_id"],
10            "title": p["name"],
11            "content": f"{p['category']} {p['brand']} {p.get('description', '')}",
12        }
13        for p in candidates
14    ]
15
16    response = ranking_client.rank(
17        request={
18            "ranking_config": RANKING_CONFIG_NAME,
19            "model": "semantic-ranker-512@latest",
20            "top_n": 20,
21            "query": query,
22            "records": records,
23        }
24    )
25
26    # Merge ranking scores back into candidate objects
27    scores = {r.id: r.score for r in response.records}
28    for p in candidates:
29        p["ranking_score"] = scores.get(p["product_id"], 0)
30
31    return sorted(candidates, key=lambda x: x["ranking_score"], reverse=True)

Gemini Live Voice Layer — Critical Config

The voice variant uses Gemini's native audio streaming API. After extensive debugging, the correct session configuration is:

typescript

1// CORRECT — transcription config at top level, NOT inside generationConfig
2const session = await ai.live.connect({
3  model: 'gemini-2.5-flash-native-audio-preview-12-2025',
4  config: {
5    outputAudioTranscription: {},    // ← top level
6    inputAudioTranscription: {},     // ← top level
7    generationConfig: {
8      responseModalities: ['AUDIO'],
9    },
10    systemInstruction: {
11      parts: [{ text: SHOPPING_ASSISTANT_PROMPT }],
12    },
13  },
14})
15
16// WRONG — putting transcription inside generationConfig silently fails
17// config: { generationConfig: { outputAudioTranscription: {} } }  ← broken

Performance Results

P99 search latency: <300ms
Recall@20: >95%
Precision gain from Ranking API: +15–25%
Feature Store real-time lookup: <2ms
Embedding generation throughput: 10K/hr