Transforming BigQuery Product Tables into Knowledge Graphs with Neo4j and Gemini

The question "find all products in the same brand family that are substitutable for this out-of-stock item" is trivial for a knowledge graph and painful for SQL. It requires three joins, conditional logic, and knowledge of which columns encode brand family membership. In a graph, it's a two-hop traversal with a relationship filter. This pipeline builds that graph from existing BigQuery catalog data — without any manual graph construction.

Graph Construction Pipeline

BigQuery (product catalog, basket data, events)
  ↓
Extraction (Python, google-cloud-bigquery)
  ├── Product nodes: id, title, category, brand, price, attrs
  ├── Category taxonomy → BELONGS_TO edges
  ├── Brand/manufacturer → SAME_BRAND edges
  └── Basket co-occurrence → BOUGHT_TOGETHER edges
         ↓
Embedding-Based Edge Inference (Vertex AI Embeddings)
  ├── Embed all products (text: name + category + brand)
  ├── Cosine similarity > 0.85 → SIMILAR_TO edge
  └── Manual rules → SUBSTITUTABLE_FOR edges
         ↓
Graph Loading (Neo4j Python driver)
  ├── Batch MERGE operations (idempotent)
  └── Relationship properties: score, created_at
         ↓
NL→Cypher Agent (Gemini 2.0 Flash)
  └── Natural language → Cypher → Neo4j → explained results

Inferring Edges from Embedding Similarity

python

1from google.cloud import aiplatform
2import numpy as np
3from sklearn.metrics.pairwise import cosine_similarity
4
5def embed_products(products: list[dict]) -> np.ndarray:
6    """Embed all products using Vertex AI text embeddings."""
7    aiplatform.init(project=PROJECT_ID, location=REGION)
8    model = TextEmbeddingModel.from_pretrained("text-embedding-004")
9
10    texts = [
11        f"{p['name']} {p['category']} {p['brand']} {p.get('description', '')}"
12        for p in products
13    ]
14    # Batch: 250 texts per request
15    embeddings = []
16    for i in range(0, len(texts), 250):
17        batch = model.get_embeddings(texts[i:i+250])
18        embeddings.extend([e.values for e in batch])
19    return np.array(embeddings)
20
21def infer_similar_edges(
22    products: list[dict],
23    embeddings: np.ndarray,
24    threshold: float = 0.85,
25) -> list[tuple]:
26    """Find product pairs with cosine similarity > threshold."""
27    sim_matrix = cosine_similarity(embeddings)
28    edges = []
29    n = len(products)
30    for i in range(n):
31        for j in range(i + 1, n):
32            if sim_matrix[i][j] > threshold:
33                edges.append((
34                    products[i]["product_id"],
35                    products[j]["product_id"],
36                    float(sim_matrix[i][j]),  # similarity score as edge property
37                ))
38    return edges

Loading Into Neo4j

python

1from neo4j import GraphDatabase
2
3driver = GraphDatabase.driver(NEO4J_URI, auth=(NEO4J_USER, NEO4J_PASSWORD))
4
5def load_products_and_edges(products: list[dict], similar_edges: list[tuple]):
6    with driver.session() as session:
7        # MERGE is idempotent — safe to re-run
8        session.run("""
9            UNWIND $products AS p
10            MERGE (product:Product {id: p.product_id})
11            SET product.name = p.name,
12                product.category = p.category,
13                product.brand = p.brand,
14                product.price = p.price
15        """, products=products)
16
17        # SIMILAR_TO edges with similarity score
18        session.run("""
19            UNWIND $edges AS e
20            MATCH (a:Product {id: e[0]}), (b:Product {id: e[1]})
21            MERGE (a)-[r:SIMILAR_TO]->(b)
22            SET r.score = e[2], r.created_at = datetime()
23        """, edges=similar_edges)
24
25        # BELONGS_TO category edges from taxonomy
26        session.run("""
27            UNWIND $products AS p
28            MERGE (cat:Category {name: p.category})
29            WITH cat, p
30            MATCH (prod:Product {id: p.product_id})
31            MERGE (prod)-[:BELONGS_TO]->(cat)
32        """, products=products)

NL→Cypher Agent

python

1import vertexai
2from vertexai.generative_models import GenerativeModel
3
4model = GenerativeModel("gemini-2.0-flash")
5
6GRAPH_SCHEMA = """
7Nodes: Product (id, name, category, brand, price)
8       Category (name)
9Relationships:
10  (Product)-[:BELONGS_TO]->(Category)
11  (Product)-[:SIMILAR_TO {score: float}]->(Product)
12  (Product)-[:SUBSTITUTABLE_FOR]->(Product)
13  (Product)-[:BOUGHT_TOGETHER {frequency: int}]->(Product)
14  (Product)-[:SAME_BRAND]->(Product)
15"""
16
17def nl_to_cypher(question: str) -> str:
18    prompt = f"""Convert this question to a Cypher query for Neo4j.
19Graph schema:
20{GRAPH_SCHEMA}
21
22Question: {question}
23
24Return ONLY the Cypher query. No explanation.
25"""
26    response = model.generate_content(prompt)
27    return response.text.strip()
28
29def query_graph(question: str) -> dict:
30    cypher = nl_to_cypher(question)
31    with driver.session() as session:
32        result = session.run(cypher)
33        records = [dict(r) for r in result]
34    return {"query": cypher, "results": records}
35
36# Example:
37# "Which categories have the most substitutable product pairs?"
38# → MATCH (p1:Product)-[:SUBSTITUTABLE_FOR]->(p2:Product)
39#   RETURN p1.category, count(*) AS pairs ORDER BY pairs DESC

Graph Algorithms for Retail Intelligence

PageRank on SIMILAR_TO graph: identifies "hub" products with many related items — typically bestsellers or category anchors worth promoting
Community detection (Louvain): finds natural product clusters that often reveal hidden sub-categories the taxonomy missed
Shortest path: discovers unexpected relationships ("this budget pasta is 3 hops from this premium brand — via shared ingredient supplier")
Supplier impact analysis: given a supplier node, MATCH all affected products and their downstream categories in a single traversal

Relationship types (SIMILAR_TO, SUBSTITUTABLE_FOR, SAME_BRAND, BOUGHT_TOGETHER, BELONGS_TO): 5
Natural language Cypher agent (Gemini): NL
Cosine similarity threshold for SIMILAR_TO edges: 0.85
MERGE-based loading — safe to re-run daily: Idempotent