Service
BigQuery data engineering with AI
Formare AI builds data engineering and analytics systems on BigQuery that make data usable by more than the two engineers who know the schema: natural-language-to-SQL agents that cannot invent column names, self-hosted data catalogs with Gemini-written column descriptions and lineage, and knowledge graphs on Neo4j. We also tune BigQuery and Gemini pipelines for cost; in one production retail environment six optimisations cut Gemini API token costs by up to 70% on source tables of more than 10 billion rows. Projects start from €8,000 or at €600–€900 per day, after a free 30-minute discovery call.
Updated
Scope
What we build on BigQuery
Most companies have years of data in BigQuery that only a few senior engineers can query. We build the layer that changes that, and we build it so that it is safe to give to non-engineers.
- Natural-language-to-SQL agents on Google ADK that read your real schema before they write a query
- Sandboxed statistical analysis: AI-written Python runs in Vertex AI's Code Interpreter, never on your servers
- Self-hosted data catalogs that sync from BigQuery, describe columns with Gemini and show lineage
- Knowledge graphs built from flat BigQuery tables, queried in plain English through a natural-language-to-Cypher agent
- BigQuery pipeline design and deployment, and Vertex AI Feature Store integration for real-time features
Fit
Which problem needs which system
| Problem | What we build | Reference build |
|---|---|---|
| Business users wait on engineers for every number | A two-agent natural-language-to-SQL assistant with persistent sessions | NL → BigQuery Assistant |
| Nobody knows what tables and columns mean | A self-hosted data catalog with daily BigQuery sync, AI column descriptions and lineage | DS Data Catalog |
| Product relationships are painful to query in SQL | A Neo4j knowledge graph built from the catalog, with a natural-language Cypher agent | BigQuery Knowledge Graphs |
| AI queries on BigQuery cost too much or run too slowly | Partition-aware SQL, materialized views and Gemini context caching | Six-optimisation production pipeline |
Cost control
Keeping BigQuery and Gemini costs predictable
The usual cost surprise is an agent that queries BigQuery on every turn without partition filters. That scans 10 billion rows instead of 50 million. At 50,000 tokens per query result and 1,000 daily active users, the Gemini bill alone reaches about €2,400 a month before any BigQuery compute is counted.
The six optimisations
- Partition-aware query generation, with the partition column and valid date range in the agent's instructions
- Materialized views for the five to ten queries agents run most often
- Gemini context caching for large schemas and document sets
- Batched embedding generation, up to 250 texts per request
- Compressing result sets to summaries before they enter the model context
- Adaptive token budgets that shrink tool output as a conversation grows
Together these cut Gemini API token costs by up to 70% and query latency from 4.2 seconds to under 800 ms in the production environment described in the article under “Read next”.
Security
Access, code execution and governance
- Workload Identity on GKE Autopilot, so no service-account key files exist anywhere in the cluster
- The agent can only reference columns that exist, because the real schema is injected before any question is answered
- AI-generated Python runs in a sandbox, not on application servers
- An audit log of model and tool calls in BigQuery, so every answer can be traced
Systems are designed for EU-region data residency from the first sprint. That is engineering guidance, not legal advice: your counsel should confirm how GDPR and the EU AI Act apply to your use case.
Cost and timeline
How long data engineering projects take and what they cost
Most engagements run 2–12 weeks, and scope is defined before work begins.
Pricing is €600–€900 per day, from €8,000 for a fixed-scope project paid in milestones, or from €4,500/mo for a retainer. Rates are in euros and VAT may apply. If you are not sure whether you need an agent, a pipeline or just a cleaner data model, the 4-Week AI Strategy & Architecture Audit (from €8,000) delivers a written architecture plan prioritised by ROI. The 30-minute discovery call is free.
Work
Data builds, and how each is used
Reference build
NL → BigQuery Assistant
Conversational data analysis — multi-agent
Ask BigQuery in plain English: an orchestrator routes to schema-aware NL→SQL and Pandas agents.
How it was built
Reference build
DS Data Catalog
Self-hosted BigQuery data platform
Self-hosted data catalog for GCP / BigQuery teams. FastAPI + React + PostgreSQL + GKE Autopilot. AI-powered column descriptions via Gemini, data lineage via Cloud Data Lineage API, quality scoring, OAuth2, and full table/column search. No stale Confluence docs — metadata syncs directly from BigQuery.
- Full BigQuery sync
- AI column descriptions
- Data lineage graph
How it was built
Reference build
BigQuery Knowledge Graphs
Graph construction + NL → Cypher agent
Turns flat BigQuery product tables into a Neo4j knowledge graph, queried in plain English via NL→Cypher.
How it was built
FAQ
Questions about BigQuery data engineering and analytics
Cost, timelines and fit, answered before the first call.
How does natural-language-to-SQL work without inventing column names?
The real table schema is put into the BigQuery agent's instructions before any question is answered, so it can only reference columns that exist. A second agent runs statistical analysis in Vertex AI's sandboxed Code Interpreter, so AI-generated Python never runs on your servers. Follow-up questions such as "now filter that to Q4" work because the session is persisted.
Do you replace our existing BI dashboards?
No. Dashboards answer the questions you already know to ask. A natural-language agent covers the one-off questions that otherwise wait for a data engineer. If a few dashboards already answer what your team asks, we say so on the discovery call.
What is a self-hosted data catalog, and why build one?
A data catalog is a searchable record of your tables and columns, what they mean and where they come from. Ours syncs metadata directly from BigQuery every day, writes column descriptions with Gemini, shows lineage from the Cloud Data Lineage API, scores data quality and runs on GKE Autopilot with Workload Identity. Because the metadata comes from BigQuery rather than from hand-written pages, it does not go stale.
How do you keep Gemini and BigQuery costs under control?
Six measures: partition-aware SQL generation, materialized views for the queries agents run most, Gemini context caching, batched embedding generation, compressing result sets before they enter the model context, and adaptive token budgets. In one production retail environment these cut Gemini API token costs by up to 70% on tables of more than 10 billion rows. The cost-optimisation article walks through each one.
Can you work with our existing BigQuery datasets?
Yes. We regularly integrate with existing BigQuery datasets and GCP IAM configurations. Access runs through Workload Identity rather than key files, and the discovery call is for understanding your current setup before scoping begins.
How long does a data engineering project take and what does it cost?
Most projects run 2–12 weeks, and scope is defined before work begins. Pricing is €600–€900 per day, from €8,000 for a fixed-scope project paid in milestones, or from €4,500/mo for a retainer. Rates are in euros and VAT may apply. If you are not sure whether you need an agent, a pipeline or just a cleaner data model, the 4-Week AI Strategy & Architecture Audit (from €8,000) ends in a written plan. The 30-minute discovery call is free.
Read next