A demo runs on a spreadsheet. A product runs on pipelines built to last.

We design, build and run data platforms on Databricks - Lakeflow ingestion and declarative pipelines, a medallion architecture on Delta Lake and managed Iceberg, migrations off Oracle, Teradata and SQL Server, and streaming that lands in seconds. One governed source under Unity Catalog, feeding your BI, your data science and your agents. Building since 2013.

Databricks PartnerLakeflow · Unity CatalogBatch → real-timeISO/IEC 27001:2022
One governed pipelineDatabricks · LakeflowRunning
Sources - databases · files · events · SaaS0 rows
Bronzelanded as-isAuto Loader
Silvertyped & conformed0 pass0 held
Goldbusiness-readyMaterialized view
Governed byUnity Cataloglineage · ACL · audit
BIData ScienceAgents
Why it matters

Most data projects don't fail on the platform. They fail on the engineering.

Buying Databricks is a decision. Getting value out of it is a build. These are the six ways it goes wrong in practice - and every one of them is a data-engineering problem with a known, named answer on the platform.

01

A one-off extract is a demo

A notebook that reads a file and writes a table gets you a prototype. No refresh, no lineage, no replay - and the day the source changes, the whole thing quietly stops being true.

Answered by the medallion architecture
02

Hand-rolled ETL breaks in silence

Bespoke Spark jobs with no retry, no idempotency, no data-quality gate. It fails at 2am, the dashboard goes stale, and a customer notices before you do.

Answered by declarative pipelines & expectations
03

Ungoverned tables can't be trusted

Legacy Hive metastore leftovers, no column lineage, permissions granted per-cluster. You can't prove what a model learned from - and in a regulated industry that's the whole ballgame.

Answered by Unity Catalog
04

Notebooks in production aren't a platform

Logic living in an interactive notebook on an all-purpose cluster, promoted by copy-paste. No environments, no review, no rollback - and a bill nobody can attribute.

Answered by Asset Bundles & Lakeflow Jobs
05

Yesterday's numbers can't drive today's decision

A nightly batch is fine for a monthly report and useless for pricing, fraud or an agent answering a customer right now. Freshness is an architecture choice, made early.

Answered by streaming ingestion
06

Data and AI end up in different places

The data governed in the lakehouse, the AI running somewhere else on a copy nobody's tracking. That gap is where trust, governance and every hard audit question fall through.

Answered by one pipeline, three consumers
What we do on Databricks

Six things, done properly, on one platform.

We're a Databricks Partner with in-house data architects and cloud architects, so the platform and the cloud underneath it get designed as one thing. This is the whole surface - from the first workspace to the pipeline that's still healthy two years later.

Lakehouse platform architecture

Workspaces, the Unity Catalog metastore, a catalog-per-environment model, external locations and volumes, compute policies, networking and secrets. The foundation you only get one clean shot at - designed before the first table lands.

Unity CatalogDelta LakeManaged Iceberg

Ingestion, every shape of it

Managed connectors for SaaS and databases via Lakeflow Connect, incremental file ingestion with Auto Loader, event streams over Kafka or Kinesis, direct application writes with Zerobus Ingest, and change data capture landed as SCD Type 1 or 2.

Lakeflow ConnectAuto LoaderZerobus IngestAUTO CDC

Declarative pipelines

Transformations expressed as streaming tables and materialized views, not orchestrated scripts - so dependencies, incremental processing, retries and recovery are the engine's job. Quality gates are expectations in the pipeline, not a test suite somebody remembers to run.

Lakeflow Declarative PipelinesApache Sparkdbt

Legacy migrations

Off Oracle, Teradata, SQL Server, Netezza, Hadoop and legacy ETL onto Databricks - assessed and transpiled with Lakebridge, bridged with Lakehouse Federation so the old system stays queryable, then run in parallel and reconciled until the numbers agree. You cut over on evidence.

LakebridgeLakehouse FederationParallel run

Real-time pipelines

Streaming ingestion and continuous transformation when the decision can't wait for tonight - Structured Streaming, Zerobus for sub-five-second application events, and Real-Time Mode on declarative pipelines for the millisecond tier. Same medallion shape, different clock.

Structured StreamingReal-Time Mode · preview

Platform engineering & FinOps

Everything as code in Databricks Asset Bundles, promoted dev → staging → prod through your CI, jobs on serverless or right-sized job compute, and spend attributed per team from the system tables with budget policies on top. The part that decides whether year two is calm.

Asset BundlesLakeflow JobsSystem tables
Databricks PartnerAWS · Azure · GCPBatch & real-timeISO/IEC 27001:2022
The platform, layer by layer

Databricks is six layers. We work in all of them.

Most teams meet Databricks as “the place the notebooks are.” It's a stack - compute on your cloud, open tables on your storage, one catalog over everything, the engineering tools, and then the analytics and AI that read the result. Knowing which layer a problem lives in is most of the job.

01Consumethe surface
Dashboards, natural-language questions and apps - what the business actually touches.
Databricks SQLAI/BI dashboardsGenieDatabricks Apps
02IntelligenceML & agents
Models, features, vector indexes and production agents - all reading the governed tables.
Mosaic AIAgent BricksVector SearchMLflowModel Serving
03Engineerthe pipelines
Ingestion, transformation and orchestration - where a data platform is actually built.
Lakeflow ConnectLakeflow Declarative PipelinesLakeflow JobsApache SparkPhoton
04Governone control plane
Unity Catalog over every table, file, model, function and dashboard on the platform.
Unity CatalogColumn lineageRow filters & masksSystem tablesDelta Sharing
05Storeopen tables
Delta Lake and managed Iceberg tables in open formats, on your own cloud storage.
Delta LakeManaged IcebergLiquid clusteringPredictive optimization
06Runyour cloud
Serverless and job compute in your own AWS, Azure or GCP account.
Serverless computeJob computeAWS · Azure · GCPCompute policies
Consume
The layer your stakeholders judge the platform by. SQL warehouses serve queries, AI/BI dashboards serve the recurring questions, Genie answers the ad-hoc ones in natural language, and Databricks Apps host the small internal tools that used to be spreadsheets.
What lives here
SQL warehouses - serverless or provisioned compute for analytics queries, with result caching
AI/BI dashboards - governed, parameterised dashboards defined as code alongside the pipelines
Genie - natural-language questions answered against curated tables and metric definitions
Metric views - one governed definition of revenue, churn or margin, reused by every surface
Your existing BI - Power BI and Tableau connect to the same warehouses - nobody has to switch tools
What we do in this layer
We build the serving layer as part of the pipeline, not after it - metric definitions in the catalog, dashboards versioned in the same bundle as the tables they read, and warehouse sizing tuned to real query patterns rather than a guess.
Intelligence
The reason the data layer has to be right. Features, models and agents on Databricks read the same Unity Catalog tables as the dashboards, under the same permissions - which is what makes “the agent answered from governed data” a technical fact rather than a slide.
What lives here
Feature Engineering in Unity Catalog - features are governed tables with lineage, not a side database
MLflow - experiments, models and traces tracked and registered in the catalog
Model Serving - real-time endpoints for models and LLMs, with inference logged
Vector Search - indexes that stay in sync with the source Delta table for retrieval
Mosaic AI & Agent Bricks - the agent framework, evaluation and tool calling
Catalog functions as tools - an agent's tools are governed SQL/Python functions, permissioned like tables
What we do in this layer
We treat this layer as a consumer of the data platform, and build it that way: features derived in the same pipelines, retrieval indexes fed from Gold tables, and agent tools registered in the catalog so an agent's reach is reviewable. The agents themselves are the Agentic AI page's subject.
Engineer
Lakeflow is the umbrella over the three jobs of data engineering: getting data in, transforming it, and running the whole thing on a schedule. Declarative pipelines are the centre of gravity - you describe the tables you want and the engine works out the dependencies, the incremental processing and the recovery.
What lives here
Lakeflow Connect - managed connectors for SaaS applications, databases and cloud storage
Auto Loader - incremental, exactly-once file ingestion that scales to billions of files
Zerobus Ingest - a push API for applications to write events straight into Delta
Lakeflow Declarative Pipelines - streaming tables, materialized views, expectations and AUTO CDC - formerly Delta Live Tables
Lakeflow Jobs - orchestration with data-aware and file-arrival triggers - formerly Workflows
Spark & Photon - the execution engines underneath, for when SQL isn't enough
What we do in this layer
This is where we spend most of our time. Declarative first, imperative only where it earns it - quality gates as expectations, CDC as AUTO CDC flows rather than hand-written merges, and orchestration that reacts to data arriving instead of guessing at a cron time.
Govern
One permission model across the whole estate - three-level catalog.schema.object naming, and the same grant covering a table, a volume of files, a model and a function. Lineage is captured automatically down to the column, which is what makes an audit answerable.
What lives here
Catalogs & schemas - a catalog per environment is the cleanest isolation boundary there is
External locations & volumes - governed access to your cloud storage, including non-tabular files
Lineage - table- and column-level, captured automatically from queries and pipelines
Row filters, column masks, ABAC - fine-grained policy where regulation needs it
System tables - billing, query history, lineage and audit as queryable tables
Lakehouse Federation - query external databases in place, under the same governance
Delta Sharing & Clean Rooms - share live data outside the account without copying it
What we do in this layer
We design the catalog model on day one, because it's the decision that's most painful to change later: environment isolation, naming, ownership and grant strategy through groups - never per-user. Depth on this layer is the Governed Data Platform page.
Store
Your data sits in open Parquet-based formats in your own object storage - not in a proprietary store you have to export from. That's what makes the lakehouse an architecture rather than a product: transactional tables, versioned, readable by engines other than Databricks.
What lives here
ACID transactions - concurrent writers, atomic MERGE, no half-written tables
Time travel - query or restore any prior version - the audit trail is the storage format
Deletion vectors & change data feed - cheap deletes and updates, and a readable stream of row changes
Schema enforcement & evolution - bad data is rejected at the boundary, new columns are additive
Liquid clustering - replaces partitioning - including automatic clustering on managed tables
Predictive optimization - OPTIMIZE, VACUUM and ANALYZE run for you, when they're worth running
What we do in this layer
We make the layout decisions that show up on the bill: clustering keys over partition columns, file sizing and compaction left to the platform, retention and vacuum windows set deliberately, and format chosen from what else needs to read the data.
Run
Databricks runs in your cloud account, against your storage, inside your network. The compute layer is where most of the money is spent and most of the waste lives - long-lived all-purpose clusters doing scheduled work is the single most common finding in an estate review.
What lives here
Serverless compute - for jobs, pipelines, notebooks and SQL - no cluster to leave running
Job compute - ephemeral clusters sized per task, started and stopped by the job
Compute policies - guardrails so nobody can spin up a 40-node cluster to read a CSV
Networking & identity - private connectivity, customer-managed keys, SSO and SCIM groups
Multi-cloud - the same architecture on AWS, Azure or GCP - chosen from where you already are
What we do in this layer
Our cloud architects own this layer alongside the data work: network, identity, key management and compute policy designed with the platform rather than handed to a different team afterwards. It's why the cloud bill and the data architecture end up agreeing with each other.

Pick a layer to see what lives there - and what we build in it.

How we build it

Raw data comes in messy. It leaves trustworthy.

We build on the medallion pattern Databricks itself documents - three stages, each a step up in quality. Data lands raw and replayable, gets typed, deduplicated and quality-gated, then curated into business-ready tables. Every layer is a governed Delta Lake or managed Iceberg table: ACID transactions, schema enforcement, time travel. Nothing is overwritten; everything is auditable back to source.

Sources
DatabasesFilesEventsSaaS apps
Bronzeingest & land
Raw
Ingested exactly as it arrived. Append-only, full history, replayable - the source of truth.
Auto Loader · Lakeflow Connect
Silverclean & conform
Validated
Typed, deduplicated, joined and quality-gated by expectations - one reliable version of the record.
Lakeflow Declarative Pipelines · Expectations
Goldserve
Business-ready
Aggregated and modeled for the question - the shared definitions everything downstream reads.
Materialized views · Databricks SQL
Consumers
BIData ScienceAgents
Delta Lake & managed Iceberg - ACID transactions · schema enforcement · time travel · deletion vectors · change data feed
Unity Catalog - one set of permissions, column-level lineage and audit across all three layers
Quality rises left to right. Governance and history run underneath the whole thing, and table maintenance is the platform's job - predictive optimization runs OPTIMIZE, VACUUM and ANALYZE for you, and liquid clustering replaces the partitioning decisions teams used to get wrong. Governance is shown here for context - it's the Governed Data Platform page's subject in full.
BronzeRawLand data exactly as it arrived - no transformation, append-only, full history. The source of truth you can always replay from, so a bug in a transform is a re-run, not a data-loss incident.
The record - as it landed01 · Auto Loader
-- bronze · land it exactly as it arrived, incrementally
CREATE OR REFRESH STREAMING TABLE bronze.orders_raw
  COMMENT 'Raw orders · append-only · replayable'
AS SELECT
  *,
  _metadata.file_path        AS _src_file,
  current_timestamp()          AS _ingested_at
FROM STREAM read_files(
  '/Volumes/landing/orders/',
  format              => 'json',
  schemaEvolutionMode => 'addNewColumns');
Auto Loader tracks which files it has already seen, so this is incremental and exactly-once without you writing bookmark logic. New columns in the source are added rather than breaking the load - and the raw payload is still there to re-read.
What this layer guarantees
Nothing dropped or overwritten
Full replay back to source
Schema evolution handled, not feared
File-level provenance on every row
Built with
Auto LoaderLakeflow ConnectZerobus IngestAUTO CDC
SilverValidatedType, deduplicate, validate and conform - join reference data and enforce quality as declared expectations. One reliable version of the record, with the failures visible instead of silently missing.
The same record - typed & quality-gated02 · Expectations + AUTO CDC
-- silver · typed, deduplicated, quality-gated
CREATE OR REFRESH STREAMING TABLE silver.orders (
  CONSTRAINT valid_amount   EXPECT (amount > 0)
    ON VIOLATION DROP ROW,
  CONSTRAINT valid_customer EXPECT (customer_id IS NOT NULL)
    ON VIOLATION FAIL UPDATE
)
AS SELECT
  payload:cust ::string            AS customer_id,
  payload:amt  ::decimal(12,2)     AS amount,
  payload:cur  ::string            AS currency,
  to_timestamp(ts)                 AS order_ts
FROM STREAM bronze.orders_raw;
Two kinds of failure, handled differently on purpose: a negative amount is a bad row (drop it and count it), a missing customer id is a broken contract (stop the update). Slowly-changing dimensions are declared, not hand-merged:
AUTO CDC INTO silver.customers … KEYS (customer_id) SEQUENCE BY seq STORED AS SCD TYPE 2
What this layer guarantees
One validated version of the truth
Enforced schema and real types
Failed rows quarantined, not dropped
Quality metrics recorded per run
Built with
Lakeflow Declarative PipelinesExpectationsAUTO CDC · SCD 1 & 2Apache Spark
GoldBusiness-readyAggregate and curate into business-ready tables modeled for the question - the shared definitions BI, data science and agents all read. Recomputed incrementally, so freshness doesn't cost a full rebuild.
The same record - aggregated to a business metric03 · Materialized view
-- gold · one definition of revenue, kept fresh incrementally
CREATE OR REFRESH MATERIALIZED VIEW gold.daily_revenue
  CLUSTER BY (order_date)
AS SELECT
  date(order_ts)      AS order_date,
  count(*)             AS orders,
  sum(amount)         AS revenue,
  avg(amount)         AS aov
FROM silver.orders
GROUP BY date(order_ts);
A materialized view refreshes incrementally where it can, so a day of new orders doesn't re-scan a year of history. CLUSTER BY is liquid clustering - Databricks recommends it over PARTITIONED BY, and CLUSTER BY AUTO hands the key choice to the platform so it can change as query patterns do.
What this layer guarantees
One consistent business definition
What BI, data science and agents all read
Incremental refresh, not full rebuild
Time travel to any prior state
Built with
Materialized viewsDatabricks SQLMetric viewsdbt

Click Bronze, Silver or Gold to follow one order through all three layers - with the code that does it.

Freshness is a design decision

From “by tomorrow morning” to “in five milliseconds” - same architecture, different clock.

“Real-time” is the most expensive word in a data requirement, and most of the time it isn't what's needed. On Databricks the medallion shape doesn't change as you move up the freshness ladder - the trigger, the ingestion path and the bill do. We pick the cheapest tier that actually answers the question.

~8 hrs
source change → visible in a dashboard
Trigger
A scheduled Lakeflow Job - usually once a night, in the window after the source systems settle.
Ingestion
Auto Loader or a Lakeflow Connect connector, picking up everything new since the last run.
Pipeline
Triggered mode: the pipeline starts, processes what's arrived, and shuts the compute down.
Cost profile
The cheapest tier by a wide margin - compute only runs while there's work, and nothing idles overnight.
Trigger
A short schedule, or a file-arrival trigger so the job runs when data shows up rather than on a clock.
Ingestion
The same Auto Loader path, run far more often - plus AUTO CDC flows for database changes.
Pipeline
Still triggered, still incremental: materialized views refresh only what changed, not the whole history.
Cost profile
Modestly more than nightly. Serverless start-up is the main overhead, and it's small.
Trigger
Continuous. The pipeline is always on, consuming as events arrive.
Ingestion
Structured Streaming from Kafka, Kinesis or Event Hubs - or Zerobus Ingest, a push API that lands application events directly in Delta in under five seconds.
Pipeline
Continuous mode with checkpointed, exactly-once state. Watermarks and state retention become real design decisions.
Cost profile
Always-on compute, so it's a step change. Worth it per table, rarely worth it estate-wide.
Trigger
Continuous, with the pipeline running in Real-Time Mode - public preview at the time of writing.
Ingestion
A low-latency source: Kafka or a direct push through Zerobus, with nothing batching in front of it.
Pipeline
Declarative pipelines in Real-Time Mode, reaching end-to-end latencies as low as five milliseconds - inside Spark, with no separate stream processor to run.
Cost profile
The most expensive tier, and the one with the least tolerance for a sloppy transform. Reserved for the paths that need it.
Right forFinancial and management reporting, monthly and weekly cycles, anything a human reads once a day. This is where most of a real estate lives, and there's no prize for moving it.
Stops working whenThe business asks “what's happening now” - pricing, fraud, stock, service. A nightly table can't answer a question about this morning, and no amount of dashboard polish hides that.
Right forOperational dashboards, intraday reporting, most “we need it fresher” requests. This tier answers the majority of real-time asks at a fraction of the cost of actual streaming.
Stops working whenThe gap between runs is itself the problem - a decision that can't wait five minutes, or an event that has to trigger an action rather than update a number.
Right forFraud and risk signals, live operational and IoT telemetry, personalisation, and agents that answer questions about the current state of the business.
Watch out forUnbounded state and late-arriving data - the two things that turn a healthy stream into a slow memory leak. Also: streaming a table nobody reads more than hourly is pure waste.
Right forThe genuinely latency-bound path - an in-request decision, a control loop, a machine reacting to a machine. Usually one or two tables in an estate, not a layer.
Be honest aboutWhether the consumer can actually use it. Milliseconds upstream mean nothing if a dashboard refreshes every thirty seconds or a human reads it twice a day. We'd rather build this tier late than everywhere.

Cycling through the tiers - click any one to stop on it.

Why it's one budget, not two

The same governed Gold layer feeds your dashboards, your data science and your agents.

This is the argument that changes the conversation - and it's why this data page sits under an agentic-led cluster. Most companies build a data platform for BI, then start a separate AI project on a separate copy. That's two budgets, two governance stories and one gap between them. On Databricks you build the curated Gold layer once, govern it once in Unity Catalog, and everything reads from it in place.

One governed Gold layer
built once · governed once
Delta / Iceberg
Business Intelligence
Dashboards, natural-language questions and governed KPI definitions straight off the Gold tables - no extract, no second version of revenue.
Databricks SQLAI/BI dashboardsGeniePower BI · Tableau
Data Science & ML
Notebooks, features and models on the same tables the dashboards use - features registered in the catalog, runs and models tracked in MLflow.
Feature Engineering in UCMLflowModel Serving
AI Agentsthe bridge up
Production agents reading governed data in place, with catalog functions as their tools and the same permissions applied at query time - not a scraped copy.
Mosaic AIVector SearchUC functions as tools
One governance model - Unity Catalog permissions, column-level lineage and audit over all three.
Business Intelligence. A revenue dashboard on Databricks SQL reading gold.daily_revenue directly - the same numbers finance signs off on, with the definition of revenue living in the catalog rather than in seventeen different queries. Someone asks a follow-up question in Genie and gets an answer from the same table.
Data Science & ML. A churn model trained on the same Gold tables the dashboards use, with features registered in Unity Catalog and runs tracked in MLflow. No separate copy to drift, no re-plumbing when the model goes to production - and the lineage from prediction back to source table is already recorded.
AI Agents. A production agent answering questions grounded in the governed Gold layer - reading Delta tables in place under the asking user's own permissions, with its tools registered as catalog functions so its reach is reviewable. This is the bridge up to the Agentic AI page.
Same data. Same governance. Three destinations - and no gap between them.
The on-ramp

Off the old estate, without the big-bang risk.

Almost nobody starts clean. You start on Oracle, Teradata, SQL Server, a Hadoop cluster or a tangle of legacy ETL that's expensive, slow and in the way. We move you off it incrementally: assess and transpile with Lakebridge, land the raw data first, rebuild the logic declaratively, keep the old system queryable through Lakehouse Federation while both run, and cut over only once the numbers reconcile.

From - the legacy estate
OraclePL/SQL
TeradataBTEQ
SQL ServerT-SQL · SSIS
Hadoop / HiveHiveQL
Netezza & appliancesnzsql
Cloud warehouseSynapse · Redshift
Legacy ETLInformatica · DataStage
To - the lakehouse on DatabricksDelta / Iceberg
Gold - business-ready tables, dashboards & metrics
Silver - the old transformation logic, rebuilt declaratively
Bronze - your data, landed as-is first
nothing is transformed until it's safely in
Land raw first - so a migration is auditable, not a leap of faith.
LakebridgeDatabricks' free, open migration tooling - we run it, then do the engineering it can'tProfile → analyze → transpile → reconcile
Phase 01
Profiler & Analyzer - survey the estate
Profiles the source system, then inventories it: every table, script, job and dependency, with complexity scored per object. This is what turns “we should migrate” into a wave plan with real effort behind it.
Phase 02
Converter - transpile the code
Transpiles proprietary dialects - PL/SQL, BTEQ, T-SQL, HiveQL - and legacy ETL job definitions into open SQL that runs on Databricks. The mechanical bulk converts automatically; what is left is the logic that was never really SQL in the first place.
Phase 03
Reconciler - prove it matches
Compares source and target row counts, schemas and aggregates automatically, so the parallel-run phase produces a signed-off difference report instead of an argument.
Where the tool stops and we start: transpiled code runs - it isn't yet a platform. Re-expressing procedural logic as declarative pipelines, choosing incremental strategies, designing the target model, wiring quality gates, orchestration, CI/CD and cost controls, and retiring the old system safely: that's the engineering, and it's the part that decides whether the migration holds.
01Assess02Land raw03Rebuild declaratively04Run in parallel & reconcile05Cut over06Decommission
Oracle PL/SQL · ExadataMigration path · Databricks
Typical pain
High licensing and hardware cost, a scale ceiling on peak workloads, and enough Exadata-specific behaviour that leaving feels risky.
Ports cleanly
Tables, views and standard-SQL schemas transpile directly. Lakebridge converts the bulk of the PL/SQL surface into open SQL that runs on Databricks.
Re-engineered
Procedural PL/SQL packages, cursors and row-by-row logic get re-expressed as set-based declarative pipelines. Proprietary types and features (Spatial, advanced queuing) need a real design decision, not a translation.
Ingestion path
CDC via LogMiner or Debezium into Auto Loader, or a Lakeflow Connect database connector; bulk unload to cloud storage for history. Federation keeps Oracle queryable throughout.
Teradata BTEQ · TPTMigration path · Databricks
Typical pain
An expensive proprietary appliance with a hard scale ceiling, and a decade of BTEQ scripts nobody wants to touch.
Ports cleanly
Relational tables and ANSI-SQL transforms move over cleanly. Lakebridge handles BTEQ and Teradata SQL dialect conversion at volume - this is one of its strongest paths.
Re-engineered
Stored procedures, macros, utility scripts (MLOAD, FastExport) and anything relying on primary-index physics get rebuilt - usually as declarative pipelines plus dbt models for the presentation layer.
Ingestion path
Bulk extract via TPT to cloud storage, then Auto Loader into Bronze; incremental CDC where the source supports it. Waves usually follow the existing subject areas.
SQL Server T-SQL · SSISMigration path · Databricks
Typical pain
Per-core licensing at scale, SSIS package sprawl, and reporting tightly coupled to the database it reads.
Ports cleanly
Tables, views and most T-SQL logic port with light changes - Lakebridge covers the dialect conversion and the SSIS package inventory.
Re-engineered
SSIS packages become declarative pipelines. T-SQL stored procedures with temp tables and imperative control flow become set-based transforms. Anything doing application logic in the database gets a conversation first.
Ingestion path
Native CDC or Debezium into Bronze, or the Lakeflow Connect SQL Server connector; BCP for bulk history.
Hadoop / Hive HiveQL · Spark on YARNMigration path · Databricks
Typical pain
An ageing cluster nobody wants to own, unpredictable performance, and a Hive metastore full of tables with no clear owner.
Ports cleanly
HiveQL and existing Spark jobs are the closest thing to a lift-and-shift on this list - the same Spark API, with Photon underneath. Parquet and ORC data can often be registered rather than rewritten.
Re-engineered
The metastore itself: everything has to land in Unity Catalog with real owners and grants, and the tangle of partition columns is usually the moment to switch to liquid clustering. Oozie or shell orchestration becomes Lakeflow Jobs.
Ingestion path
Distributed copy to cloud storage, then register or convert in place; existing streams repoint at Structured Streaming.
Netezza & appliances nzsql · nzloadMigration path · Databricks
Typical pain
End-of-life hardware, no path to scale, and specialised skills getting harder to hire every year.
Ports cleanly
Star schemas, dimensional models and standard SQL transforms move over. Lakebridge covers the dialect and the object inventory.
Re-engineered
Appliance-specific SQL, distribution and organising keys, and stored procedures get re-expressed. Physical tuning knowledge doesn't transfer - and mostly doesn't need to, because clustering and optimization are automated here.
Ingestion path
Bulk extract via nzunload to cloud storage → Auto Loader into Bronze; incremental where the source allows.
Cloud warehouse Synapse · RedshiftMigration path · Databricks
Typical pain
Compute and storage coupled tighter than you'd like, awkward support for semi-structured and unstructured data, and a separate stack the moment ML enters the picture.
Ports cleanly
SQL-based models and views transpile well. If the data is already in cloud object storage in an open format, a lot of it can be read in place rather than moved.
Re-engineered
Warehouse-specific SQL and any procedural layer. The bigger job is usually consolidating the surrounding sprawl - the separate ML environment, the copies feeding it, the second governance model.
Ingestion path
Lakehouse Federation first so both are queryable from one place, then move workload by workload with unload-to-storage plus Auto Loader.
Legacy ETL Informatica · DataStage · Talend · SSISMigration path · Databricks
Typical pain
Per-connector licensing, opaque black-box jobs, and business logic locked in a visual tool that only two people can read.
Ports cleanly
Source-to-target mappings and straightforward SQL expressions translate directly. Lakebridge reads the job definitions and gives you a real inventory with complexity scores - often the first honest picture the team has had.
Re-engineered
Visual mappings, reusable transformations and proprietary functions get rebuilt declaratively and versioned in git. This is where the value is: the logic stops being a diagram in a licensed tool and becomes reviewable code.
Ingestion path
Point ingestion at the sources directly - Lakeflow Connect or Auto Loader - and retire the tool, rather than paying to run it against a new target.

Select a source system to see the typical pain, what ports cleanly, what has to be re-engineered, and the ingestion path onto Databricks.

The part that decides year two

A pipeline that works is easy. A platform that stays working is engineering.

Most Databricks estates we're called into don't have a technology problem - they have a software-engineering problem wearing a data costume. No environments, no code review, no cost attribution, and a bill growing faster than the value. This is the discipline we bring with us.

Everything as code, promoted not copied

Jobs, pipelines, schemas, permissions and dashboards live in one Databricks Asset Bundle in your repo. The same bundle is deployed to dev, staging and prod with different targets - so a change is a pull request, a release is a pipeline run, and a rollback is a git revert.

dev
per-engineer catalog, small compute
staging
prod-shaped data, full test run
prod
service principal, no human writes
Git folders for source control, service principals for prod - no notebook is the source of truth
Catalog-per-environment in Unity Catalog, so a dev job physically cannot write to prod
Expectations and data tests run in staging on prod-shaped data before promotion
Terraform for the account-level scaffolding around it - workspaces, metastore, network

The levers that actually move the bill

Databricks spend is mostly a design outcome, not a discount negotiation. These are the levers in the order we normally pull them - relative impact on a typical estate, biggest first.

Serverless & job compute instead of long-lived all-purpose clustersbiggest lever
Scheduled work on an interactive cluster somebody left running is the most common finding in an estate review, and the easiest to fix.
Incremental materialized views instead of full rebuildshigh
Recomputing a year of history every night to add one day of data is invisible on a dashboard and very visible on the bill.
Predictive optimization & liquid clustering instead of hand-tuned maintenancehigh
Let the platform decide when OPTIMIZE and VACUUM are worth running, and stop paying for nightly maintenance jobs that mostly do nothing.
The right freshness tier per table, not per platformmedium
Streaming everything because two tables need it is the most expensive architectural habit we see.
Photon and warehouse sizing matched to real query patternsmedium
Measured from query history rather than chosen once at setup and never revisited.
Retiring zombie streams, orphaned tables and unbounded statehousekeeping
Every estate over a year old has some. Nobody notices until someone goes looking.
Every job, pipeline and warehouse gets tagged and attributed from the system tables, with budget policies on top - so “which team spent this” has an answer before finance asks.
Millions of tiny files in Bronze
Auto Loader with a sane trigger interval, and compaction left to predictive optimization. One write per event is a metadata bomb, not a stream.
Partitioned by date, then region, then hour
CLUSTER BY - liquid clustering instead of partition columns. No cardinality trap, no rewrite when the query pattern changes, and automatic clustering on managed tables.
Production logic in an interactive notebook
Code in the repo, deployed as a Databricks Asset Bundle, executed by a service principal on job compute. Notebooks are for exploring, not for running the business.
Tables still in the legacy Hive metastore
Migrate them into Unity Catalog and switch the old metastore off. Two permission models running side by side is the same as having none.
No quality gate - users find the errors
Expectations on every Silver table: warn, drop the row, or fail the update. Quarantine bad records so somebody can look at them, rather than dropping them quietly.
One giant pipeline that does everything
One pipeline per domain, with clear table ownership and its own schedule - so a bad day at the CRM source doesn't hold up the finance close.
The stack

The Databricks data-engineering stack, end to end.

The platform's native engines at the core, the open formats underneath, and the tools we compose in around them when they fit what you already run. Filter by layer - this is representative of what a team actually standardizes on, not an exhaustive product list.

All layers
Ingest
Transform
Storage
Govern
Orchestrate & CI/CD
Serve & BI
AI & agents
Native · open · composed
Ingestin
LCLakeflow ConnectDatabricks-native
ALAuto LoaderDatabricks-native
ZBZerobus IngestDatabricks-native
CDCAUTO CDC flowsDatabricks-native
SSStructured StreamingOpen standard
KfKafka · Kinesis · Event HubsWe compose
FvFivetran · DebeziumWe compose
Transform & pipelinesbuild
LDPLakeflow Declarative PipelinesDatabricks-native
SkApache Spark · PySparkOpen standard
PhPhotonDatabricks-native
LDLakeflow DesignerDatabricks-native
dbtdbtWe compose
Storage & formatkeep
ΔDelta LakeOpen standard
IceManaged IcebergOpen standard
LQLiquid clusteringDatabricks-native
POPredictive optimizationDatabricks-native
S3S3 · ADLS · GCSWe compose
Governprove
UCUnity CatalogDatabricks-native
LinColumn-level lineageDatabricks-native
RLSRow filters & masksDatabricks-native
MVMetric viewsDatabricks-native
SysSystem tablesDatabricks-native
FedLakehouse FederationDatabricks-native
DSDelta SharingOpen standard
Orchestrate & CI/CDrun
LJLakeflow JobsDatabricks-native
DABAsset BundlesDatabricks-native
GitGit foldersDatabricks-native
TFTerraform providerOpen standard
AfAirflowWe compose
Serve & BIread
SQLDatabricks SQLDatabricks-native
WHSQL warehousesDatabricks-native
AIBIAI/BI dashboardsDatabricks-native
GenGenieDatabricks-native
AppDatabricks AppsDatabricks-native
LBLakebaseDatabricks-native
PBIPower BI · TableauWe compose
AI & agentsact
MAIMosaic AI Agent FrameworkDatabricks-native
ABAgent BricksDatabricks-native
VSVector SearchDatabricks-native
MSModel ServingDatabricks-native
MLMLflowOpen standard
FEFeature Engineering in UCDatabricks-native

Product names verified current at time of writing - Databricks renames fast, and a few of these have had two names in two years (Delta Live Tables → Lakeflow Declarative Pipelines; Workflows → Lakeflow Jobs). Items marked preview are exactly that: we'll tell you which parts of a design are resting on something not yet GA.

How the engagement runs

One real domain in production before we scale anything.

We don't sell a twelve-month foundation programme with value at the end. We build the foundation and one genuinely useful pipeline on it, together, so there's something in production to judge us by - then repeat the pattern domain by domain.

01
Weeks 1–2
Assess
Estate inventory and a Lakebridge Analyzer profile if there's a legacy source, target architecture, security and network model, cost envelope.
You getArchitecture & a costed wave plan
02
Weeks 2–4
Foundation
Workspaces, Unity Catalog metastore, catalog-per-environment, external locations and volumes, compute policies, the CI/CD skeleton and the first bundle.
You getA governed landing zone
03
Weeks 4–8
First domain, end to end
One real subject area from source to Gold: ingestion, declarative pipelines with expectations, the metric definitions, a dashboard people use.
You getA pipeline in production
04
Weeks 8+
Scale out & migrate
Domain by domain against the wave plan. Legacy workloads run in parallel and reconcile before anything is switched off.
You getCutover evidence, wave by wave
05
Ongoing
Run, tune, hand over
On-call and runbooks, monthly cost and performance review, optimization passes, and pairing with your engineers until they own it.
You getYour team running it

Indicative, not a template.

Two clean sources and a greenfield account move faster than this. A Teradata estate with thousands of BTEQ scripts, or an ungoverned Hive metastore with a decade of history in it, moves slower - and we'd rather say so in week one. The shape holds: foundation, one real domain, then repeat. What changes is how many waves and how long each takes.

In production

Governed ETL, feeding live BI - in production today.

In production

A large automotive supplier

We built governed ETL pipelines feeding live business intelligence for a large automotive supplier - data fresh enough for the business to act on, governed end to end. On the insurance side of the same relationship, we added intelligent document processing over ACORD forms, policy Q&A and inspection reports. One team, the platform and the pipelines, in production - not a pilot.

Anonymized production engagement. Named reference, platform detail and metrics available under NDA.

Why Focaloid for Databricks

Cloud and data, solved by one team.

Cloud + data in one team

In-house data architects and cloud architects who solve the platform and the cloud together - natively on AWS, Azure and GCP. Networking, identity, storage and the lakehouse designed as one thing, with no hand-off and no two vendors pointing at each other.

Databricks-native, and agent-ready

A Databricks Partner that also ships production agents. The governed foundation we build is the same foundation the agents we run read from - so the data and the AI on top come from one place, and the governance story is one story.

Migrations without the big bang

Since 2013, moving legacy estates off Oracle, Teradata, SQL Server, on-prem appliances and legacy ETL - assessed, transpiled, parallel-run and reconciled. You cut over on evidence, not optimism, and the old system gets decommissioned on purpose rather than forgotten.

Governed from the first table

Built for Unity Catalog from day one - catalog-per-environment, column-level lineage, row filters and column masks where the data needs them - and delivered by an ISO/IEC 27001:2022 certified practice. The data your BI and your AI depend on is data you can prove.

Partners & certifications
Databricks PartnerMember of the Claude Partner NetworkISO/IEC 27001:2022 Certified
Who it's for

Built for teams whose data is in the way of what's next.

We've bought Databricks and we're not sure we've set it up right.

Our data's scattered across systems and our AI project is stalled on it.

We're on Oracle and SQL Server, we know we need to modernize, and a big-bang migration terrifies us.

Our dashboards are always a day behind the business.

Our Databricks bill is growing faster than the value and nobody can explain why.

We need our BI, our data science and our AI to run off the same trusted numbers.

Usually a CTO, VP of Engineering, Head of Data or Data Engineering, a Chief Data Officer, or a data-platform lead - on Databricks across AWS, Azure and GCP, in the US, Europe and APAC.

Where this leads

The foundation is step one. Here's what it's for.

This is the bottom of the cluster - the foundation everything above it stands on. The natural sequence ascends from here: build the data, govern it, then put agents on top.

The natural sequence: Data EngineeringGoverned Data PlatformAgentic AI - foundation to outcome, one platform end to end.

Common questions

Before you book.

We've chosen Databricks. Where do you actually start?

With the foundation and one real domain, in parallel. The foundation is the part that's expensive to change later: the Unity Catalog metastore, a catalog-per-environment model, external locations and volumes, compute policies, identity, network and the CI/CD skeleton. The domain is one genuinely useful subject area taken from source to Gold with quality gates and a dashboard on the end. You get something in production in weeks rather than a foundation programme with value promised at the end - and the pattern that domain establishes is what every later wave copies.

Do we have to move everything at once?

No, and you shouldn't. We land your raw data first, rebuild the logic incrementally in declarative pipelines, and run old and new side by side until the numbers reconcile. Lakebridge's reconciler compares row counts, schemas and aggregates between source and target, so parallel run produces a signed-off difference report rather than an argument. You cut over domain by domain, on evidence - and only then does the old system get decommissioned.

Can Databricks work alongside our existing warehouse during a migration?

Yes - that's usually the sane on-ramp. Lakehouse Federation lets Databricks query your existing databases and warehouses in place, governed by Unity Catalog, without copying the data first. You get one query surface across old and new from early on, then move workloads across when each is ready instead of holding everything until a single cutover date.

Delta Lake or Iceberg?

Both are first-class now. Unity Catalog manages Delta and Apache Iceberg tables, and reads foreign Iceberg tables managed elsewhere - so the format is a compatibility decision about what else needs to read your data, not a lock-in decision. Either way you get the same governance, and the same automated maintenance: predictive optimization handling OPTIMIZE, VACUUM and ANALYZE, and liquid clustering instead of hand-tuned partitioning. Our default is Unity Catalog managed tables, and we'll pick the format from your interoperability requirements rather than a preference.

What about real-time?

There's a ladder, not a switch - and the honest answer is that most requirements stop on the second rung. Nightly batch, incremental micro-batch on a schedule or file-arrival trigger, continuous streaming through Structured Streaming or Zerobus Ingest for sub-five-second application events, and Real-Time Mode on declarative pipelines for millisecond end-to-end latency (in preview at the time of writing). The medallion architecture is the same at every rung; the trigger, the ingestion path and the cost change. We'll pick the cheapest tier that answers your question, and tell you what the next rung up would cost before you commit to it.

Is our data governed and secure?

Unity Catalog gives you one permission model, column-level lineage and audit across every layer, plus row filters, column masks and attribute-based policies where the data needs them. Data stays in your cloud account and your storage. We work to an ISO/IEC 27001:2022 certified practice, and the governance layer is deep enough that it's a page of its own: the Governed Data Platform.

How do you keep the Databricks bill under control?

Mostly by designing for it, then measuring it. Serverless or right-sized job compute instead of long-lived all-purpose clusters; incremental materialized views instead of full rebuilds; predictive optimization and liquid clustering instead of hand-tuned maintenance jobs; and streaming reserved for the tables that genuinely need it. Then everything is tagged and attributed from the system tables with budget policies on top, so spend has an owner and a trend line before finance asks about it.

Does this only make sense if we're doing AI?

No. The same platform powers your BI, your reporting and your data science - AI is one consumer of it, not a precondition. Plenty of clients build the data foundation first and add agents later, or never. It stands on its own; it just happens to also be the only sane place to put agents when you get there.

Do you work with our team, or replace them?

Work with them, and hand over. We pair with your engineers through the build, leave the whole thing as reviewable code in your repo with runbooks, and taper as your team takes over. Some clients keep us on for on-call and a monthly cost-and-performance review; some don't need us after the second wave. Both are a good outcome.

The next step

Build the foundation before you build the AI.

Start with a Databricks assessment - we'll map your current estate, the migration path off it, and the lakehouse architecture that feeds your BI, your data science and your agents from one governed source. You'll get an architecture, a costed wave plan and an honest view of what's hard.

Databricks PartnerLakeflow · Unity CatalogBatch & real-timeISO/IEC 27001:2022