Kafka vs Databricks: when to use each (decision framework)

The question that comes up in every architecture review

Sooner or later in data engineering someone asks: "Do we actually need Kafka here, or can Databricks just read it directly?" The answer is rarely obvious from a whiteboard, and getting it wrong costs real money in infrastructure, operational overhead and engineering hours spent fighting an architecture that was never right for the workload.

This article is the framework I use to make that call quickly, with concrete criteria rather than gut feel. A note on where it comes from: my own experience is Confluent Cloud for Kafka, a self-hosted Kafka Connect, and Databricks on a regulated customer data platform. I have not run self-managed Kafka brokers, so anything here about Databricks real-time mode comes from the vendor documentation, which I link at the end.

First, get the mental model right

These tools are not competitors. They solve different problems in the same data stack. Kafka is a distributed commit log, a durable, ordered transport layer for events. Databricks is a compute platform built on Apache Spark, designed for transformations, aggregations and machine learning at scale. The typical production architecture has them working together:

Source systems → Kafka → Databricks → Data warehouse → Applications

Kafka moves data. Databricks processes it. The confusion arises because Databricks can read a database directly over JDBC, Kafka can do lightweight transformations with Kafka Streams, and Databricks Structured Streaming can now read from and write to Kafka with very low latency. Each of those capabilities exists. None of them replaces what the other tool does well.

The third piece is ingestion, specifically the difference between CDC-based and polling-based approaches, which changes the cost and reliability equation significantly.

Where each tool sits on the latency ruler

Latency is where most of these debates start, so it helps to see the ranges side by side.

Two things to notice. First, the old rule that Databricks cannot go below a few seconds is out of date. Databricks documents a real-time mode for Structured Streaming with end-to-end latency as low as five milliseconds, and it reads from and writes to Kafka. Second, it is a different operating mode with different costs, covered below.

When Kafka is the right choice

Kafka's core value proposition is not throughput. It is durability plus fan-out plus replay in a single system. Together those three properties are hard to get from any other component in a typical data stack.

Multiple independent consumers on the same data stream. Kafka's consumer group model means ten services can read the same topic at their own pace, each maintaining its own offset. With direct database reads or file-based ingestion, adding a new consumer means another query against the source, another pipeline to maintain and another point of failure. With Kafka, adding a consumer means adding a consumer group, with no upstream changes.

Event replay and audit requirements. Kafka retains the log for a configurable window: hours, days, or indefinitely with tiered storage. When a downstream job fails, a new consumer comes online or a bug is found in processing logic, you can replay from any offset. That is not possible with JDBC ingestion or most file-based approaches once the source data changes.

Event-driven architecture with decoupled producers and consumers. If more than three services need to react to the same business events (order placed, payment processed, user registered), Kafka is the right distribution mechanism. Direct service-to-service calls or shared database writes create coupling that gets more expensive to unwind as the system grows.

Low latency delivery with a simple path. Kafka brokers typically deliver to consumers in tens to a couple of hundred milliseconds in well-tuned clusters. If a fraud model, a live alerting pipeline or a user-facing feature needs results in under a few seconds, Kafka should be in the path. Databricks real-time mode can also meet that bar for processing, but it still reads from a transport like Kafka, so the question becomes whether you want the processing in Databricks, in a lightweight consumer, or in Kafka Streams.

When Databricks is the right choice

Databricks earns its place in the stack when the work is genuinely computational: complex joins, aggregations across large windows, ML feature engineering, medallion architecture transformations, or anything that needs multi-step logic across historical data.

Complex transformations and aggregations. Kafka Streams can filter, map and do simple stateful aggregations. It is a poor fit for a many-way join across a multi-terabyte historical dataset while a feature engineering pipeline runs for a recommender model. Spark is built for that. Databricks is the right answer for any transformation that needs more than a few operations or touches data at terabyte scale.

Batch workloads with hourly or daily schedules. If a business report runs at 6 AM and nobody is waiting on it at 6:01 AM, there is no business case for a streaming pipeline. A scheduled batch job is simpler to operate and cheaper than continuous streaming infrastructure for the same workload, because compute only runs while the job runs.

Delta Lake and the lakehouse pattern. ACID transactions, time travel, schema evolution and data layout optimisation are Delta Lake capabilities that make Databricks the right home for curated data layers. The medallion pattern (raw bronze, validated silver, aggregated gold) is well proven and Databricks implements it well. Kafka is not involved in those layers.

Machine learning pipelines. Feature engineering at scale, model training on historical data and batch inference belong in Databricks. Kafka's role ends when the events are delivered. Databricks takes over when the computation begins.

What real-time mode changes, and what it costs

Databricks documents a real-time mode for Structured Streaming that processes data as it arrives rather than in periodic batches, with reported end-to-end latency as low as five milliseconds. It is worth knowing the constraints from the documentation before you rely on it:

• Dedicated compute. It schedules all stages of the query simultaneously, so the number of available task slots must be at least the number of tasks across all stages. You are paying for capacity that stays up, not capacity that spins up per batch.
• Update output mode only. Append and complete modes are not supported.
• Narrower feature support. The reference lists Kafka as a supported source and sink, and Event Hubs and Kinesis as sources, but not Delta as a sink, not serverless compute, and not session windows or stream-to-stream outer joins.
• Language and runtime caveats. It supports Scala, Java and Python, with Python at low throughput noted as seeing latency of up to a few hundred milliseconds on older runtimes.

The practical reading: real-time mode makes Databricks a credible choice for low-latency processing that Kafka alone cannot do well, such as stateful enrichment. It does not remove the reasons to have Kafka in the path, which are fan-out, replay and decoupling. Check the current release status in the documentation before putting it on a critical path.

The ingestion question: CDC vs JDBC

Once you decide whether Kafka is in the path, the next question is how data gets into the pipeline from relational source systems. This is where teams make expensive mistakes most often.

Log-based CDC with Kafka Connect and Debezium reads the database transaction log directly. Every insert, update and delete is captured as an event in a Kafka topic with sub-second latency from the original commit. There is no query load on the source database during steady-state operation, and deletes are captured reliably. It is the right approach for production databases with high write rates, large tables, or workloads that need true real-time replication. The operational cost is higher: you configure and maintain Debezium connectors, ensure the database grants log access, and manage schema evolution through Schema Registry. At scale the reliability and efficiency advantages are substantial.

Kafka JDBC source connector polls the source database on a configurable interval, running a SQL query for rows where an incrementing column or timestamp has advanced. It is simpler to set up and works without transaction log access. The trade-offs are meaningful: polling creates periodic load on the source, delete events are not captured, and latency is bounded by the poll interval. Teams often stretch that interval to minutes to avoid overwhelming the source. It suits cases where several downstream consumers need the same database data buffered in Kafka and minutes of latency is acceptable.

Databricks JDBC direct reads connect a Spark cluster directly to the source database and read on demand during job execution, with parallel partitioning for throughput. There is no intermediate buffering, no replay and no fan-out. If three teams need the same data, each runs its own query. For single-consumer batch workloads with hourly or daily schedules this is the simplest and often cheapest architecture. It becomes the wrong choice when the database cannot sustain the query load, when several consumers exist, or when the latency requirement drops below the job schedule interval.

Both JDBC approaches put query load on source databases. For production OLTP systems with significant write rates or large tables, log-based CDC through Debezium is the right long-term approach. The short-term cost of setup is lower than the long-term cost of polling at scale.

Decision matrix

The table summarises which approach fits which requirement. "Recommended" means the natural fit. "Viable" means it works with trade-offs worth understanding. "Avoid" means the architecture fights the tool's design.

What this costs: how to think about it

I am deliberately not quoting dollar figures here. Cost depends on cloud, region, instance types, throughput, retention and team size, and a generic number in an article goes stale quickly. Use the vendors' own pricing calculators for Confluent Cloud and Databricks with your real volumes. These are the levers that decide the outcome:

• Always-on versus scheduled. A Kafka cluster plus continuous streaming consumers runs around the clock. A scheduled batch job pays for compute only while it runs. If nobody needs data fresher than the schedule, the continuous option is paying for latency nobody uses.
• Real-time mode capacity. Because it holds task slots for every stage at once, real-time streaming is sized for peak and stays on. Budget it as standing capacity.
• Managed versus self-managed Kafka. A managed service trades a per-throughput premium for operational leverage. The honest comparison includes engineering time spent on upgrades, rebalancing and on-call, not only the infrastructure invoice.
• Source database load. Polling queries and full scans consume source capacity that is often the most expensive capacity you own. CDC avoids it.

The single most common cost mistake is standing up streaming infrastructure for a workload whose consumers would not notice the difference between five seconds and ten minutes. If your reporting dashboard refreshes every 15 minutes, that is a batch workload.

Five patterns that get teams into trouble

Assuming micro-batch behaves like a message broker. Default Structured Streaming is a micro-batch engine. In practice its end-to-end latency includes scheduling and checkpoint writes on top of the trigger interval, so teams that set a 500 ms trigger and expect Kafka-like delivery are surprised. If you need millisecond-level processing, either move to real-time mode with its dedicated compute and constraints, or keep that path in Kafka and a lightweight consumer.

Using Kafka for complex transformations. Kafka Streams handles stateless and simple stateful transformations well. Multi-way joins across large datasets, ML inference pipelines and complex windowing against historical data are not what it is designed for. Teams that push this logic into Kafka end up with hard-to-debug topologies and state store management problems that would not exist in Databricks. Move complex logic downstream.

Running JDBC full-table scans on production databases. A full scan of a large production table over JDBC creates substantial I/O and CPU load on the source. Scheduled at 2 AM it is manageable. Scheduled hourly it can degrade production traffic. Scheduled every 15 minutes it causes incidents. Log-based CDC reads the transaction log rather than querying the table, has little steady-state impact on the source and captures deletes, which polling cannot.

Building a separate pipeline per consumer instead of a shared topic. When three teams each build their own JDBC job against the same source table, you have three queries hitting that table, three pipelines to maintain, three latency profiles and three failure modes. One Kafka topic with three consumer groups is strictly better: one query or one CDC connector on the source, identical events for every consumer, independent offsets, and a single place to monitor ingest.

Introducing Kafka because "we might need real-time someday". Kafka adds operational complexity. Running it well means understanding partition rebalancing, consumer lag monitoring, schema evolution, connector management and retention policies. If today's requirement is hourly batch and next year's might be five-minute near-real-time, build the batch pipeline. Moving from Databricks batch to Kafka-backed streaming later is a well-understood exercise. Paying for streaming infrastructure a year before you need it is waste.

Six scenarios with concrete recommendations

Real-time database replication to a data lake. You need database changes in your lakehouse within seconds. Use Kafka Connect with Debezium for CDC into Kafka topics, consumed by Databricks Structured Streaming into the Delta bronze layer. Debezium captures every change with sub-second latency and little database overhead, and Databricks handles the transformation and merge logic through your medallion layers.

Daily log processing for BI dashboards. You have large daily application logs to transform for a BI layer. Use Databricks only: batch jobs scheduled off-peak, writing to a gold Delta table that the BI tool reads. Adding Kafka would add cost and complexity without improving the outcome for people who refresh dashboards once a day.

SQL Server to Snowflake every four hours, single consumer. One team, one destination, four-hour cadence. Use Databricks JDBC direct reads with partitioned parallel reads into Snowflake through the Spark Snowflake connector. Simple, cheap, easy to maintain, and no Kafka required.

Payment event fraud detection in under a second. The model must evaluate each payment in well under a second. Use Kafka as the transport, with a lightweight scoring service consuming the payments topic. If the scoring needs heavy stateful enrichment, Databricks real-time mode is worth evaluating against its constraints, but the transport stays Kafka.

User clickstream for ML feature engineering. Clicks must be captured in real time and serve both live personalisation and offline model training. Use Kafka for capture and distribution, a real-time consumer for personalisation, and Databricks consuming the same topic for feature engineering into a feature store. Both paths read one topic without duplicating ingestion.

Schema evolution with audit requirements. A regulatory workload requires every version of every event to be preservable while the schema evolves. Use Kafka with Confluent Schema Registry: Avro or Protobuf schemas registered centrally, Kafka's log retention as the historical record, and compatibility rules that let consumers evolve safely.

The decision in four questions

When the architecture debate starts, these four questions cut through most of the ambiguity.

1. What is your actual P95 latency requirement? Not what you might need eventually, but what the business needs today, measured at the consumer end. If it is above five minutes, you have a batch workload. If it is under five seconds, you want Kafka in the path.

2. How many independent consumers will read this data? If the answer is one, JDBC or direct ingestion is probably enough. If it is three or more, Kafka's fan-out pays for itself quickly in reduced source load and eliminated pipeline duplication.

3. Do you need to replay events? If a consumer fails and you need to reprocess the last 48 hours, Kafka's log retention gives you that. JDBC and file-based approaches do not. If replay is a hard requirement, Kafka is structurally necessary.

4. What is the transformation complexity? If the work is filtering and routing, Kafka Streams is fine. If it is multi-table joins, aggregations across days of history or ML inference, it belongs in Databricks. Both can coexist: Kafka handles delivery and Databricks handles computation.

The architecture most platforms converge on

The pattern I see recur is a three-path architecture.

• Hot path (sub-second to 5 seconds): Kafka producers → Kafka topics → stream processing → operational databases or caches → real-time user-facing features.
• Warm path (5 seconds to 5 minutes): Kafka topics → Databricks Structured Streaming → Delta silver → near-real-time analytics.
• Cold path (hourly to daily): cloud storage or scheduled JDBC jobs → Databricks batch → Delta gold → data warehouse → BI tools.

Not every platform needs all three paths. Start with the cold path. It is the cheapest to operate and covers most business intelligence workloads. Add the warm path when a critical use case needs minute-level freshness. Add the hot path when a product feature genuinely needs sub-five-second delivery. That sequencing keeps cost and complexity in step with the business value delivered at each stage.

Conclusion

The Kafka vs Databricks question is rarely about which technology is better. It is about which problem you are solving. Kafka moves events reliably to many consumers with durable replay. Databricks transforms data at scale with rich compute. They are answers to different questions that arise at different points in the same pipeline.

The practical rule: if you have more than two independent consumers, need replay, or need delivery in a few seconds, Kafka is probably justified. If you are doing batch processing with a single consumer and no replay requirement, Databricks alone with JDBC ingestion is likely cheaper and simpler. Both over-engineering (paying for streaming you do not use) and under-engineering (batch pipelines missing a latency target) show up in engineering hours and invoices. Pick the architecture that matches what the business needs today, and plan the path to the next tier for when requirements change.

Frequently asked questions

Do I need Kafka if I already have Databricks?

Only if you need something Databricks alone does not give you: three or more independent consumers on the same stream, replay from an arbitrary offset, or producers and consumers decoupled in time. If none of those apply and your freshness need is minutes or longer, Databricks with JDBC or file ingestion is cheaper to run and simpler to operate.

What latency can Databricks Structured Streaming achieve?

It depends on the mode. Default micro-batch processing typically lands in seconds to tens of seconds end to end, once scheduling and checkpoint writes are counted. Databricks also documents a real-time mode that reports end-to-end latency as low as five milliseconds, but it needs dedicated compute that schedules all stages at once, supports update output mode only, and has operator limits. Check its release status and limitations before designing around it.

Should I use log-based CDC or the Kafka JDBC connector?

Use log-based CDC through Debezium for production databases with high write rates, large tables or a real-time requirement. It reads the transaction log, adds no steady-state query load and captures deletes. The JDBC source connector polls, so it misses deletes and its latency is bounded by the poll interval you configure, which teams often set to minutes to protect the source.

What does choosing the wrong architecture cost?

It shows up as always-on streaming infrastructure serving a workload a scheduled batch job would handle, or as batch pipelines missing a latency target. Costs vary widely by cloud, region and team, so model your own numbers with the vendors' pricing calculators instead of trusting a generic figure.