Moving a live streaming estate onto private networking without a pause

Who

I led it, for 200+ teams of producers and consumers across every environment.

What

Moved the Kafka estate from public endpoints to Confluent Private Service Connect, and architected a real-time data exchange between isolated environments.

When

2025, cut over one team at a time, ahead of schedule.

Where

Rocket India, on Confluent Cloud and GCP.

Why

Public networking was a posture nobody wanted to keep defending, and isolated environments still needed to share data securely.

In 2025 I led the migration of Rocket India’s Kafka estate to Confluent Private Service Connect, moving every producer and consumer in every environment onto private networking, and separately architected a real-time data exchange platform for secure, low-latency sharing across isolated environments.

The estate was reachable over public networking, a security posture nobody wanted to keep defending and increasingly out of step with what 200+ teams needed. At the same time, some of those teams sat in genuinely isolated environments that still needed to share data with each other in real time, securely, without punching a hole through the isolation to do it.

• Nothing could take the estate down to make this change. Producers and consumers were live, in production, for 200+ teams.
• A migration that required a maintenance window would have been a much easier project and a much worse one to be on the other end of.
• Existing topics could not be re-keyed, so their data had to move to the new cluster with its partition assignment as it stood.

PSC meant a new cluster, so topic data was moved across to it, and I moved the estate incrementally, team by team, verifying zero data loss at each cutover rather than attempting a single cluster-wide switch, which kept the blast radius of any single cutover to one team rather than the whole estate.

For cross-environment sharing, I used Confluent cluster linking to replicate topics between clusters directly, with end-to-end lag, retry and DLQ monitoring built into the replication path itself rather than left to whichever team was consuming on the other side.

The full estate moved to private networking with zero data loss, ahead of schedule, cutting over one team at a time rather than the whole estate at once, and PSC is now the secure-by-default standard for 200+ teams. The cluster-linking based data exchange platform gives isolated environments a monitored, real-time path to share data without weakening the isolation that put them in separate environments in the first place.

Before and after.

• Before: producers and consumers reached the estate over public networking. After: private networking through PSC, in every environment.
• Before: a single cluster-wide switch was the obvious risk. After: one team per cutover, with zero data loss verified at each.
• Before: isolated environments could not share data in real time without weakening isolation. After: a monitored cluster-linking path.
• After: PSC is the secure-by-default standard for 200+ teams.

Skills this work shows.

• Network and security architecture: Private Service Connect on Confluent Cloud and GCP.
• Zero-downtime migration planning: incremental cutover with a one-team blast radius.
• Cross-cluster replication design: Confluent cluster linking with lag, retry and DLQ monitoring.
• Infrastructure as code: Terraform.
• Programme leadership across 200+ teams, delivered ahead of schedule.

I under-communicated the DNS behaviour early. Two teams debugged connectivity for a day before we produced a one-page explanation of what resolves where. A diagram on day one would have been worth more than the runbook we wrote on day ten.