How Cloudflare Scaled Security Scans 10x Without Breaking Kafka

Cloudflare's Security Insights feature runs automated scans across every customer account, zone, and DNS record to flag misconfigurations. The problem? Scans ran every week or two, and many free-tier accounts weren't scanned at all. To close the gap, the team needed to go from 10 scans per second to 100 — a 10x increase — while the existing system was already drowning in timeouts, crashing processes, and a backlog of millions of events.

Kafka: not a queue, but close enough

The scanning pipeline is triggered by a scheduler that publishes messages to Apache Kafka. Specialized Go microservices (checkers) consume those messages and write results to a Postgres database via an internal API. Kafka's partitioned model means only one consumer per partition per consumer group, and messages must be processed in order within a partition. That creates two bottlenecks: slow messages block the consumer, and the number of consumers is capped by the number of partitions.

Rather than adding partitions (which would strain the shared Kafka broker), the team introduced parallel processing. Checkers now consume messages in batches and process each one in a separate goroutine. The trade-off — more work to redo on crash, slightly higher memory — was acceptable.

Fast lane, slow lane

Some scans take minutes or hours (accounts with thousands of assets) while most take milliseconds. To avoid head-of-line blocking, the team split each checker into two consumer groups: a 'fast lane' and a 'slow lane'. The fast lane skips messages that look expensive; the slow lane handles them with dedicated resources. Simple, effective.

Database writes: from 500,000 round trips to milliseconds

The original code inserted each insight with a separate SQL INSERT ... ON CONFLICT DO UPDATE — up to 500,000 round trips per API call. The team tried Postgres COPY into a temporary table, but that caused system table bloat. Their hybrid solution: use UNNEST for small batches, COPY for large ones. Result: inserts that took seconds for huge sets, milliseconds for small ones.

The latency that killed throughput

The primary database sat in Portland, Oregon, but the API ran active-active in both Portland and Amsterdam. Even at light speed, the round trip between Amsterdam and Portland added ~50ms. That doesn't sound terrible, but with high request volume, the connection pool in Amsterdam quickly exhausted, causing timeouts. Worse, Kafka partitions assigned to Amsterdam-bound consumer processes lagged badly — exactly half the partitions fell behind. The fix: switch the API to active-passive, routing all traffic to the Portland instance. Latency dropped from ~3 seconds to ~10ms.

Scheduler: stop the stampede

The original scheduler looped over accounts where last_scheduled_at + frequency <= now. Because many accounts shared the same last_scheduled_at, scans arrived in huge spikes. Large accounts with many zones caused cascading floods. The team made three changes: schedule zones independently (each zone gets its own last_scheduled_at), randomize the initial timestamp for existing accounts and zones, and add adaptive rate limiting to scheduling. Scans now spread uniformly over time.

The result: a 10x throughput increase, automatic scanning enabled for millions of accounts, and scanning frequency doubled for everyone. No new partitions, no new hardware — just careful engineering.