Capacity

What one host sustains, what gives first, and how to take the measurement again. Written for #355, whose problem statement was "nobody knows what fleet size this supports, and a customer finding out during a rollout is the worst way to learn".

Everything here was measured with tests/load_test.sh driving tools/loadgen. Every request in it is real: a real Ed25519 signature over the server's canonical string, a real nonce, a real batch inserted into a real PostgreSQL. The two things that would be cheapest to stub - the signature verify and the nonce insert - are exactly the two a capacity figure has to include.

---

1. The headline figure, and what it is worth

Two figures, because they answer two questions. Both measured after the per-event commit was removed (DLP-110, §2), both on a contended host.

	ingestion only	every event matches a policy
Sustained	3,385 events/s at p99 630 ms, concurrency 64	2,622 events/s at p99 283 ms, concurrency 16
Knee	concurrency 128: 2,588 events/s, p99 4,001 ms	concurrency 32: 2,281 events/s, p99 1,147 ms
Fleet, at 60 events/endpoint/hour	~203,000 endpoints	~157,000 endpoints
Fleet, at 600 events/endpoint/hour	~20,300 endpoints	~15,700 endpoints
Fleet, at 3,600 events/endpoint/hour	~3,385 endpoints	~2,622 endpoints
Errors	zero at every step, including past the knee	zero at every step
Foreign CPU during the run	~4,832% (48 cores' worth on 16)	~1,671%

What a policy match costs: at least 23% of throughput, and most of the headroom. The matching run carried a real decision on every event, so the server correlated, opened incidents and scored user risk inside the measured request - 827 incidents from 120,180 events. Throughput fell from 3,385 to 2,622, and the knee moved from concurrency 128 down to 32. The "at least" is load-honest: the ingestion-only run had three times the foreign CPU and still went faster, so the true gap on a quiet host is wider than 23%, not narrower.

The concurrency figure is the more useful half. Ingestion alone degrades gracefully and only past 64 in flight; the correlation path is queueing by 32. Whatever is sized from these numbers should be sized from the matching column, because a fleet that never matches a policy is a fleet nobody would buy.

Both are floors, not capacities. The measurement host has 16 cores and was running tens of cores' worth of unrelated work throughout, at a load average of 19-29. An earlier run of the identical sweep put concurrency 8 at 285 events/s rather than 713 - the same code, the same database, a different amount of somebody else's work on the same machine.

That variance is the reason tests/load_test.sh prints the load average beside every row and refuses to report a number without it. A capacity figure quoted without the contention that produced it outlives the conditions it was true under, and the person who repeats it will not know that.

For history, the pre-DLP-110 figure, taken the same way on the same contended host: 1,011 events/s at p99 587 ms, concurrency 16, with the knee at 32 where throughput fell to 724 events/s. Ingestion is 3.3x that now, under heavier contention, and the knee is two doublings further out.

What this figure does not cover, stated so nobody has to infer it:

• One server process and one PostgreSQL on one host. Nothing here says what two API replicas against one database do, and the WAL finding below suggests that is the interesting question. The harness can now run it (DLP_LOAD_REPLICAS, §4); no figure from it is recorded yet, because on this host one process against two would measure the other work sharing the cores (DLP-1180).
• Seconds per step, not hours. How growth, vacuum and retention interact with ingestion over a long run is not measured; the harness's soak mode (§4) is how to, on a quiet host.
• More than one policy. The matching column carries one decision per event for one published policy, so it measures correlation, incident creation and risk scoring at their simplest; a tenant with many overlapping policies pays more per event, and nothing here says how much. The ingestion column carries no decision at all. They are kept apart because a mix of the two answers neither question.
• Ten thousand devices are seeded and driven round-robin, so per-device quota behaviour is realistic. They share one keypair. The server still verifies every signature and still rejects every replayed nonce; what is not exercised is key diversity, which costs the server nothing - a verify is a verify.
• Enrollment is not measured. It is a one-off cost that says nothing about steady state.

The ticket's acceptance criterion, by arithmetic from the floors. It asks for 10,000 endpoints at a realistic rate with p99 under a second. At 600 events per endpoint per hour that is 1,667 events/s, which both columns clear below their knee - the matching one at p99 283 ms. At 3,600 per endpoint per hour it is 10,000 events/s, which one host does not reach. An earlier version of this paragraph said 600 did not clear; that was arithmetic from the pre-DLP-110 1,011 events/s and was not updated when the figure moved. None of it has been run as 10,000 endpoints on a quiet host, which is the measurement to take before quoting a number to a customer (DLP-1180).

2. What gives first: one commit per event

Sampling pg_stat_activity while the load ran, across every step:

   117  LWLock:WALWrite
    13  IO:WALSync
     2  Client:ClientRead
     1  LWLock:BufferContent
     1  IO:WALWrite

The database spends its time writing the write-ahead log. The cause is in apps/management-api/src/domain/ingest.rs: store_event is called once per event with the pool rather than a transaction, so every event in a batch is its own transaction and its own commit. A twenty-event batch costs twenty commits, plus the batch-idempotency row, the quota row and the watermark update.

Two observations support that reading over the obvious alternative:

• It is not disk fsync. Setting synchronous_commit = off - a measurement, never a recommendation - moved throughput from 3,506 to 3,788 events/s, about 8%. If the bottleneck were waiting for the disk, removing the wait would have transformed it. The contention is on inserting into the WAL buffer, which is CPU and lock, not IO.
• Throughput does not follow concurrency. 713 events/s at 8, 1,011 at 16, and down to 724 at 32 while p99 quadruples. Adding load adds queueing, not work done. That is serialisation on a shared resource, not saturation of one.

Tracked as DLP-110. A batch is already atomic from the agent's point of view - one batch_id, one idempotency record, one response - so committing it once is a change to how it is stored rather than to what it means.

Fixed, and what it was worth

store_event now takes the batch's transaction instead of the pool, and the watermark update and the idempotency record went in with it. One commit for the batch, rather than one per event plus three.

Measured by counting, not by timing. tests/ingest_commits.sh reads xact_commit for the database before and after a fixed run. A count is unaffected by whatever else this host is doing, which is exactly the objection that makes every throughput figure here a floor.

Two changes, measured together in one window so the host is the same for all three rows:

	25 batches x 20 events	per batch
Before	965 transactions	~39
One transaction per batch	656	~26
Plus: no correlation read for an event no policy matched	271	~11

The first change removed the per-event commits. The second removed the last per-event read: correlate_event fetched the stored event only to see it had no policy id and return - a round trip to the database to be told what the agent had already said in the event it sent. That is most of the traffic. A fleet reports file activity constantly and matches a policy rarely; in this run 500 events produced 500 such lookups and zero incidents.

Together: 965 to 271, a 72% reduction, and nothing now scales with the number of events in a batch. What is left is the request's own cost - a replay-protection nonce insert, two rate-limiter updates and the stored-events counter, each once per request however many events it carries. That is the next thing to batch, and it is a request-level concern rather than an ingestion one.

Throughput, with the usual caveat. Runs interleaved before/after/ before/after at concurrency 16, on the same contended host, load average 18.8 rising to 22.9 across the set:

	events/s	p50
after	1,578	182 ms
before	1,188	258 ms
after	653	369 ms
before	472	650 ms

The absolute numbers move by a factor of two between the pairs, which is the host and not the code. Within each pair the improvement is 33% and 38%, with p50 latency down in both - and that within-pair comparison is the only part of this table worth quoting.

What it cost in behaviour

A storage failure now fails the whole batch rather than one event in it. The partial-acceptance concern noted above turned out not to need savepoints: malformed events are rejected by validation before any SQL runs, so the transaction is never poisoned by one and the other nineteen still store. Only a genuine database error aborts the batch - and Postgres has already aborted the transaction at that point, so continuing would produce a run of misleading per-event errors and then roll back anyway. Every event is marked for retry, including those that inserted before the failure, because the rollback took them with it and an ack saying "accepted" for an event that is not in the database is the one lie this path must never tell.

3. The second thing to look at

Not yet a bottleneck, and worth knowing before it becomes one:

• request_nonces did not keep up, and neither did event_batches. Confirmed rather than assumed, and the assumption was wrong. Housekeeping ran every 300 seconds and deleted at most 10,000 rows per table per pass: 33 rows a second, against a measured 169 requests a second, each of which writes one nonce and one batch idempotency record. Above roughly 33 requests a second - 667 events a second at the default batch size, well under any figure in §1 - both tables grew for ever.

  The failure was invisible in exactly the way that matters: every pass logged a healthy nonces=10000 while falling further behind. A prune that always removes its maximum is not evidence that it is keeping up; it is the signature of a prune that is not.

  Fixed: a pass now repeats the batched delete until a batch comes back short, capped at 64 batches with a warning if the cap is reached. Batching still keeps one statement from locking the table. Measured against a seeded backlog of 25,000 expired nonces: before, one pass removed 10,000 and left 15,000; after, it removed all 25,000.

• Correlation runs inline in the ingestion path (DLP-TASK-050), and §1 now measures it: tests/load_test.sh with DLP_LOAD_MATCH_POLICY=1 makes every event carry a decision, so correlation, incident creation and risk scoring all happen inside the measured request. The count of incidents opened is printed beside the throughput, so the figure can be checked rather than believed.

• Database size grew to 41 MB for 48,960 events - about 880 bytes an event including indexes. A million events is under a gigabyte, so storage is not the constraint; retention (#371) is about the clock, not the disk.

4. Taking the measurement again

cargo build --release -p dlp-management-api -p dlp-loadgen

sudo -u postgres createdb dlp_load
sudo -u postgres env \
  DATABASE_URL='postgres:///dlp_load?host=/var/run/postgresql' \
  bash tests/load_test.sh 10000 20        # devices, seconds per step

# The same sweep, with every event matching a policy, so the number includes
# correlation, incident creation and risk scoring.
sudo -u postgres env \
  DATABASE_URL='postgres:///dlp_load?host=/var/run/postgresql' \
  DLP_LOAD_MATCH_POLICY=1 \
  bash tests/load_test.sh 10000 20

# Two API processes against the one database (DLP-1180). The generator
# spreads its workers evenly over them, and every result row carries how
# many requests each one served; a step where a replica served nothing says
# so, because that figure is one server's.
sudo -u postgres env \
  DATABASE_URL='postgres:///dlp_load?host=/var/run/postgresql' \
  DLP_LOAD_REPLICAS=2 \
  bash tests/load_test.sh 10000 20

# A soak instead of a sweep: one concurrency held for two hours, a row per
# step with the database size and dead rows beside it, then the first, last
# and lowest throughput. Pick the concurrency from a sweep's knee.
sudo -u postgres env \
  DATABASE_URL='postgres:///dlp_load?host=/var/run/postgresql' \
  DLP_LOAD_SOAK_MINUTES=120 DLP_LOAD_SOAK_CONCURRENCY=16 \
  bash tests/load_test.sh 10000 60
sudo -u postgres dropdb dlp_load

The replicas are started one after another, so the first applies migrations alone. Each port is checked free before a replica starts on it: a server left over from an earlier run would otherwise take this run's load and report it as ours. Replicas share one host here, so a two-replica figure from one machine says whether the database or the process gives first, not what two machines would do.

The harness raises the rate limits and ingestion quotas for the run. They are a real control and they work - tests/rate_limit.sh proves it - but a capacity measurement that spends its time being refused measures the refusal.

It sweeps concurrency until p99 crosses one second, then takes one more step so the shape past the knee is on the record. "It got slower" and "it fell over" are different failures and an operator heading for one needs to know which.

Read the load column. If it is not close to the core count, the numbers are about the machine rather than the product.