perf(contacts): index contact search and enforce lowercase emails at the database level

[pausan]

Description

Contact search ran email ILIKE '%term%', which no index can serve, so every request scanned every contact in the project. At 2M contacts that is ~416 ms for the first page plus ~856 ms for the uncapped total — per debounced keystroke, across three UI surfaces (contacts page, command palette, segment contact picker), and the contacts page has no minimum search length.

Every application write path already normalized via ContactService.normalizeEmail, and 20260615120000_normalize_contact_emails repaired existing rows — but nothing enforced it, so search had to keep using ILIKE to defend against a raw write reintroducing mixed case.

This makes the invariant unconditional with a BEFORE INSERT OR UPDATE OF email trigger, which lets search drop to plain LIKE, and adds the two indexes the list and search queries need.

The trigger's WHEN clause is evaluated in C, so already-normalized input never enters PL/pgSQL, and UPDATE OF email keeps it clear of the far more frequent subscribed/snoozedUntil writes from the bounce, complaint and snooze paths.

Both indexes are needed — they cover disjoint cases:

	serves	write cost
(projectId, createdAt DESC, id DESC)	ordered walk for the unfiltered list and common terms	~9%
GIN (email gin_trgm_ops)	selective and zero-match search terms	~25%

Shipping only the btree regresses a zero-match search to 585 ms — worse than doing nothing — because with no trigram alternative the planner walks the ordered index across the whole project hunting for 21 rows that do not exist. Every prefix a user types before their term matches is a zero-match search. Shipping only the trigram leaves the unfiltered list untouched at ~189 ms, since that query has no email predicate for a trigram index to apply to.

The backfill fails loudly rather than silently merging case-variant duplicates: a correct merge has to reassign emails, events, workflow executions and segment memberships, and should not happen unreviewed at deploy time.

Before → after at 2M contacts

Postgres 16.15, en_US.utf8, shared_buffers=2GB, warm cache, median of 5 EXPLAIN ANALYZE runs. 1,999,962 contacts in the queried project (2.4M rows total). Synthetic corpus — the harness is included in this PR (yarn workspace @plunk/db bench:contact-search).

term	matches	page1	count
gmail.com	45%	416 → 0.37 ms	856 → 298 ms
ez (2 chars)	16%	385 → 0.99 ms	747 → 261 ms
nguyen	2%	345 → 3.2 ms	320 → 71 ms
martinez	2%	361 → 4.1 ms	345 → 76 ms
elena.pons	0.01%	352 → 10.2 ms	355 → 14 ms
zzqx	0	340 → 0.12 ms	348 → 0.11 ms
(no search — default list)	—	199 → 0.23 ms

Write side, 50k-row bulk insert, median of 3:

	rows/sec	vs before
before	41,828	—
after, pre-normalized input (production shape)	27,759	−34%
after, mixed-case input (trigger body runs)	26,376	−37%

Pre-normalized and mixed-case differ by ~2%, confirming the WHEN short-circuit works — essentially the whole write cost is index maintenance, dominated by the GIN index. Note this measures bulk insert, the worst case for GIN maintenance; trickle single-row inserts profile differently. fastupdate is on by default, and gin_pending_list_limit is the tuning lever if write throughput binds.

The count is the one thing not fully fixed: gmail.com stays around 300 ms because counting 900k matching rows means visiting 900k rows, and no index changes that. The drop from 856 ms is LIKE replacing ILIKE. Capping it (SELECT count(*) FROM (SELECT id FROM ... LIMIT 10000) t) measured 13.6 ms in spot checks, but that changes what the UI can display, so it is deliberately out of scope here.

Deployment note — build the indexes CONCURRENTLY first

This is an availability optimization, not a correctness requirement. migrate deploy produces exactly the same end state either way, and the migration is transactional, so a timeout part-way through rolls the whole thing back cleanly with no partial state. Fresh installs and development databases can ignore this section entirely.

What it buys you on an existing large contacts table is not blocking writes. Prisma runs a migration inside a single transaction and CREATE INDEX CONCURRENTLY cannot run there, so the plain CREATE INDEX takes SHARE and the CREATE TRIGGER takes SHARE ROW EXCLUSIVE, both held until commit:

operation on contacts	during the migration
SELECT	unaffected — neither lock conflicts with ACCESS SHARE
INSERT / UPDATE / DELETE	blocked until the migration commits

On a multi-million-row contacts the index builds take long enough to stall contact ingestion entirely — event tracking, imports and the API all write contacts. Blocked writes queue rather than fail, so without a lock_timeout they accumulate and can exhaust the connection pool, turning a slow migration into a wider outage.

CREATE INDEX IF NOT EXISTS is used specifically so you can avoid that: build both indexes concurrently ahead of the deploy and those statements become no-ops.

1. Against the primary, over the direct connection (not PgBouncer in transaction mode — CONCURRENTLY cannot run through it):

SET maintenance_work_mem = '2GB';

CREATE EXTENSION IF NOT EXISTS pg_trgm;

CREATE INDEX CONCURRENTLY IF NOT EXISTS "contacts_projectId_createdAt_id_idx"
  ON contacts ("projectId", "createdAt" DESC, id DESC);

CREATE INDEX CONCURRENTLY IF NOT EXISTS "contacts_email_trgm_idx"
  ON contacts USING GIN (email gin_trgm_ops);

2. Verify neither landed invalid. CONCURRENTLY can fail partway and leave an index that costs writes while serving no reads:

SELECT c.relname, i.indisvalid
FROM pg_class c JOIN pg_index i ON i.indexrelid = c.oid
WHERE c.relname IN ('contacts_projectId_createdAt_id_idx',
                    'contacts_email_trgm_idx');

Any indisvalid = false → DROP INDEX CONCURRENTLY and rebuild before continuing.

3. Deploy normally. migrate deploy runs the migration; the index statements no-op, and the trigger, backfill check and statistics target apply in milliseconds.

If the backfill's RAISE EXCEPTION fires, case-variant duplicates appeared after the 20260615 merge. The full runbook — including how to list them and the rollback order — is in packages/db/prisma/migrations/20260918120000_contact_email_lowercase/README.md.

Type of Change

• feat: New feature (MINOR version bump)
• fix: Bug fix (PATCH version bump)
• feat!: Breaking change - new feature (MAJOR version bump)
• fix!: Breaking change - bug fix (MAJOR version bump)
• docs: Documentation update (no version bump)
• chore: Maintenance/dependencies (no version bump)
• refactor: Code refactoring (no version bump)
• test: Adding tests (no version bump)
• perf: Performance improvement (PATCH version bump)

Testing

• Full suite passes: 47 files, 1210 tests.
• Three new tests cover the trigger itself via raw prisma.contact.create, which bypasses the service layer — lowercase on insert, lowercase on update, and collapse onto the unique constraint.
• One existing test changed. SegmentService's case-insensitivity test created two contacts differing only by case in the same project, which the trigger plus the (projectId, email) unique constraint now correctly reject — case variants are one person, as 20260615120000_normalize_contact_emails established. Rewritten to assert that a mixed-case operand still matches a single contact.
• prisma migrate diff --from-migrations --to-schema-datamodel reports no difference detected, so the declared indexes match the migration and migrate dev will not emit a spurious DROP/CREATE.
• Typecheck and ESLint clean.
• Benchmark harness added under packages/db/benchmarks/contact-search/, run by hand (not wired into CI — it seeds millions of rows and takes minutes).

A note on the schema declaration: a composite (projectId, email) GIN index via btree_gin measured 10–20% better than the plain trigram index, but text_ops is the default GIN opclass for text, so Postgres omits it from the catalog while Prisma requires it spelled out as raw("text_ops") — leaving the schema permanently drifted. Not worth that margin.

Checklist

• PR title follows conventional commits format
• Code builds successfully
• Tests pass locally
• Documentation updated (if needed)

Related Issues

Closes #488

[driaug]

Ideally we strive for a no-manual-intervention deploy which this index makes not possible. If someone does not read the instructions then it is possible that they lock their deployment.

Worst case we put it under a new major version but ideally we find a way to make it more fluid for self-hosters.

[pausan]

Thanks for your quick reply. I agree with your framing and your concerns. Having some "migration notes" that you need to do or remember to do in advance is not a good idea. Migrations should be automatic. Luckily, this migration is indeed automatic.

There's no manual intervention needed at all. There's nothing to be executed manually in advance; everything is prepared already to run this migration automatically without any prior manual work.

The reason there's an explicit mention to run these queries CONCURRENTLY in the PR is because on big databases it might take a while, and I thought, it might be a handy recommendation for really BIG databases. The limitation of not being able to add CONCURRENTLY to the migration itself, is because of Prisma running things in a single transaction.

As you can read in the PR, basically reads/SELECTs can still work while the automatic migration is applied, however, writes/updates/deletes will get locked during the duration of the automated migration. That's also a reason why I added this caveat. On small databases, locking writes/updates/deletes for few seconds should be totally fine. I was more thinking on a site that has a database with more contacts (e.g like maybe useplunk.com), not sure on how many contacts you might have, but if the database is big, like really big, having a lock on writes when running this migration concurrently might disrupt your clients, so it might be better to run in advance.

I just ran some benchmarks for 1M, 5M and 10M contacts, on how much time would the writes be locked on my personal computer (AMD Ryzen AI 9 365, 20 cores, 64GB RAM) . Even though computers might vary in performance, at least we can have a sense of magnitude.

Migration duration (non-concurrent)

Postgres 16, local NVMe, no concurrent load, shared_buffers=2GB. Applied as a single transaction exactly as Prisma does.

rows	table size	total	UPDATE scan	dup check	btree	GIN trgm	ANALYZE
1M	300 MB	7.3 s	0.40	0.31	0.76	3.88	1.86
5M	1.5 GB	30.0 s	1.77	1.68	4.30	20.04	2.10
10M	3.1 GB	54.1 s	3.24	3.31	7.46	37.78	2.19

By looking at this experiment/benchmark, it looks like the migration time grows linearly. 1 min migration for 10M rows does not seem excessive, but 100 million contacts would then take 10min of blocked writes on contacts table; maybe I was being extremely cautious when pointing out the CONCURRENT option, nonetheless again, no sure if you in your production server, or other people, will have 1M, 10M or 100M rows or more, thus, I'd rather edge on the side of caution and point it out.

Of course, we should consider this ran this benchmark on my personal computer, times will vary in different computers depending on available memory and CPU. Some servers will definitely be faster, some will definitely be slower.

Hopefully this clarifies this PR and unlocks it.

Happy to hear your thoughts.

[pausan]

@driaug I rebased this image on top of current next branch. Migration is automatic.

Have a look and let me know if there is anything for me to do on this branch or you think is good to go.