feat: migrate push to SSE with WebSocket rollout compatibility

[niemyjski]

Moves Svelte real-time updates to authenticated SSE while preserving WebSocket delivery for legacy Angular and cached clients. Both transports keep the existing snake_case payloads. Replica-local routing prevents one server from consuming another server's token revocation; expiring Redis leases bound shared connection counts after replica loss. The SSE client now uses the configured API origin, avoiding the frontend dev proxy used by browser runs that repeatedly received push 429s. Latest main is integrated.

Validation on the final head: 3,018 hosted API tests passed (three skipped), and all 77 browser tests passed without retries, including the saved-view flows that previously failed with repeated push 429s. Website, client, Docker, version, and CLA checks passed. Locally, 792 frontend unit tests, npm run validate, production build, the isolated Chromium SSE-abort regression, and website verification passed; the focused SSE client suite passed 21/21 after the origin change.

Not merge-ready yet: local service-backed dogfood awaits an exclusive Aspire runtime slot. The PR remains draft and requires reviewer re-approval. The deployment chart keeps push disabled pending ingress streaming validation. The website workflow now pins Deno 2.9.7 to run the inherited Lume 3.3.1 toolchain.

Implementation and verification details

• /api/v2/push accepts authenticated SSE and retains WebSocket upgrades. EnableWebSockets remains an alias for EnablePush.
• Bounded SSE queues coalesce repeated invalidations and prioritize critical notifications. Delivery has no replay; clients refresh current state after reconnecting.
• OAuth revocation and membership removal update local connection access without expanding scoped-token grants. Tests cover shared-bus fanout and owner-only revocation across two simulated replicas.
• Redis leases use atomic limits, renewal, and expiry; the in-memory fallback is process-local. Shutdown cleanup and transport metrics support rollout.
• HTTP examples and OpenAPI describe the SSE response and authentication/error statuses.
• All 37 review threads were inspected and are resolved. Historical CHANGES_REQUESTED remains for the reviewer to revisit.
• The preceding hosted browser run passed with three retries in chart, invitation, and organization-navigation tests; the final-head run passed all 77 tests without retries.

[Copilot]

Pull request overview

This PR migrates real-time push from WebSockets to SSE while keeping a temporary WebSocket compatibility path for rolling deploys, updates both frontends to consume SSE, and adds rollout/shutdown configuration for long-lived push connections.

Changes:

• Adds SSE connection, middleware, broker fanout, diagnostics, and EnablePush configuration.
• Replaces Svelte/Angular WebSocket clients with fetch/stream-based SSE clients and updates tests.
• Adds Kubernetes drain settings and documents push being disabled until ingress supports SSE.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/Exceptionless.Web/Hubs/SseConnection.cs	Adds bounded SSE write queue, deduplication, keep-alives, and disposal.
src/Exceptionless.Web/Hubs/SseConnectionManager.cs	Manages SSE connections, keep-alives, cleanup, and metrics.
src/Exceptionless.Web/Hubs/SseMiddleware.cs	Adds authenticated /api/v2/push SSE endpoint.
src/Exceptionless.Web/Hubs/WebSocketPushMiddleware.cs	Adds temporary WebSocket compatibility middleware.
src/Exceptionless.Web/Hubs/WebSocketConnectionManager.cs	Reworks WebSocket manager for compatibility fanout and metrics.
src/Exceptionless.Web/Hubs/MessageBusBroker.cs	Fans out push messages to SSE and WebSocket managers.
src/Exceptionless.Web/Hubs/MessageBusBrokerMiddleware.cs	Removes old WebSocket-only middleware.
src/Exceptionless.Web/Startup.cs	Wires SSE and WebSocket push middleware behind EnablePush.
src/Exceptionless.Web/Program.cs	Adds host shutdown timeout for drain alignment.
src/Exceptionless.Web/Bootstrapper.cs	Registers the SSE connection manager.
src/Exceptionless.Web/Utility/Handlers/ThrottlingMiddleware.cs	Renames push throttle exemption to SSE-oriented naming.
src/Exceptionless.Core/Configuration/AppOptions.cs	Replaces EnableWebSockets with EnablePush and legacy fallback.
src/Exceptionless.Core/Bootstrapper.cs	Updates push-disabled startup warning.
src/Exceptionless.Core/Utility/AppDiagnostics.cs	Adds SSE/WebSocket push connection counters.
src/Exceptionless.Web/ClientApp/src/lib/features/websockets/sse-client.svelte.ts	Adds Svelte SSE client with reconnect and stream parsing.
src/Exceptionless.Web/ClientApp/src/lib/features/websockets/sse-client.test.ts	Adds Svelte SSE client unit tests.
src/Exceptionless.Web/ClientApp/src/lib/features/websockets/web-socket-client.svelte.ts	Removes Svelte WebSocket client.
src/Exceptionless.Web/ClientApp/src/lib/features/websockets/web-socket-client.test.ts	Removes WebSocket client tests.
src/Exceptionless.Web/ClientApp/src/routes/(app)/+layout.svelte	Switches app layout push integration to SSE.
src/Exceptionless.Web/ClientApp.angular/components/websocket/websocket-service.js	Rewrites Angular legacy push service to SSE.
tests/Exceptionless.Tests/Hubs/SseTests.cs	Adds SSE manager, broker, and deduplication tests.
tests/Exceptionless.Tests/Hubs/SseIntegrationTests.cs	Adds HTTP pipeline tests for SSE endpoint behavior.
tests/Exceptionless.Tests/Hubs/FakeHttpResponse.cs	Adds fake response helper for SSE tests.
tests/Exceptionless.Tests/Hubs/WebSocketCompatibilityTests.cs	Adds WebSocket compatibility and dual-fanout tests.
tests/Exceptionless.Tests/Hubs/WebSocketTests.cs	Removes old WebSocket broker tests.
tests/Exceptionless.Tests/Hubs/WebSocketConnectionManagerTests.cs	Removes old WebSocket manager tests.
tests/Exceptionless.Tests/Hubs/TestWebSocket.cs	Updates test WebSocket encoding and close count layout.
tests/Exceptionless.Tests/appsettings.yml	Enables push for test host configuration.
tests/http/push.http	Adds manual SSE smoke requests for localhost.
k8s/exceptionless/templates/api.yaml	Adds drain lifecycle settings and disables push pending ingress migration.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 3 comments.

[niemyjski]

Addressed the outstanding review feedback in the latest pushes: restored the Angular legacy client to the existing WebSocket path with only a deprecation note, tightened SSE/WebSocket lifecycle cleanup, fixed the SSE queue and keepalive backpressure edge case, and stopped the Svelte client from retrying when push is disabled.\n\nLocal validation is green (, , , , and Using launch settings from /Users/blake/Projects/Exceptionless/copilot-worktrees/Exceptionless/niemyjski-stunning-robot/tests/Exceptionless.Tests/Properties/launchSettings.json...
Running tests from /Users/blake/Projects/Exceptionless/copilot-worktrees/Exceptionless/niemyjski-stunning-robot/tests/Exceptionless.Tests/bin/Debug/net10.0/Exceptionless.Tests.dll (net10.0|arm64)
[+94/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(2s)

[+94/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(5s)

[+239/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(8s)

[+338/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(11s)

[+483/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(14s)

[+506/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(17s)

[+569/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(20s)

[+684/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(23s)

[+743/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(26s)

[+835/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(29s)

skipped Exceptionless.Tests.Pipeline.EventPipelineTests.GeneratePerformanceDataAsync (0ms)
Used to create performance data from the queue directory
from /Users/blake/Projects/Exceptionless/copilot-worktrees/Exceptionless/niemyjski-stunning-robot/tests/Exceptionless.Tests/bin/Debug/net10.0/Exceptionless.Tests.dll (net10.0|arm64)
[+900/x0/?1] Exceptionless.Tests.dll (net10.0|arm64)(32s)

[+1042/x0/?1] Exceptionless.Tests.dll (net10.0|arm64)(35s)

[+1151/x0/?1] Exceptionless.Tests.dll (net10.0|arm64)(38s)

skipped Exceptionless.Tests.Repositories.EventRepositoryTests.GetAsyncPerformanceAsync (0ms)
Performance Testing
from /Users/blake/Projects/Exceptionless/copilot-worktrees/Exceptionless/niemyjski-stunning-robot/tests/Exceptionless.Tests/bin/Debug/net10.0/Exceptionless.Tests.dll (net10.0|arm64)
skipped Exceptionless.Tests.Repositories.EventRepositoryTests.GetAsync (0ms)
elastic/elasticsearch-net#2463
from /Users/blake/Projects/Exceptionless/copilot-worktrees/Exceptionless/niemyjski-stunning-robot/tests/Exceptionless.Tests/bin/Debug/net10.0/Exceptionless.Tests.dll (net10.0|arm64)
[+1236/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(41s)

[+1277/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(44s)

[+1424/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(47s)

[+1567/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(50s)

[+1650/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(53s)

[+1792/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(56s)

[+1864/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(59s)

[+1873/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 02s)

[+1873/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 05s)

[+1874/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 08s)

[+1875/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 11s)

[+1876/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 14s)

/Users/blake/Projects/Exceptionless/copilot-worktrees/Exceptionless/niemyjski-stunning-robot/tests/Exceptionless.Tests/bin/Debug/net10.0/Exceptionless.Tests.dll (net10.0|arm64) passed (1m 14s 985ms)

Test run summary: Passed!
total: 1880
failed: 0
succeeded: 1877
skipped: 3
duration: 1m 15s 407ms), the PR checks are green, and the open inline threads have been replied to and resolved.

[ejsmith]

The steady-state SSE fanout design should not require sticky sessions: each long-lived stream remains local to the replica that accepted it, Redis pub/sub delivers notifications to every API replica, and only the replica owning a given local connection writes to it. Reconnects can land on any replica.

However, the shared connection metadata currently has two scale-out failure modes that should be addressed before enabling push in production:

1. Authentication-token revocation can be consumed destructively by a non-owning replica, leaving the owning replica's stream connected.
2. Redis connection reservations have no lease or expiry, so an ungraceful pod/node failure can leave permanent entries that eventually cause users with no live streams to receive 429 responses.

The current tests use a single host/manager and cannot expose these cases. Please add a two-replica test with shared connection state/message bus covering cross-replica delivery and token revocation, plus stale-reservation behavior after simulated replica loss.

[niemyjski]

Follow-up feedback audit complete at 5ee86353a9412d2d19394025fccee7b4dea25b01.

• Live inventory before this summary: 37 review threads, 70 inline comments, 42 submitted reviews, and 2 issue comments. All 37 threads were already resolved and none were open, so there was no open thread to reply to or resolve.
• Classification: 4 live-line findings are already fixed/current (EnablePush with legacy EnableWebSockets, SSE resource disposal, and fetch-based SSE auth); 31 old bot/Copilot findings are outdated or superseded by the lifecycle redesign and deleted code; 2 human scale-out findings are already fixed by process-local ownership and expiring Redis leases. Narrow exception-allowlist prescriptions were not safe as written, but their cleanup and backpressure concerns are covered by the current design and tests.
• The one still-actionable item was @ejsmith requested true two-replica shared-message-bus regression coverage. RCA: production fanout/revocation behavior existed, but the test called handlers directly and therefore did not prove bus subscription behavior across replicas. Commit 5ee86353a adds a shared-bus test proving fanout reaches both local replicas and token revocation closes only the owner replica connection while preserving the non-owner connection.
• Focused proof: SSE broker 10/10, manager 7/7, deduplication 9/9, HTTP integration 7/7, WebSocket compatibility 14/14, in-memory leases 2/2, real Redis lease expiry 1/1, and push option compatibility 3/3. dotnet build Exceptionless.slnx --no-restore passed with 0 warnings and 0 errors.
• Full thermo-nuclear branch-vs-main review found no remaining structural, concurrency, authorization, resource-cleanup, compatibility, duplication, atomicity, or file-size blocker; the largest changed file is 770 lines.
• CI on this exact head is terminal green: API coverage, E2E, client, Docker, website, version, and CLA all passed.

GitHub still reports the historical CHANGES_REQUESTED decision; I did not dismiss it. The requested production fixes and combined replica coverage are now present for reviewer re-approval.

[github-actions]

Package	Line Rate	Branch Rate	Complexity	Health
Exceptionless.Insulation	40%	36%	290	❌
Exceptionless.Web	86%	71%	8313	✔
Exceptionless.Core	77%	68%	10564	✔
Exceptionless.AppHost	38%	41%	147	❌
Summary	79% (27040 / 34013)	69% (12528 / 18265)	19314	✔