Durable delivery

Durable notification delivery is opt-in through DurableDelivery=true, together with PersistNotifications=true and EnableQueue=true. It requires PostgreSQL and the migration DurableNotificationDelivery from the release containing this change. Historical 5.0.2 does not provide this contract. Existing in-memory mode remains available, without a crash-recovery guarantee.

Set RunBackgroundServices=false in API hosts and true only in Worker hosts. This flag covers queue processing and periodic cleanup; it does not remove persistence or producer registration. Use the actual host service collection to verify the result, including registrations from packages.

Producer and consumer contract

Send, direct email/push and schedule commands require a nonempty OperationId in durable mode. The same tenant, operation and content return the original notification; changed content produces a conflict. The notification and its outbox intent commit together, joining a caller transaction when one exists. Direct email/push delivery payloads are snapshotted, including recipient and channel data.

Gryd.Infrastructure.Messaging.PostgresOutbox owns the generic mechanism. The producer maps and migrates its own table; Notifications maps it in the notifications schema. Business meaning belongs to the producer. Each claim selects the first unfinished message of an aggregate, even when an earlier message is not yet due, under a short transaction with FOR UPDATE SKIP LOCKED. Eligible aggregates are ordered by priority (critical first), then availability time. Future schedules remain ineligible regardless of priority.

A lease fences workers before the effect. Expired claims can be recovered; expired external Sending claims become ReconciliationRequired. Local inbox writes, attempt/result and a unique delivery receipt commit with completion. Failure rolls local effects back and records a bounded retry. Acknowledgement lost after an external send can never establish “exactly once” delivery: it requires reconciliation. An external error potentially representing partial delivery is not automatically retried.

In-memory domain events are not the durability mechanism. Required local effects must be explicit calls in the shared transaction; volatile observers run after producer commit. A reaction that cannot be lost must have a durable producer intent and consumer deduplication with its local effect.

Access and operational state

Management controllers require GrydNotifications.Manage, satisfied by authenticated permission=manage:notifications or permission=admin:system. Inbox queries are bound to the principal's user ID. Host tenant resolution must derive from a validated identity; strict write protection requires RequireTenantMatchOnWrite=true, TenantSaveChangesInterceptor and ProtectTenantWrites() in the mapped model. Bulk SQL requires explicit scope because it bypasses SaveChanges interceptors. Operational outbox discovery is privileged; business effects run in the scope recorded by the trusted producer, cleared between operations.

GET /api/v1/notifications/{id}/delivery reports Pending, Claimed, Sending, Completed, DeadLettered or ReconciliationRequired, attempt budget and lease. Completed can also represent an effective pre-send cancellation; inspect the notification status.

POST /api/v1/notifications/{id}/reconcile accepts operationId, delivered and evidenceReference (maximum 512 characters). The authenticated actor is recorded with the decision in delivery_resolutions. Confirmed delivery completes the intent. Confirmed non-delivery authorizes one new attempt, preserving any future scheduled date. Repeating the same decision does not increase the allowance. Partial delivery must not be classified as non-delivery of the whole notification.

At the first durable migration, unresolved legacy notifications are imported with Kind=notification.legacy.v1 and ReconciliationRequired. Their operation ID is derived from the notification ID; legacy-unverified marks that a complete original request hash/snapshot is unavailable. RequiresNewOperation=true exposes this limitation. Confirming delivery closes the occurrence; confirming non-delivery cancels it without replay. Sending again requires a new, complete request. Stop legacy producers and workers before applying the migration and switching to durable mode. Do not mix legacy producers with durable consumers during the cutover.

Before restarting workers after a database restore, call PostgresOutbox.QuarantineExternalAfterRestoreAsync explicitly under migration/recovery exclusion. Even an external Pending row in the backup may have been sent after the snapshot. This operation quarantines external Pending/Claimed/Sending rows for evidence-based reconciliation. It must not run on ordinary application startup. Keep producers and workers stopped during recovery.

Cleanup discovers bounded batches under an exclusive maintenance lock and applies effects within each tenant's scope. Query pages are limited to 200 records; detail collections use split queries to avoid cartesian multiplication. Template cache identity includes template ID, content part and version.

The SignalR hub uses tenant+user groups and closes on authentication expiry. The package still lacks an end-to-end distributed publisher and immediate revocation of already open sockets. Persistent inbox delivery is independent of that future transport; do not claim live multi-replica delivery from registration alone.

PostgreSQL regressions cover transaction rollback, repeated operations, worker concurrency, expired leases, local deduplication, external lost confirmation, tenant isolation, reconciliation and migrations. Consumer applications must additionally test their own restore procedure and host composition.