Advanced

This page covers production-level topics: retry policies, scheduling, observability, multi-tenancy, database schema, migrations, and testing strategies.

Retry Policies

GrydNotifications uses Polly-based retry policies for transient failures in email delivery, push notifications, and webhook calls.

Configuration

builder.Services.AddGrydNotifications(options =>
{
    options.Retry.MaxRetries = 3;
    options.Retry.BackoffType = BackoffType.Exponential;
    options.Retry.BaseDelay = TimeSpan.FromSeconds(5);
    options.Retry.MaxDelay = TimeSpan.FromMinutes(5);
    options.Retry.UseJitter = true;
    options.Retry.NonRetryableReasons = new HashSet<DeliveryFailureReason>
    {
        DeliveryFailureReason.InvalidRecipient,
        DeliveryFailureReason.InvalidTemplate,
        DeliveryFailureReason.AuthenticationFailed
    };
});

Backoff Types

Type	Formula	Example (BaseDelay = 5s)
Fixed	BaseDelay	5s, 5s, 5s
Linear	BaseDelay × attempt	5s, 10s, 15s
Exponential	BaseDelay × 2^(attempt-1)	5s, 10s, 20s

When UseJitter = true, a random variation (±25%) is added to prevent thundering herd when multiple notifications retry simultaneously.

Retry Flow

Send Attempt 1 ──► Failure (transient)
       │
       ▼
  Wait BaseDelay (5s + jitter)
       │
       ▼
Send Attempt 2 ──► Failure (transient)
       │
       ▼
  Wait BaseDelay × 2 (10s + jitter)
       │
       ▼
Send Attempt 3 ──► Failure (transient)
       │
       ▼
  MaxRetries exceeded → Mark as Failed
  Reason stored in notification record

Non-Retryable Reasons

If a failure reason matches any value in NonRetryableReasons, the notification is immediately marked as Failed without retrying — saving resources on permanent errors.

Dead Letters

Notifications that exhaust all retry attempts are moved to a dead-letter state:

// Query failed notifications for investigation
var deadLetters = await _mediator.Send(new GetNotificationsQuery
{
    Status = NotificationStatus.Failed,
    Page = 1,
    PageSize = 50
}, ct);

Queue Processing

The background queue processes notifications asynchronously when EnableQueue = true:

options.Queue.BatchSize = 10;          // Notifications per batch
options.Queue.PollingInterval = 
    TimeSpan.FromSeconds(5);           // Poll frequency
options.Queue.MaxConcurrency = 4;      // Parallel senders
options.Queue.MessageExpiration = null; // No expiration by default

Queue Architecture

Producer (API/Command)
       │
       ▼
┌─────────────────┐
│  Queue Storage   │  (in-memory Channel<T> — default)
│  (pending msgs)  │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Queue Processor │  (BackgroundService)
│  4 workers       │────► Email Sender
│  10 per batch    │────► Push Sender
│  5s polling      │────► InApp Sender
└─────────────────┘

Scheduling Notifications

Send Later

Schedule a notification for future delivery:

await _mediator.Send(new ScheduleNotificationCommand
{
    Channel = NotificationChannel.Email,
    Recipients = new[] { "user@example.com" },
    Subject = "Your trial expires tomorrow",
    TemplateId = trialExpiryTemplateId,
    Variables = new Dictionary<string, object>
    {
        ["user_name"] = "John",
        ["expiry_date"] = DateTime.UtcNow.AddDays(1)
    },
    ScheduledAt = DateTime.UtcNow.AddDays(6) // Send 1 day before trial ends
}, ct);

Cancel a Scheduled Notification

await _mediator.Send(new CancelScheduledNotificationCommand
{
    NotificationId = notificationId
}, ct);

Integration with GrydJobs

For complex scheduling (recurring reminders, cron expressions), integrate with the GrydJobs module:

// Register via GrydNotifications.Scheduling package
builder.Services.AddGrydNotifications(options => { /* ... */ })
    .AddSchedulingIntegration();

// Recurring weekly digest
await _mediator.Send(new CreateRecurringNotificationJobCommand
{
    JobName = "weekly-digest",
    CronExpression = "0 9 * * MON", // Every Monday at 9 AM
    Channel = NotificationChannel.Email,
    TemplateId = weeklyDigestTemplateId,
    RecipientQuery = "active-subscribers" // Named query
}, ct);

Observability

OpenTelemetry Integration

GrydNotifications emits traces and metrics via Gryd.Observability.OpenTelemetry:

builder.Services.AddGrydObservability(options =>
{
    options.ServiceName = "my-api";
    options.EnableTracing = true;
    options.EnableMetrics = true;
});

Metrics

Metric	Type	Description
gryd.notifications.sent	Counter	Total notifications sent
gryd.notifications.failed	Counter	Total notifications failed
gryd.notifications.retried	Counter	Total retry attempts
gryd.notifications.queue.depth	Gauge	Current queue depth
gryd.notifications.queue.processing_time	Histogram	Queue processing latency
gryd.notifications.delivery.duration	Histogram	End-to-end delivery time

All metrics include dimensions: channel, status, tenant_id.

Traces

Each notification generates a trace span:

notification.send
  ├── template.render (if template used)
  ├── channel.email.send (MailKit/SendGrid)
  ├── channel.push.send (FCM)
  └── channel.inapp.persist (PostgreSQL + SignalR)

Health Checks

builder.Services.AddHealthChecks()
    .AddGrydNotificationsHealthCheck();

The health check verifies:

• Database connectivity — Can reach PostgreSQL notifications schema
• Queue processor — BackgroundService is running (if enabled)
• SignalR hub — Hub is accepting connections (if enabled)
• SMTP — Can connect to SMTP server (if email enabled)

GET /health

{
  "status": "Healthy",
  "checks": {
    "gryd-notifications": {
      "status": "Healthy",
      "data": {
        "database": "OK",
        "queue": "OK",
        "signalr": "OK",
        "smtp": "OK"
      }
    }
  }
}

Multi-Tenancy

GrydNotifications is multi-tenancy aware through ITenantOptionalAware:

Tenant Isolation

All commands and queries accept an optional TenantId:

await _mediator.Send(new SendNotificationCommand
{
    TenantId = currentTenant.Id, // Nullable<Guid>
    Channel = NotificationChannel.Email,
    Recipients = new[] { "user@example.com" },
    Subject = "Welcome!"
}, ct);

When TenantId is set:

• Notifications are scoped to the tenant
• Templates fall back: tenant-specific → global
• SignalR groups include tenant: tenant_{tenantId}
• Queries only return tenant-scoped data

Template Tenant Fallback

Lookup: (key="welcome", locale="pt-BR", tenantId="abc")
  1. key="welcome", locale="pt-BR", tenantId="abc"    ← Tenant + locale
  2. key="welcome", locale="pt",    tenantId="abc"    ← Tenant + language
  3. key="welcome", locale="pt-BR", tenantId=null     ← Global + locale
  4. key="welcome", locale="pt",    tenantId=null     ← Global + language
  5. key="welcome", locale=null,    tenantId=null     ← Default fallback

Per-Tenant Provider Configuration

Override email/push settings per tenant:

// In a custom ITenantNotificationConfigProvider
public class TenantNotificationConfig : ITenantNotificationConfigProvider
{
    public Task<EmailOptions?> GetEmailOptionsAsync(Guid tenantId, CancellationToken ct)
    {
        // Load from database, cache, etc.
        return Task.FromResult(new EmailOptions
        {
            DefaultFrom = "noreply@tenant-domain.com",
            SmtpHost = "smtp.tenant-provider.com"
        });
    }
}

Database Schema

All tables live in the notifications schema:

-- Core tables
notifications.notifications          -- Main notification records
notifications.notification_recipients -- Per-recipient tracking
notifications.notification_templates  -- Scriban templates
notifications.notification_queue      -- Async queue (if enabled)

-- In-App tables
notifications.user_notifications     -- Per-user in-app notifications
notifications.user_devices           -- Push notification device tokens

Key Indexes

Table	Index	Purpose
notifications	ix_notifications_status	Queue processing
notifications	ix_notifications_tenant_created	Tenant + date queries
user_notifications	ix_user_notifications_user_unread	Unread count badge
user_notifications	ix_user_notifications_collapse_key	Collapse key lookup
notification_templates	ix_templates_key_locale_tenant	Template resolution

Migrations

Apply on Startup

var app = builder.Build();
await app.ApplyGrydNotificationsMigrationsAsync();

Generate SQL Script

dotnet ef migrations script \
  --project src/Modules/Notifications/GrydNotifications.Infrastructure \
  --context NotificationsDbContext \
  --output migrations.sql

Add a New Migration

dotnet ef migrations add AddCustomColumn \
  --project src/Modules/Notifications/GrydNotifications.Infrastructure \
  --startup-project src/Core/Gryd.API \
  --context NotificationsDbContext

Testing

Unit Testing Handlers

using NSubstitute;
using Xunit;

public class SendEmailNotificationHandlerTests
{
    private readonly IEmailSender _emailSender = Substitute.For<IEmailSender>();
    private readonly INotificationRepository _repo = Substitute.For<INotificationRepository>();
    private readonly SendEmailNotificationHandler _sut;

    public SendEmailNotificationHandlerTests()
    {
        _sut = new SendEmailNotificationHandler(_emailSender, _repo);
    }

    [Fact]
    public async Task Handle_ValidEmail_SendsAndPersists()
    {
        // Arrange
        var command = new SendEmailNotificationCommand
        {
            Recipients = new[] { "test@example.com" },
            Subject = "Test",
            Body = "<p>Hello</p>"
        };

        _emailSender.SendAsync(Arg.Any<EmailMessage>(), Arg.Any<CancellationToken>())
            .Returns(Result.Success());

        // Act
        var result = await _sut.Handle(command, CancellationToken.None);

        // Assert
        Assert.True(result.IsSuccess);
        await _emailSender.Received(1).SendAsync(
            Arg.Is<EmailMessage>(m => m.To.Contains("test@example.com")),
            Arg.Any<CancellationToken>());
        await _repo.Received(1).AddAsync(
            Arg.Any<Notification>(),
            Arg.Any<CancellationToken>());
    }
}

Testing Controllers

using MediatR;
using NSubstitute;
using Microsoft.AspNetCore.Mvc;

public class NotificationsControllerTests
{
    private readonly IMediator _mediator = Substitute.For<IMediator>();
    private readonly ILogger<NotificationsController> _logger = Substitute.For<ILogger<NotificationsController>>();
    private readonly NotificationsController _sut;

    public NotificationsControllerTests()
    {
        _sut = new NotificationsController(_mediator, _logger);
    }

    [Fact]
    public async Task Send_ValidRequest_ReturnsOk()
    {
        // Arrange
        var request = new SendNotificationRequest { /* ... */ };
        _mediator.Send(Arg.Any<SendNotificationCommand>(), Arg.Any<CancellationToken>())
            .Returns(Result<Guid>.Success(Guid.NewGuid()));

        // Act
        var result = await _sut.Send(request, CancellationToken.None);

        // Assert
        Assert.IsType<OkObjectResult>(result);
    }
}

Integration Tests with TestContainers

using Testcontainers.PostgreSql;

public class NotificationsIntegrationTests : IAsyncLifetime
{
    private readonly PostgreSqlContainer _postgres = new PostgreSqlBuilder()
        .WithImage("postgres:16-alpine")
        .Build();

    public async Task InitializeAsync()
    {
        await _postgres.StartAsync();
        // Apply migrations
    }

    public async Task DisposeAsync() => await _postgres.DisposeAsync();

    [Fact]
    public async Task FullNotificationLifecycle_SendAndRead()
    {
        // Create factory with real database
        var factory = new WebApplicationFactory<Program>()
            .WithWebHostBuilder(builder =>
            {
                builder.ConfigureServices(services =>
                {
                    services.AddGrydNotifications(options =>
                    {
                        options.ConnectionString = _postgres.GetConnectionString();
                        options.InApp.Enabled = true;
                    });
                });
            });

        var client = factory.CreateClient();

        // Send notification
        var response = await client.PostAsJsonAsync("/api/v1/notifications/send", new
        {
            channel = "InApp",
            userId = Guid.NewGuid(),
            title = "Test",
            body = "Integration test"
        });

        Assert.Equal(HttpStatusCode.Created, response.StatusCode);
    }
}

Testing SignalR Hub

[Fact]
public async Task NotificationHub_ReceivesRealTimeNotification()
{
    var factory = new WebApplicationFactory<Program>();
    var server = factory.Server;

    var connection = new HubConnectionBuilder()
        .WithUrl(
            $"{server.BaseAddress}hubs/notifications",
            options => options.HttpMessageHandlerFactory = _ => server.CreateHandler())
        .Build();

    var received = new TaskCompletionSource<UserNotificationDto>();
    connection.On<UserNotificationDto>("ReceiveNotification", n =>
    {
        received.SetResult(n);
    });

    await connection.StartAsync();

    // Trigger notification via API
    var client = factory.CreateClient();
    await client.PostAsJsonAsync("/api/v1/notifications/send", new { /* ... */ });

    // Wait for real-time delivery
    var notification = await received.Task.WaitAsync(TimeSpan.FromSeconds(5));
    Assert.Equal("Test", notification.Title);
}

Performance Tips

Tip	Description
Batch emails	Use BulkBatchSize and BulkDelayBetweenBatches for mass emails
Enable queue	Process notifications asynchronously to avoid blocking API requests
Use collapse keys	Prevent notification spam for repetitive events
Set expiration	DefaultTimeToLive avoids stale in-app notifications
Redis backplane	Required for SignalR with multiple server instances
Template caching	Templates are cached by default — avoid disabling in production
Tune concurrency	Adjust Queue.MaxConcurrency based on provider rate limits
Monitor queue depth	Alert on gryd.notifications.queue.depth growing continuously