Monitoring and Maintaining AlertD

This page describes how to monitor a running AlertD deployment, diagnose fault conditions, and perform routine and emergency maintenance: watching application logs, rotating secrets, applying software updates, managing AWS service quotas, backing up data, and recovering from failures.

Monitoring Application Health

The most direct signal of AlertD’s health is the application container’s log stream in CloudWatch Logs. The container writes structured log lines on startup, on every incoming request, and whenever the request pipeline raises an exception. Watching this stream for errors is how you detect problems before users do.

Locating the Application Log Group

CloudFormation creates a CloudWatch Logs log group for the AlertD application task. Its name is derived from your stack name and follows the pattern /ecs/<stack-name>/app.

1. In the AWS console, open CloudWatch → Log groups.
2. Filter by your CloudFormation stack name; the application log group is the one ending in /app.
3. Click the log group to see the per-task log streams. Each restart or rolling deployment creates a new stream.

Watching the Live Stream

To tail the application logs in real time from your terminal:

aws logs tail /ecs/<stack-name>/app --follow

Add --since 1h to backfill the last hour, or --filter-pattern ERROR to surface only error-level lines:

aws logs tail /ecs/<stack-name>/app --follow --filter-pattern ERROR

What to Look For

A healthy AlertD task logs steady request lines and periodic background-job lines. Treat these as fault signals:

ALB Target Health

The ALB performs an HTTP health check against the AlertD task. In EC2 → Load Balancers → <AlertD ALB> → Target groups, the application target group should show all targets as healthy. An unhealthy target usually points back to a problem visible in the application log stream (failed startup, dependency unreachable, etc.).

Handling Fault Conditions

This section maps user-visible symptoms to the diagnostic steps and the section that contains the fix. Start by tailing the application log stream (Monitoring Application Health) — it is the fastest way to localize most issues.

The AlertD URL Returns 502 or 503

The ALB cannot reach a healthy application target.

1. In EC2 → Load Balancers → Target groups, confirm whether the application target is unhealthy.
2. In ECS → Clusters → <AlertD cluster> → Services → <app service>, check whether desired/running counts match.
3. Open the application log group and read the latest log stream from the top — startup failures (bad LLM key, unreachable Aurora) appear in the first few lines.
4. Apply the matching fix below, then force a new deployment (see Recovering the Software → ECS Task Failure).

Repeated 401/403 from the LLM Provider

The OpenAI or Anthropic API is rejecting AlertD’s requests.

1. Confirm the symptom in the application log group (api.openai.com or api.anthropic.com 4xx lines).
2. Rotate the LLM API key per Rotating the LLM API Key.

AccessDenied When Querying AWS

AlertD cannot assume the customer’s monitoring role.

1. In the application log group, look for STS AssumeRole errors with the target role ARN.
2. In the target AWS account, open IAM → Roles → AlertD-Role → Trust relationships and confirm the Principal matches the AlertD ECS task role ARN.
3. Confirm the ReadOnlyAccess managed policy is still attached.
4. See Advanced Setup → Troubleshooting for the full diagnostic checklist.

Database Connection Errors

The application cannot reach Aurora.

1. In the application log group, look for connection refused, connection reset, or PostgreSQL driver errors.
2. In RDS → Databases → <AlertD cluster>, confirm the cluster status is Available and the writer endpoint resolves.
3. Confirm the application security group has egress to the Aurora security group on port 5432.
4. If the cluster is in Backing-up or Modifying state, wait — Aurora Serverless v2 fails over automatically; the application reconnects on its own.

Stuck or Slow Queries

A user-issued query never completes.

1. The application enforces a 2-minute timeout on queries (see FAQ → What’s the query timeout?). A query that runs longer than that will be cancelled and surfaced as an error.
2. If many queries time out, check Aurora capacity (RDS → <AlertD cluster> → Monitoring → ACU utilization). If ACUs are saturated, raise the maximum capacity on the cluster.
3. Suggest the user narrow scope (region filter, more specific resource type) per the FAQ performance tips.

Authentication Failures at the AlertD Login Screen

Users cannot complete Google or GitHub login.

1. Confirm outbound TCP 443 is open from end-user browsers to auth.demo.alertd.ai (see Prerequisites → Network Access).
2. See Simple Setup → Troubleshooting → Authentication Issues for known browser/popup workarounds.

If none of the above match, capture the latest application log lines and contact support.

Secrets Management

AlertD stores two categories of secrets in AWS Secrets Manager:

1. Aurora database credentials — created automatically by CloudFormation when the stack is deployed.
2. LLM API key — created only if you selected openai or anthropic as the model profile (see Deployment Step 5).

Rotating the LLM API Key

If your OpenAI or Anthropic API key is leaked, deactivated, or simply being rotated on a schedule:

1. Generate a new key in the OpenAI or Anthropic console.
2. In the AWS console, open Secrets Manager and locate the AlertD LLM key secret (its name is prefixed with your CloudFormation stack name).
3. Choose Retrieve secret value → Edit, paste the new key, and save.
4. Force a new ECS deployment so the AlertD task picks up the new value:

   aws ecs update-service \
     --cluster <AlertD-cluster-name> \
     --service <AlertD-app-service-name> \
     --force-new-deployment

5. The AlertD task restarts with the new key. End users experience no downtime because ECS performs a rolling deployment.

Rotating the Aurora Database Credentials

Aurora Serverless v2 credentials live in Secrets Manager and can be rotated using AWS’s built-in rotation:

1. Open Secrets Manager and select the AlertD database secret.
2. Choose Edit rotation and enable Automatic rotation with the AWS-managed Lambda rotation function for RDS.
3. Set the rotation schedule (e.g., every 30 days).
4. Trigger an immediate rotation with Rotate secret immediately to verify the workflow.
5. After rotation, force a new ECS deployment (same command as above) so the application reconnects with the new credentials.

For a manual one-time rotation, follow the same procedure with Rotate secret immediately.

Cryptographic Keys

AlertD does not require customers to create or manage any KMS keys. Encryption at rest uses the AWS-managed default keys for Aurora, EBS, Secrets Manager, and CloudWatch Logs; rotation of those keys is handled automatically by AWS. See the Security Model for details.

Software Patches and Upgrades

AlertD is delivered as a CloudFormation template that references pinned container images for the application and Pulsar tasks. Aurora is patched by AWS on its standard maintenance schedule.

Upgrading AlertD

1. Open the CloudFormation console and select your AlertD stack.
2. Choose Update → Replace current template.
3. Provide the latest AlertD template URL (the canonical URL stays the same; new versions are published in place):

   https://alertd-publicassets.s3.us-west-1.amazonaws.com/cloudformation/AlertD_RDS.yaml

4. Step through the wizard; leave parameter values unchanged unless release notes indicate otherwise.
5. Acknowledge the IAM capabilities checkbox and submit.
6. CloudFormation performs a rolling update of the ECS services. The ALB drains old tasks and shifts traffic to new ones, so there is no downtime.

Aurora Patching

Aurora Serverless v2 receives minor-version patches automatically during the cluster’s maintenance window. Major-version upgrades are opt-in and can be performed from the RDS → Modify workflow on the AlertD database cluster. We recommend taking a manual snapshot (see Backup and Recovery below) before any major-version change.

Container Image Patching

The AlertD application and Pulsar container images are published with security patches applied. Picking up the latest images is done by running the stack update above; there is no separate image-pull step.

Managing AWS Service Limits

AlertD is constrained by the standard AWS account quotas listed below. Most accounts are well within the default limits, but consolidated or large-scale environments may need to request increases through the Service Quotas console .

To request an increase: open Service Quotas in the AWS console, find the quota, and choose Request quota increase. Increases are usually approved within a few hours for common services.

Backup and Recovery

Data Stores in Scope

AlertD stores all customer state in Aurora Serverless v2 (queries, session history, query results, execution plans, workspace metadata). The application and Pulsar tasks are stateless — their disks are ephemeral and require no backup.

Aurora Automated Backups

Aurora Serverless v2 takes continuous backups and retains them for 7 days by default (configurable up to 35 days), with point-in-time recovery (PITR) to any second within the retention window.

To verify or change backup settings:

1. Open RDS → Databases → <AlertD cluster>.
2. Under Maintenance & backups, confirm the backup retention period and the preferred backup window.
3. To extend retention, choose Modify and set a longer retention period.

Taking a Manual Snapshot

Manual snapshots are retained until you delete them and are not subject to the retention window.

1. Open RDS → Databases → <AlertD cluster>.
2. Choose Actions → Take snapshot, give it a descriptive name, and submit.

We recommend taking a manual snapshot before any AlertD stack upgrade or Aurora major-version change.

Restoring from a Backup

The AlertD stack owns the Aurora cluster, so restoring from a backup requires rebuilding the stack around the restored cluster rather than swapping the endpoint inside the running stack:

1. Open RDS → Databases → <AlertD cluster> → Actions → Restore to point in time (or Restore snapshot) and let RDS provision a new cluster from the backup.
2. Delete the existing AlertD CloudFormation stack (keep the restored Aurora cluster — do not delete it).
3. Redeploy the AlertD CloudFormation stack and, during stack creation, import the restored Aurora cluster and its Secrets Manager secret as existing resources rather than letting CloudFormation create new ones. See AWS — Bringing existing resources into CloudFormation management .
4. Once the stack reaches CREATE_COMPLETE, the AlertD application reconnects to the restored data.

Configuration

The entire deployment configuration is captured in the CloudFormation template. To restore configuration from scratch, redeploy the stack with the same parameters and import the restored Aurora cluster as described above.

Recovering the Software

ECS Task Failure

ECS automatically detects task failures and replaces them. If a task is crash-looping:

1. Open ECS → Clusters → <AlertD cluster> → Tasks to see the stopped task and its exit reason.
2. Open CloudWatch Logs for the corresponding log group to inspect the failure.
3. Common causes: invalid LLM API key, unreachable Aurora endpoint, or Secrets Manager permission issues. Fix the underlying configuration and force a new deployment:

   aws ecs update-service \
     --cluster <AlertD-cluster-name> \
     --service <AlertD-app-service-name> \
     --force-new-deployment

Aurora Failure

For transient Aurora issues, Aurora Serverless v2 automatically fails over. For persistent corruption or accidental data loss, restore from a backup using the procedure in Backup and Recovery.

Failed Stack Update

If a CloudFormation update fails:

1. CloudFormation will attempt an automatic rollback to the previous working state.
2. If the rollback succeeds, investigate the cause (Events tab) before retrying.
3. If the rollback also fails, choose Continue rollback and skip any non-recoverable resources, or open a support ticket.

Complete Rebuild

If the stack cannot be recovered (e.g., accidental deletion of critical resources):

1. Take a manual snapshot of the Aurora cluster (if it is still present).
2. Delete the AlertD CloudFormation stack.
3. Redeploy the stack following the Deployment Guide and import the restored Aurora cluster during stack creation (see Restoring from a Backup).

Support

If you need help with any of the procedures above, contact AlertD support at support@alertd.ai. See the FAQ → Getting Help section for response-time expectations.