Skip to Content
OperationsMonitoring

Monitoring AlertD

AlertD runs entirely inside your AWS account, on services you already know how to watch. Every component publishes metrics to Amazon CloudWatch, and the application writes its logs to CloudWatch Logs. This page explains what the deployment is made of, which metrics tell you it is healthy, and which alarms we recommend.

The AlertD stack does not create CloudWatch alarms for itself. Everything below is a recommendation you set up once, in the CloudWatch console, after deployment.


What AlertD deploys

Throughout this page, <stack-name> is the name you gave the CloudFormation stack when you deployed AlertD (for example alertd-prod).

ComponentAWS serviceWhere to find it
Application containerAmazon ECS on AWS FargateECS cluster <stack-name>-cluster, service whose name starts with <stack-name>-SvcApp
Messaging container (Apache Pulsar)Amazon ECS on AWS FargateSame cluster, service whose name starts with <stack-name>-SvcPulsar
DatabaseAmazon Aurora PostgreSQL (Serverless v2)RDS cluster <stack-name>-aurora-cluster, instance <stack-name>-aurora-instance
Application dataAmazon EFSFile system named <stack-name>-app-data
Reports and storageAmazon S3Bucket name is in the stack output LoggingBucketName
Internal load balancerApplication Load BalancerEC2 console → Load Balancers, tagged with your stack
Public entry pointAmazon CloudFrontDistribution with the comment <stack-name> AlertD distribution
Auto-update and self-healingAWS Lambda, Amazon EventBridge, Amazon SNSLambda function and SNS topic both named <stack-name>-auto-update
Application logsCloudWatch LogsLog group /ecs/<stack-name>

ECS service names end in a random suffix that CloudFormation generates. To list them, open ECS → Clusters → <stack-name>-cluster → Services, or run:

aws ecs list-services --cluster <stack-name>-cluster

Metrics to watch

Open CloudWatch → Metrics → All metrics and pick the namespace from the table. The columns show the metric name, what a bad value means for AlertD, and a sensible alarm threshold to start with.

Application and messaging containers

The ECS cluster has Container Insights enabled, so you get per-service CPU, memory and task-count metrics under the ECS/ContainerInsights namespace in addition to the standard AWS/ECS metrics.

Namespace / metricWhat it tells youSuggested alarm
ECS/ContainerInsights → RunningTaskCountNumber of application (or Pulsar) tasks actually running. Should be 1 for each service.Below 1 for 5 minutes
AWS/ECS → CPUUtilizationCPU used by the service as a percentage of what it reserved. Sustained high values slow down analyses.Above 85% for 15 minutes
AWS/ECS → MemoryUtilizationMemory used as a percentage of what the service reserved. Near 100% means tasks are at risk of being restarted.Above 85% for 15 minutes

Filter the metrics by ClusterName = <stack-name>-cluster and the ServiceName you found above.

Load balancer

The internal Application Load Balancer sits between CloudFront and the application. Its metrics are the clearest signal that the web interface is up. Use the AWS/ApplicationELB namespace and filter by the load balancer and target group of your stack.

MetricWhat it tells youSuggested alarm
HealthyHostCountApplication tasks passing the load balancer health check. Should be 1 per target group (the High Availability configuration has two target groups, one per application service).Below 1 for 5 minutes
UnHealthyHostCountTasks failing the health check.Above 0 for 10 minutes
HTTPCode_Target_5XX_CountServer errors returned by the application.Above 10 in 5 minutes
TargetResponseTimeHow long the application takes to answer.p95 above 5 seconds for 15 minutes

Database

The Aurora Serverless v2 cluster scales its capacity automatically between the minimum and maximum you set at deployment (the AuroraMinCapacity and AuroraMaxCapacity parameters, measured in Aurora Capacity Units, or ACUs). Use the AWS/RDS namespace and filter by DBClusterIdentifier = <stack-name>-aurora-cluster.

MetricWhat it tells youSuggested alarm
ServerlessDatabaseCapacityCurrent number of ACUs in use. If it sits at your maximum for long periods, raise AuroraMaxCapacity with a stack update.At or near the maximum for 30 minutes
ACUUtilizationCapacity in use as a percentage of the maximum.Above 90% for 30 minutes
CPUUtilizationDatabase CPU.Above 90% for 15 minutes
DatabaseConnectionsOpen connections from the application. A sudden drop to zero usually means the application is down.Equal to 0 for 10 minutes
FreeableMemoryMemory available to the database.Below 10% of the current capacity

Application data (EFS)

The file system uses Elastic throughput, so there are no burst credits to run out of. Use the AWS/EFS namespace and filter by FileSystemId (shown in the stack output AppDataFileSystemId).

MetricWhat it tells youSuggested alarm
PercentIOLimitHow close the file system is to its I/O limit.Above 90% for 15 minutes
ClientConnectionsApplication tasks connected to the file system. Should match the number of running application tasks.Equal to 0 for 10 minutes
StorageBytesTotal data stored. Useful for tracking growth and cost.Informational

Public entry point (CloudFront)

CloudFront metrics are always published in the US East (N. Virginia) Region, whichever Region you deployed AlertD to. Switch to us-east-1 in the CloudWatch console, choose the AWS/CloudFront namespace and filter by your distribution.

MetricWhat it tells youSuggested alarm
5xxErrorRatePercentage of requests that failed at the application or load balancer.Above 5% for 5 minutes
4xxErrorRateClient errors, mostly expired sessions and bad URLs. A sharp rise after a change is worth a look.Informational
RequestsTraffic volume.Informational

Auto-update and self-healing

A scheduled Lambda function checks that the application service is running and, when it is healthy, looks for signed AlertD releases to apply. If the application has been down for more than two hours, it rolls the service back to the last known-good version. Use the AWS/Lambda namespace and filter by FunctionName = <stack-name>-auto-update.

MetricWhat it tells youSuggested alarm
ErrorsThe check itself failed to run.Above 0 in 1 hour
InvocationsThe check runs several times an hour.Below 1 in 1 hour

This component exists only when the stack was deployed with EnableAutoUpdate set to true (the default).


Notifications from the auto-update check

The auto-update function publishes its decisions (for example a rollback to the last known-good version) to the SNS topic <stack-name>-auto-update. To be told when it acts, open SNS → Topics → <stack-name>-auto-update → Create subscription and subscribe an email address, a chat integration or the alerting pipeline you already use.


Application logs

All container output goes to the CloudWatch Logs group /ecs/<stack-name>. Each task writes to its own log stream; the stream name starts with ecs/app/ for the application and ecs/pulsar/ for the messaging container.

  • Open CloudWatch → Log groups → /ecs/<stack-name> and use Logs Insights to search across streams. A good first query during an incident:

    fields @timestamp, @message | filter @message like /ERROR|FATAL/ | sort @timestamp desc | limit 100
  • Log retention is 7 days and is managed by the stack. If you need to keep logs longer, add a subscription filter to forward them to Amazon S3, Amazon OpenSearch Service or your log platform. Changing the retention directly on the log group is possible, but the next stack update sets it back to 7 days.

Logs are also searchable from inside AlertD, in the diagnostics views, without leaving the application. CloudWatch Logs is the place to look when the application itself is not reachable.


How AlertD checks its own health

Two health checks run continuously, and both target the application’s web port:

  • Container health check — ECS requests the application’s home page every 60 seconds. After 5 consecutive failures the task is marked unhealthy and ECS replaces it. New tasks get a 5-minute grace period to start.
  • Load balancer health check — The load balancer makes the same request every 60 seconds. A task needs 2 consecutive successes to receive traffic and is taken out of rotation after 5 consecutive failures.

Both ECS services have the deployment circuit breaker enabled. If a new version fails to start during an update, ECS rolls the service back to the previous version automatically.


Creating an alarm

All of the alarms above are created the same way. As an example, here is the most important one: the application has no healthy task behind the load balancer.

  1. Open CloudWatch → Alarms → All alarms → Create alarm
  2. Click Select metric, choose ApplicationELB → Per AppELB, per TG Metrics, and tick HealthyHostCount for your stack’s load balancer and target group
  3. Set Statistic to Minimum and Period to 1 minute
  4. Under Conditions, choose Lower than 1, and set Datapoints to alarm to 5 out of 5
  5. Under Notification, pick an SNS topic that reaches your on-call rotation (create one here if you do not have one)
  6. Name the alarm, for example alertd-no-healthy-tasks, and click Create alarm

Repeat for the database and container alarms in the tables above. If you prefer the command line, the same alarm is:

aws cloudwatch put-metric-alarm \ --alarm-name alertd-no-healthy-tasks \ --namespace AWS/ApplicationELB \ --metric-name HealthyHostCount \ --dimensions Name=LoadBalancer,Value=<app/.../...> Name=TargetGroup,Value=<targetgroup/.../...> \ --statistic Minimum --period 60 \ --evaluation-periods 5 --datapoints-to-alarm 5 \ --threshold 1 --comparison-operator LessThanThreshold \ --treat-missing-data breaching \ --alarm-actions <sns-topic-arn>

The LoadBalancer and TargetGroup dimension values are the last part of each resource’s ARN, starting at app/ and targetgroup/ respectively. You can copy them from the EC2 → Load Balancers and Target groups pages.


Next steps

Last updated on