Monitoring AlertD
AlertD runs entirely inside your AWS account, on services you already know how to watch. Every component publishes metrics to Amazon CloudWatch, and the application writes its logs to CloudWatch Logs. This page explains what the deployment is made of, which metrics tell you it is healthy, and which alarms we recommend.
The AlertD stack does not create CloudWatch alarms for itself. Everything below is a recommendation you set up once, in the CloudWatch console, after deployment.
What AlertD deploys
Throughout this page, <stack-name> is the name you gave the CloudFormation stack when you deployed AlertD (for example alertd-prod).
| Component | AWS service | Where to find it |
|---|---|---|
| Application container | Amazon ECS on AWS Fargate | ECS cluster <stack-name>-cluster, service whose name starts with <stack-name>-SvcApp |
| Messaging container (Apache Pulsar) | Amazon ECS on AWS Fargate | Same cluster, service whose name starts with <stack-name>-SvcPulsar |
| Database | Amazon Aurora PostgreSQL (Serverless v2) | RDS cluster <stack-name>-aurora-cluster, instance <stack-name>-aurora-instance |
| Application data | Amazon EFS | File system named <stack-name>-app-data |
| Reports and storage | Amazon S3 | Bucket name is in the stack output LoggingBucketName |
| Internal load balancer | Application Load Balancer | EC2 console → Load Balancers, tagged with your stack |
| Public entry point | Amazon CloudFront | Distribution with the comment <stack-name> AlertD distribution |
| Auto-update and self-healing | AWS Lambda, Amazon EventBridge, Amazon SNS | Lambda function and SNS topic both named <stack-name>-auto-update |
| Application logs | CloudWatch Logs | Log group /ecs/<stack-name> |
ECS service names end in a random suffix that CloudFormation generates. To list them, open ECS → Clusters → <stack-name>-cluster → Services, or run:
aws ecs list-services --cluster <stack-name>-clusterMetrics to watch
Open CloudWatch → Metrics → All metrics and pick the namespace from the table. The columns show the metric name, what a bad value means for AlertD, and a sensible alarm threshold to start with.
Application and messaging containers
The ECS cluster has Container Insights enabled, so you get per-service CPU, memory and task-count metrics under the ECS/ContainerInsights namespace in addition to the standard AWS/ECS metrics.
| Namespace / metric | What it tells you | Suggested alarm |
|---|---|---|
ECS/ContainerInsights → RunningTaskCount | Number of application (or Pulsar) tasks actually running. Should be 1 for each service. | Below 1 for 5 minutes |
AWS/ECS → CPUUtilization | CPU used by the service as a percentage of what it reserved. Sustained high values slow down analyses. | Above 85% for 15 minutes |
AWS/ECS → MemoryUtilization | Memory used as a percentage of what the service reserved. Near 100% means tasks are at risk of being restarted. | Above 85% for 15 minutes |
Filter the metrics by ClusterName = <stack-name>-cluster and the ServiceName you found above.
Load balancer
The internal Application Load Balancer sits between CloudFront and the application. Its metrics are the clearest signal that the web interface is up. Use the AWS/ApplicationELB namespace and filter by the load balancer and target group of your stack.
| Metric | What it tells you | Suggested alarm |
|---|---|---|
HealthyHostCount | Application tasks passing the load balancer health check. Should be 1 per target group (the High Availability configuration has two target groups, one per application service). | Below 1 for 5 minutes |
UnHealthyHostCount | Tasks failing the health check. | Above 0 for 10 minutes |
HTTPCode_Target_5XX_Count | Server errors returned by the application. | Above 10 in 5 minutes |
TargetResponseTime | How long the application takes to answer. | p95 above 5 seconds for 15 minutes |
Database
The Aurora Serverless v2 cluster scales its capacity automatically between the minimum and maximum you set at deployment (the AuroraMinCapacity and AuroraMaxCapacity parameters, measured in Aurora Capacity Units, or ACUs). Use the AWS/RDS namespace and filter by DBClusterIdentifier = <stack-name>-aurora-cluster.
| Metric | What it tells you | Suggested alarm |
|---|---|---|
ServerlessDatabaseCapacity | Current number of ACUs in use. If it sits at your maximum for long periods, raise AuroraMaxCapacity with a stack update. | At or near the maximum for 30 minutes |
ACUUtilization | Capacity in use as a percentage of the maximum. | Above 90% for 30 minutes |
CPUUtilization | Database CPU. | Above 90% for 15 minutes |
DatabaseConnections | Open connections from the application. A sudden drop to zero usually means the application is down. | Equal to 0 for 10 minutes |
FreeableMemory | Memory available to the database. | Below 10% of the current capacity |
Application data (EFS)
The file system uses Elastic throughput, so there are no burst credits to run out of. Use the AWS/EFS namespace and filter by FileSystemId (shown in the stack output AppDataFileSystemId).
| Metric | What it tells you | Suggested alarm |
|---|---|---|
PercentIOLimit | How close the file system is to its I/O limit. | Above 90% for 15 minutes |
ClientConnections | Application tasks connected to the file system. Should match the number of running application tasks. | Equal to 0 for 10 minutes |
StorageBytes | Total data stored. Useful for tracking growth and cost. | Informational |
Public entry point (CloudFront)
CloudFront metrics are always published in the US East (N. Virginia) Region, whichever Region you deployed AlertD to. Switch to us-east-1 in the CloudWatch console, choose the AWS/CloudFront namespace and filter by your distribution.
| Metric | What it tells you | Suggested alarm |
|---|---|---|
5xxErrorRate | Percentage of requests that failed at the application or load balancer. | Above 5% for 5 minutes |
4xxErrorRate | Client errors, mostly expired sessions and bad URLs. A sharp rise after a change is worth a look. | Informational |
Requests | Traffic volume. | Informational |
Auto-update and self-healing
A scheduled Lambda function checks that the application service is running and, when it is healthy, looks for signed AlertD releases to apply. If the application has been down for more than two hours, it rolls the service back to the last known-good version. Use the AWS/Lambda namespace and filter by FunctionName = <stack-name>-auto-update.
| Metric | What it tells you | Suggested alarm |
|---|---|---|
Errors | The check itself failed to run. | Above 0 in 1 hour |
Invocations | The check runs several times an hour. | Below 1 in 1 hour |
This component exists only when the stack was deployed with EnableAutoUpdate set to true (the default).
Notifications from the auto-update check
The auto-update function publishes its decisions (for example a rollback to the last known-good version) to the SNS topic <stack-name>-auto-update. To be told when it acts, open SNS → Topics → <stack-name>-auto-update → Create subscription and subscribe an email address, a chat integration or the alerting pipeline you already use.
Application logs
All container output goes to the CloudWatch Logs group /ecs/<stack-name>. Each task writes to its own log stream; the stream name starts with ecs/app/ for the application and ecs/pulsar/ for the messaging container.
-
Open CloudWatch → Log groups →
/ecs/<stack-name>and use Logs Insights to search across streams. A good first query during an incident:fields @timestamp, @message | filter @message like /ERROR|FATAL/ | sort @timestamp desc | limit 100 -
Log retention is 7 days and is managed by the stack. If you need to keep logs longer, add a subscription filter to forward them to Amazon S3, Amazon OpenSearch Service or your log platform. Changing the retention directly on the log group is possible, but the next stack update sets it back to 7 days.
Logs are also searchable from inside AlertD, in the diagnostics views, without leaving the application. CloudWatch Logs is the place to look when the application itself is not reachable.
How AlertD checks its own health
Two health checks run continuously, and both target the application’s web port:
- Container health check — ECS requests the application’s home page every 60 seconds. After 5 consecutive failures the task is marked unhealthy and ECS replaces it. New tasks get a 5-minute grace period to start.
- Load balancer health check — The load balancer makes the same request every 60 seconds. A task needs 2 consecutive successes to receive traffic and is taken out of rotation after 5 consecutive failures.
Both ECS services have the deployment circuit breaker enabled. If a new version fails to start during an update, ECS rolls the service back to the previous version automatically.
Creating an alarm
All of the alarms above are created the same way. As an example, here is the most important one: the application has no healthy task behind the load balancer.
- Open CloudWatch → Alarms → All alarms → Create alarm
- Click Select metric, choose ApplicationELB → Per AppELB, per TG Metrics, and tick
HealthyHostCountfor your stack’s load balancer and target group - Set Statistic to
Minimumand Period to1 minute - Under Conditions, choose Lower than
1, and set Datapoints to alarm to5 out of 5 - Under Notification, pick an SNS topic that reaches your on-call rotation (create one here if you do not have one)
- Name the alarm, for example
alertd-no-healthy-tasks, and click Create alarm
Repeat for the database and container alarms in the tables above. If you prefer the command line, the same alarm is:
aws cloudwatch put-metric-alarm \
--alarm-name alertd-no-healthy-tasks \
--namespace AWS/ApplicationELB \
--metric-name HealthyHostCount \
--dimensions Name=LoadBalancer,Value=<app/.../...> Name=TargetGroup,Value=<targetgroup/.../...> \
--statistic Minimum --period 60 \
--evaluation-periods 5 --datapoints-to-alarm 5 \
--threshold 1 --comparison-operator LessThanThreshold \
--treat-missing-data breaching \
--alarm-actions <sns-topic-arn>The LoadBalancer and TargetGroup dimension values are the last part of each resource’s ARN, starting at app/ and targetgroup/ respectively. You can copy them from the EC2 → Load Balancers and Target groups pages.
Next steps
- High Availability — remove the single points of failure in the default deployment
- Backup and Recovery — protect your data and rebuild AlertD in another Region