High Availability
The standard AlertD deployment is built to recover from failures on its own, but it runs a single copy of each component to keep cost down. For teams that need faster recovery or protection against the loss of an entire Availability Zone, AlertD offers a High Availability (HA) configuration: a variant of the CloudFormation template that deploys the same application with redundant components, which we enable together with your team. This page explains what the standard deployment already does, what the HA template adds, and what it costs to run.
High availability is optional. AlertD is a monitoring and analysis tool, not a system in the request path of your own applications. Many teams are comfortable with the few minutes of automatic recovery the standard deployment provides.
What the standard deployment already does
Every AlertD deployment, standard or HA, has the following built in:
- Two Availability Zones. The private subnets, the load balancer, the EFS mount targets and the ECS services all span two Availability Zones (AZs). When a task fails, ECS can start its replacement in the other AZ.
- Automatic task replacement. ECS keeps one application task and one messaging (Pulsar) task running at all times. If a task crashes or fails its health check, ECS starts a new one. Expect a few minutes of downtime while the new task starts.
- Safe updates. Both services use the ECS deployment circuit breaker. If a new version does not start, ECS rolls back to the previous one automatically. AlertD’s auto-update function also rolls the application back to the last known-good version if it stays down for an extended period.
- Durable storage. Aurora keeps six copies of your data across three AZs, regardless of how many database instances you run. Amazon S3 and Amazon EFS are regional services that are highly available by design.
- Global entry point. CloudFront serves the application from AWS edge locations and is not tied to a single AZ.
Single points of failure in the standard deployment
| Component | Standard deployment | Effect of a failure |
|---|---|---|
| Application task | One ECS task | ECS starts a replacement. Recovery takes a few minutes while the new task is scheduled, pulls its image and starts up. |
| Database instance | One Aurora Serverless v2 instance | Aurora recreates the instance, typically in under 10 minutes. During an AZ-wide outage it cannot, and an instance must be added in another AZ by hand. |
| Internet access (NAT gateway) | One NAT gateway, in one AZ | AlertD cannot reach AWS APIs, Amazon Bedrock or AlertD’s sign-in service until the AZ recovers, even if its tasks are running in the other AZ. Applies only when the stack created its own VPC. |
| Messaging task (Pulsar) | One ECS task | ECS starts a replacement in a minute or two. Work in progress at that moment runs again on its next schedule. No change is needed; see below. |
What the High Availability template adds
The HA template is a complete AlertD template, not an add-on. It takes the same parameters as the standard one and deploys the same application, with these differences:
Application: a hot standby
AlertD runs as a single orchestrator process that owns the data-collection scheduler, the analyses and the application’s shared files. Only one orchestrator may be active against a database at a time, and the application enforces this itself. High availability for the application is therefore active/standby rather than active/active: a second copy does not add capacity, it stands ready to take over.
The HA template creates two application services in the ECS cluster, each running one task from the same image and configuration, and gives each its own target group on the load balancer. The load balancer’s routing rule sends 100% of traffic to the active target group and 0% to the standby’s. The standby task is running and health-checked the whole time, but receives no user traffic.
Failover is automatic:
- The active task stops or fails its health check, and its target group has no healthy target left
- A CloudWatch alarm on that target group fires and invokes a small Lambda function created by the template
- The function swaps the weights on the routing rule, so 100% of traffic now goes to the standby’s target group
- The standby takes over the orchestrator role as soon as the failed task has released it, finishes its start-up and begins serving
- ECS replaces the failed task. The new task starts in waiting mode and becomes the standby
The roles simply swap on each failure, so nothing has to be switched back by hand. Because the standby task is already running, recovery skips the scheduling and image-pull time that the standard deployment needs; what remains is the health-check detection time and the application’s own start-up.
Database: a reader instance in a second AZ
Aurora separates storage from compute. Your data is always stored in three AZs, but the compute that serves queries is a database instance, and the standard template creates only one. The HA template adds a reader instance in a different AZ, which makes the cluster Multi-AZ.
The reader is kept at a low capacity while the writer is healthy, so it costs little to run. If the writer fails, Aurora promotes the reader to writer, typically in under 60 seconds, and Aurora Serverless v2 then scales it up automatically to match the workload. AlertD connects through the cluster’s writer endpoint, which always points at the current writer, so nothing in the application changes.
Network: a NAT gateway in each AZ
When the template creates the VPC, the HA variant provisions one NAT gateway per Availability Zone, each serving the private subnet in its own AZ. An outage of one AZ no longer cuts off internet access for tasks running in the other.
Everything else: unchanged
The messaging service, the EFS file system, the S3 bucket, CloudFront, the load balancer itself and the auto-update function are identical to the standard deployment.
| Component | Standard | High Availability |
|---|---|---|
| Application | 1 task, 1 target group | 2 tasks in 2 target groups (active and hot standby), routing weights 100/0, swapped automatically on failure |
| Database | 1 Aurora Serverless v2 instance | Writer plus a low-capacity reader in another AZ, scales up on failover |
| NAT gateways (new VPC) | 1 | 1 per AZ (2) |
| Messaging (Pulsar) | 1 task | 1 task |
| Recovery from an application task failure | A few minutes | Health-check detection plus application start-up, no scheduling or image-pull delay |
| Recovery from a database instance failure | Under 10 minutes, same AZ only | Under a minute, survives an AZ outage |
What it costs
High availability means paying for resources that sit idle most of the time. The HA configuration adds the following resources to a standard deployment:
| Additional resource | How it is billed |
|---|---|
| Standby application task (2 vCPU, 4 GB) on AWS Fargate | Per vCPU-hour and per GB-hour, around the clock |
| Aurora Serverless v2 reader instance | Per Aurora Capacity Unit (ACU) hour. Idles at the cluster’s minimum capacity and scales up only after a failover |
| Second NAT gateway (new VPC only) | Per hour, plus per GB of data processed. Traffic is split between the two gateways, so the total processed volume is about the same |
| Failover alarm and Lambda function | Per alarm and per invocation, a few per month |
Storage, backups, CloudFront and the load balancer cost the same in both modes. Use the AWS Pricing Calculator with the figures above and your Region to estimate the difference for your deployment.
Enabling High Availability
High Availability is enabled together with AlertD. Rather than publishing the HA template for self-service deployment, we schedule a session with your team and walk through the deployment, or the move from an existing standard installation, with you. This keeps the switch predictable: HA changes how the application fails over, and we want to confirm with you that every piece behaves as expected before you rely on it.
To get started, contact your AlertD representative or support and ask to enable High Availability. Let us know:
- Whether this is a new installation or an existing one you want to upgrade with its data in place
- The Region you deploy in
- For an existing installation, the AlertD version you are running (the
Versionparameter of your stack)
We will schedule time with you to enable it. Once it is done, you will see two application services in the ECS cluster, two target groups on the load balancer, two instances under the Aurora cluster (the cluster’s Multi-AZ column reads Yes) and, for a new VPC, two NAT gateways in the VPC console.
Test it
You can rehearse both failovers from the AWS console at any time:
- Database: open RDS → Databases, select the cluster
<stack-name>-aurora-cluster, then Actions → Failover. Aurora swaps the writer and reader roles. Open connections are dropped, AlertD reconnects to the new writer, and any analysis running at that moment is retried. - Application: open ECS → Clusters →
<stack-name>-cluster → Tasks, select the active application task and click Stop. Within the health-check window the alarm fires, the routing weights swap to the standby, and ECS starts a replacement task that becomes the new standby. You can watch the swap on the load balancer’s listener rule in the EC2 console.
Both tests cause a brief interruption for signed-in users, so run them outside working hours the first time.
Messaging (Pulsar)
AlertD uses Apache Pulsar as an internal message bus between the orchestrator and its background work. It carries messages that are in flight; it does not store your data or configuration, all of which live in the database, the S3 bucket and the EFS file system.
For that reason the messaging service does not need high availability, and the HA template leaves it as a single task. If the task fails, ECS starts a replacement within a minute or two, and the application reconnects to it automatically. Scheduled work that was interrupted runs again on its next scheduled run. Making Pulsar itself highly available would require a multi-node cluster that costs far more than the brief interruption it prevents.
For protection against the loss of an entire AWS Region, see Backup and Recovery.