Deployment ECS on EC2 not working
Postmortem

While adding container command with out latest monthly release (see Releasenotes) we hit a corner case w/ ECS containers running on EC2 and enabling the required feature for the execute commands. The issue only affected new deployments, already running services continued to work correctly. The AWS support was helpful to debug the issue under the hood as the ECS console and API did not yield any helpful error messages indicating the cause.

As a result, we released a new version of the r3 CLI stack which ensures that the feature will be disabled by default for services running on EC2 clusters and enabled for FARGATE services only.

Further work will be done with AWS to find out why the feature is not yet working on EC2 despite the official documentation says so.

We apologize for the inconvenience!

Posted Jul 02, 2024 - 11:09 CEST

Resolved
This incident has been resolved.
Posted Jul 02, 2024 - 10:59 CEST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 01, 2024 - 11:12 CEST
Identified
The issue has been identified and a fix is being implemented.
Posted Jun 25, 2024 - 17:02 CEST
Investigating
We are currently investigate this issue.
As a side note: ECS on Fargate is still working.
Posted Jun 25, 2024 - 13:37 CEST
This incident affected: r3 suite.