[BUG]: Fail to deploy on Github Actions

[coding-velociraptor]

Description:

Starting on Monday 23th February, we are unable to use Copilot v1.34.0 in our Github Actions. When ever we run any copilot svc deploy... command, it fails with the following error:

execute deployment 1 of 1 in group 1: deploy service backend to environment staging: deploy service: generate template: override template: unmarsal CDK transformed YAML template: yaml: line 1041: could not find expected ':'

The same output happens when using copilot svc package.... We haven't change any dependency or infrastructure. Dependencies are pinned to be:

{
    "aws-cdk-lib": "2.141.0",
    "aws-cdk": "2.141.0",
}

A later cdk version doesn't work either.

Any advice or hint is highly appreciated.

Update:

It works without the CDK overrides.

Details:

Linux + Mac
Copilot v1.34.0

Observed result:

execute deployment 1 of 1 in group 1: deploy service backend to environment staging: deploy service: generate template: override template: unmarsal CDK transformed YAML template: yaml: line 1041: could not find expected ':'

Expected result:

No errors.

Debugging:

Different Github infra, later cdk version

[felixwuerthcs]

We're facing the exact same issue

[felixwuerthcs]

this might be related (date/time aligns) to the release of a new github actions standard ubuntu runner that updated aws library versions: https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md

[coding-velociraptor]

We have tried a Mac OS X runner instead, however it's the same result. Also, we have seen failures before the image change. Are you able to address it with another image?

[felixwuerthcs]

it seems that you can't define 24.04.1 as a release for the github hosted standard runner. i'm currently trying to override the newer versions of aws cli and sam cli by the old ones within the workflow. this is a workaround to unblock our teams. but it seems that you need to adopt to the updated versions as there's no going back. and overriding it is not so efficent.

[felixwuerthcs]

this did not help, unfortunately. also downgrading to 22.04 https://github.com/actions/runner-images/blob/ubuntu24/20250223.1/images/ubuntu/Ubuntu2204-Readme.md which does not seem to have all these library updates, did not help.

[coding-velociraptor]

Are you able to replicate it on our local machine with any aws-cli, aws-cdk, aws-cdk-lib, cdk combination?

[felixwuerthcs]

no, deployment from local (macOS) to AWS seems to work as far as I know

[felixwuerthcs]

the only solution for us so far is using docker to build and deploy in the github action.

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build and run deployment container
        run: |
          docker build -t deploy-image -f Dockerfile.deploy .
          docker run --rm \
            -v ${{ github.workspace }}:/app \
            deploy-image

(i removed specifics from our setup)

[coding-velociraptor]

@felixwuerthcs thanks for sharing. We were able to mitigate using the Docker image approach. This issue kick-starts our migration away from AWS Copilot CLI, also considering alternatives.

[sveneh]

The issue is the CDK Notice system. The CDK-generated Cloudformation gets mixed with Text-based notices which the CDK pushes sometimes.

Our workaround was to add

{
  "notices": false,
  "context": {
    // ...
  }
}

to cdk.json

[bathsundeep-graticule]

The issue is the CDK Notice system. The CDK-generated Cloudformation gets mixed with Text-based notices which the CDK pushes sometimes.

Thank you for this, easy fix. CDK docs suggest you could set this with an env var in your runner as well CDK_NOTICES=false