ECR token pushes the line over 4096. Which is unresolvable without rebuilding NGINX

[matthew-williams]

[ecr-proxy-7d847d9f8d-ph4cj] 2025/01/14 18:46:41 [emerg] 62#62: too long parameter, probably missing terminating """ character in /usr/local/openresty/nginx/conf/nginx.conf:82                                                                                                                                       
[ecr-proxy-7d847d9f8d-ph4cj] nginx: [emerg] too long parameter, probably missing terminating """ character in /usr/local/openresty/nginx/conf/nginx.conf:82                                                                                                                                                           
[ecr-proxy-7d847d9f8d-ph4cj] 2025-01-14 18:46:41,129 WARN exited: nginx (exit status 1; not expected)                                                                                                                                                                                                                 
[ecr-proxy-7d847d9f8d-ph4cj] 2025-01-14 18:46:44,133 INFO spawned: 'nginx' with pid 63                                                                                                                                                                                                                                
[ecr-proxy-7d847d9f8d-ph4cj] 2025/01/14 18:46:44 [emerg] 63#63: too long parameter, probably missing terminating """ character in /usr/local/openresty/nginx/conf/nginx.conf:82                                                                                                                                       
[ecr-proxy-7d847d9f8d-ph4cj] nginx: [emerg] too long parameter, probably missing terminating """ character in /usr/local/openresty/nginx/conf/nginx.conf:82                                                                                                                                                           
[ecr-proxy-7d847d9f8d-ph4cj] 2025-01-14 18:46:44,142 WARN exited: nginx (exit status 1; not expected)   

After further research we discovered all the Token lines were around 4698 characters long, which is way past NGINX limit. Without rebuilding NGINX there is not a way to recover from this error as far as I can tell.

[egeturgay]

I've come across the same problem on a similar stack, not using the same code/repo so I can't provide you a PR/diff but here's how I worked around it. I hope it helps

nginx.conf

    set $auth_part1 "PART1";
    set $auth_part2 "PART2";
    set $aws_auth_header "$auth_part1$auth_part2";


      proxy_set_header  X-Forwarded-User   "Basic $aws_auth_header";
      proxy_set_header  Authorization      "Basic $aws_auth_header";

renew_token.sh

while true; do
  TOKEN="$(aws ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken')"
  length=${#TOKEN}
  middle=$((length / 2))
  part1=${TOKEN:0:middle}
  part2=${TOKEN:middle}
  [ -n "${TOKEN}" ] && break
  echo "Warn: Unable to get new token, wait and retry!"
  sleep 30
done

old_part1=$(grep -m1 'auth_part1' "$CONFIG" | sed -E 's/.*auth_part1.*"([^"]+)".*/\1/')
old_part2=$(grep -m1 'auth_part2' "$CONFIG" | sed -E 's/.*auth_part2.*"([^"]+)".*/\1/')

sed -i "s|$old_part1|$part1|g" "$CONFIG"
sed -i "s|$old_part2|$part2|g" "$CONFIG"

startup.sh

TOKEN="$(aws ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken')"
length=${#TOKEN}
middle=$((length / 2))
part1=${TOKEN:0:middle}
part2=${TOKEN:middle}

sed -i "s|PART1|$part1|g" "$CONFIG"
sed -i "s|PART2|$part2|g" "$CONFIG"

[matthew-williams]

I've come across the same problem on a similar stack, not using the same code/repo so I can't provide you a PR/diff but here's how I worked around it. I hope it helps

Nice! I totally solved it using a lua block before I saw your reply, but I think either way works.:

• renew_token.sh

# replace the existing token with the new token we retrieved from AWS in the config
# sed -i "s|${ESC}{EXISTING_TOKEN%??}|${ESC}{NEW_TOKEN}|g" $NGINX_CONFIG_DIR/nginx.conf

# create the token file with the new token to fix ecr tokens being too long
echo -n "Basic ${ESC}{NEW_TOKEN}" >> $NGINX_CONFIG_DIR/token
sed 's/^/"/;s/$/"/' $NGINX_CONFIG_DIR/token

• nginx.conf

  server {
    listen $PROXY_$PROXY_PORT $PROXY_LISTENER_OPTIONS default_server;

    set_by_lua_block ${ESC}auth_headers {
        local f = io.open("/usr/local/openresty/nginx/conf/token")
        local c = f:read("*all")
        f:close()
        return c
      }

and

      # Add AWS ECR authentication headers
      proxy_set_header  X-Real-IP          ${ESC}remote_addr;
      proxy_set_header  X-Forwarded-For    ${ESC}remote_addr;
      proxy_set_header  X-Forwarded-User   ${ESC}auth_headers;
      proxy_set_header  Authorization      ${ESC}auth_headers;
      proxy_set_header  X-Forwarded-Proto  ${ESC}scheme;

I'll probably submit a PR at some point.

[whitfin]

Hi both!

Thanks for looking at this; is this some new change to ECR? I've never seen it but admittedly haven't used this project for a few months. Did they change their token format or something?

Totally happy to accept a PR for this; I can review or look at fixing myself this weekend as needed too!

[matthew-williams]

Thanks for looking at this; is this some new change to ECR? I've never seen it but admittedly haven't used this project for a few months. Did they change their token format or something?

Not as far as I can tell

Totally happy to accept a PR for this; I can review or look at fixing myself this weekend as needed too!

• Feat fix token size #2

Side note this also all works on FROM openresty/openresty:1.27.1.1-alpine which I am using locally. But, I left that out of the PR cause I'll leave that up to you to change versions of things.

[egeturgay]

I think AWS has increased its token size hence the problem.
The returned token on my local laptop is still small but when it's requested within an EC2 instance, it seems to go beyond 4096 bytes.
Also, the first request to the API endpoint usually returns a shorter token but the subsequent requests receive a larger token.
I had a look at the payload and it's the encrypted bit which is longer.
I also tried against the public API endpoint vs metadata service, they both return the long token.

[matthew-williams]

I think AWS has increased its token size hence the problem. The returned token on my local laptop is still small but when it's requested within an EC2 instance, it seems to go beyond 4096 bytes. Also, the first request to the API endpoint usually returns a shorter token but the subsequent requests receive a larger token. I had a look at the payload and it's the encrypted bit which is longer. I also tried against the public API endpoint vs metadata service, they both return the long token.

My PR should fix that, it just outputs the token to a file and uses lua to manage the token injection into the headers.

[whitfin]

I'll get to reviewing that ASAP. It looks like this is something recent as the upstream repository also hit it: Lotto24/aws-ecr-http-proxy#31

Edit: oh, ha, I didn't even realise it was you filing that @egeturgay! That other stack is no longer maintained AFAICT, which is why I forked over to here initially. I'll try get this merged and pushed out this weekend!

[whitfin]

I've merged this over, and double checked that it's all still working. I'm going to close this, and I'll push out a new image to Docker Hub as soon as we resolve #3 :) thank you both!