Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Linkerd2-proxy failed to run with privileged permission #10722

Closed
daixiang0 opened this issue Apr 10, 2023 · 6 comments · May be fixed by linkerd/linkerd2-proxy-init#225
Closed

Linkerd2-proxy failed to run with privileged permission #10722

daixiang0 opened this issue Apr 10, 2023 · 6 comments · May be fixed by linkerd/linkerd2-proxy-init#225
Assignees
@adleong @mateiidavid
Labels
bug wontfix

Comments

@daixiang0
Copy link
Contributor

daixiang0 commented Apr 10, 2023 •
edited
Loading

What is the issue?

Linkerd2-proxy failed to run with privileged permission in CentOS 8.

The install params are --set proxyInit.runAsRoot=true --set "proxyInit.iptablesMode=nft".

Try many version, all fail.

How can it be reproduced?

diff --git a/charts/partials/templates/_proxy.tpl b/charts/partials/templates/_proxy.tpl
index 8caf7d384..f9a633340 100644
--- a/charts/partials/templates/_proxy.tpl
+++ b/charts/partials/templates/_proxy.tpl
@@ -156,15 +156,9 @@ readinessProbe:
 {{ include "partials.resources" .Values.proxy.resources }}
 {{- end }}
 securityContext:
-  allowPrivilegeEscalation: false
-  {{- if .Values.proxy.capabilities -}}
-  {{- include "partials.proxy.capabilities" . | nindent 2 -}}
-  {{- end }}
-  readOnlyRootFilesystem: true
-  runAsNonRoot: true
-  runAsUser: {{.Values.proxy.uid}}
-  seccompProfile:
-    type: RuntimeDefault
+  privileged: true
+  runAsNonRoot: false
+  runAsUser: 0
 terminationMessagePolicy: FallbackToLogsOnError
 {{- if or (.Values.proxy.await) (.Values.proxy.waitBeforeExitSeconds) }}
 lifecycle:

Then deploy as below:

bin/linkerd install --crds | kubectl apply -f - &&  bin/linkerd install --set proxyInit.runAsRoot=true    --set "proxyInit.iptablesMode=nft"   | kubectl apply -f -

Logs, error output, etc

It failed to work in 2m:

linkerd        linkerd-destination-bd75949c5-mw927             0/4     PostStartHookError   4 (2s ago)   5m7s    10.244.0.79   spr-loong.localdomain   <none>           <none>
linkerd        linkerd-identity-6fc966499f-x9qf2               2/2     Running              0            5m7s    10.244.0.80   spr-loong.localdomain   <none>           <none>
linkerd        linkerd-proxy-injector-c7cdd5c74-49xjk          0/2     PostStartHookError   1 (2s ago)   5m7s    10.244.0.81   spr-loong.localdomain   <none>           <none>

Then destination and proxy-injector pods crashed.

destination pod log:

[   133.029989s] ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=status: Unknown, message: "controller linkerd-identity-headless.linkerd.svc.cluster.local:8080: service in fail-fast", details: [], metadata: MetadataMap { headers: {} } error.sources=[controller linkerd-identity-headless.linkerd.svc.cluster.local:8080: service in fail-fast, service in fail-fast]
[   135.014859s]  WARN ThreadId(01) linkerd_app: Waiting for identity to be initialized...
[   144.032957s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.80:8080}: linkerd_reconnect: Failed to connect error=endpoint 10.244.0.80:8080: connect timed out after 1s error.sources=[connect timed out after 1s]
[   145.136510s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:endpoint{addr=10.244.0.80:8080}: linkerd_reconnect: Failed to connect error=endpoint 10.244.0.80:8080: connect timed out after 1s error.sources=[connect timed out after 1s]
[   146.031966s]  WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_stack::failfast: Service entering failfast after 3s
[   146.032047s] ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=status: Unknown, message: "controller linkerd-identity-headless.linkerd.svc.cluster.local:8080: service in fail-fast", details: [], metadata: MetadataMap { headers: {} } error.sources=[controller linkerd-identity-headless.linkerd.svc.cluster.local:8080: service in fail-fast, service in fail-fast]
[   150.015555s]  WARN ThreadId(01) linkerd_app: Waiting for identity to be initialized...

output of linkerd check -o short

N/A

Environment

  • K8s version: 1.26
  • Host OS: CentOS 8
  • Linkerd2: edge-23.3.1 and some others

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

yes
@daixiang0 daixiang0 added the bug label Apr 10, 2023
@adleong
Copy link
Member

adleong commented Apr 11, 2023

Hi @daixiang0, is there any additional information in the logs of the proxy-init containers?

@daixiang0
Copy link
Contributor Author

@adleong I enable debug log:

time="2023-04-12T01:44:17Z" level=debug msg="tracing script execution as [1681263857]"
time="2023-04-12T01:44:17Z" level=debug msg="using 'iptables-nft' to set-up firewall rules"
time="2023-04-12T01:44:17Z" level=debug msg="using 'iptables-nft-save' to list all available rules"
time="2023-04-12T01:44:17Z" level=info msg="/sbin/iptables-nft-save -t nat"
time="2023-04-12T01:44:17Z" level=info msg="# Generated by iptables-nft-save v1.8.8 (nf_tables) on Wed Apr 12 01:44:17 2023\n*nat\n:PREROUTING ACCEPT [0:0]\n:INPUT ACCEPT [0:0]\n:OUTPUT ACCEPT [0:0]\n:POSTROUTING ACCEPT [0:0]\nCOMMIT\n# Completed on Wed Apr 12 01:44:17 2023\n"
time="2023-04-12T01:44:17Z" level=info msg="/sbin/iptables-nft -t nat -N PROXY_INIT_REDIRECT"
time="2023-04-12T01:44:17Z" level=info msg="/sbin/iptables-nft -t nat -A PROXY_INIT_REDIRECT -p tcp --match multiport --dports 4190,4191,4567,4568 -j RETURN -m comment --comment proxy-init/ignore-port-4190,4191,4567,4568/1681263857"
time="2023-04-12T01:44:17Z" level=info msg="/sbin/iptables-nft -t nat -A PROXY_INIT_REDIRECT -p tcp -j REDIRECT --to-port 4143 -m comment --comment proxy-init/redirect-all-incoming-to-proxy-port/1681263857"
time="2023-04-12T01:44:17Z" level=info msg="/sbin/iptables-nft -t nat -A PREROUTING -j PROXY_INIT_REDIRECT -m comment --comment proxy-init/install-proxy-init-prerouting/1681263857"
time="2023-04-12T01:44:17Z" level=info msg="/sbin/iptables-nft -t nat -N PROXY_INIT_OUTPUT"
time="2023-04-12T01:44:17Z" level=info msg="/sbin/iptables-nft -t nat -A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -j RETURN -m comment --comment proxy-init/ignore-proxy-user-id/1681263857"
time="2023-04-12T01:44:17Z" level=info msg="/sbin/iptables-nft -t nat -A PROXY_INIT_OUTPUT -o lo -j RETURN -m comment --comment proxy-init/ignore-loopback/1681263857"
time="2023-04-12T01:44:17Z" level=info msg="/sbin/iptables-nft -t nat -A PROXY_INIT_OUTPUT -p tcp --match multiport --dports 4567,4568 -j RETURN -m comment --comment proxy-init/ignore-port-4567,4568/1681263857"
time="2023-04-12T01:44:17Z" level=info msg="/sbin/iptables-nft -t nat -A PROXY_INIT_OUTPUT -p tcp -j REDIRECT --to-port 4140 -m comment --comment proxy-init/redirect-all-outgoing-to-proxy-port/1681263857"
time="2023-04-12T01:44:17Z" level=info msg="/sbin/iptables-nft -t nat -A OUTPUT -j PROXY_INIT_OUTPUT -m comment --comment proxy-init/install-proxy-init-output/1681263857"
time="2023-04-12T01:44:17Z" level=info msg="/sbin/iptables-nft-save -t nat"
time="2023-04-12T01:44:17Z" level=info msg="# Generated by iptables-nft-save v1.8.8 (nf_tables) on Wed Apr 12 01:44:17 2023\n*nat\n:PREROUTING ACCEPT [0:0]\n:INPUT ACCEPT [0:0]\n:OUTPUT ACCEPT [0:0]\n:POSTROUTING ACCEPT [0:0]\n:PROXY_INIT_OUTPUT - [0:0]\n:PROXY_INIT_REDIRECT - [0:0]\n-A PREROUTING -m comment --comment \"proxy-init/install-proxy-init-prerouting/1681263857\" -j PROXY_INIT_REDIRECT\n-A OUTPUT -m comment --comment \"proxy-init/install-proxy-init-output/1681263857\" -j PROXY_INIT_OUTPUT\n-A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -m comment --comment \"proxy-init/ignore-proxy-user-id/1681263857\" -j RETURN\n-A PROXY_INIT_OUTPUT -o lo -m comment --comment \"proxy-init/ignore-loopback/1681263857\" -j RETURN\n-A PROXY_INIT_OUTPUT -p tcp -m multiport --dports 4567,4568 -m comment --comment \"proxy-init/ignore-port-4567,4568/1681263857\" -j RETURN\n-A PROXY_INIT_OUTPUT -p tcp -m comment --comment \"proxy-init/redirect-all-outgoing-to-proxy-port/1681263857\" -j REDIRECT --to-ports 4140\n-A PROXY_INIT_REDIRECT -p tcp -m multiport --dports 4190,4191,4567,4568 -m comment --comment \"proxy-init/ignore-port-4190,4191,4567,4568/1681263857\" -j RETURN\n-A PROXY_INIT_REDIRECT -p tcp -m comment --comment \"proxy-init/redirect-all-incoming-to-proxy-port/1681263857\" -j REDIRECT --to-ports 4143\nCOMMIT\n# Completed on Wed Apr 12 01:44:17 2023\n"

Not much useful info.

Could one of you (linkerd2-proxy experts) try to deploy with privileged proxy?

This failure makes me very confused that why it works with limited permission but fail with full permission.

@daixiang0
Copy link
Contributor Author

daixiang0 commented Apr 13, 2023 •
edited
Loading

After arduous debug, I find the cause is proxy-uid in proxy-init.

When I set proxy run as root (user 0), but I do not sync proxy-uid, so the rule does not ignore the traffic from root which lead to failure.

But in the proxy-init code, it can not ignore root user:

https://github.com/linkerd/linkerd2-proxy-init/blob/a18218e878664829d7a4a86905d80a5c45c09a32/internal/iptables/iptables.go#L116-L119

@olix0r is there any history issue for it?

@daixiang0 daixiang0 mentioned this issue Apr 13, 2023
Draft
@adleong
Copy link
Member

adleong commented Apr 13, 2023

@daixiang0 can you give us some more context about what problem you're trying to solve? why do you need to run the proxy as root? generally, we want to run the proxy with a UID that we know isn't used by any other processes so that we can identify which packets are coming from the proxy by their UID. Running the proxy as root could be problematic since there may be other processes also running as root.

@daixiang0
Copy link
Contributor Author

daixiang0 commented Apr 14, 2023 •
edited
Loading

@daixiang0 can you give us some more context about what problem you're trying to solve? why do you need to run the proxy as root? generally, we want to run the proxy with a UID that we know isn't used by any other processes so that we can identify which packets are coming from the proxy by their UID. Running the proxy as root could be problematic since there may be other processes also running as root.

I want to use device in the proxy which need root and privileged permission.

@stale
Copy link

stale bot commented Jul 13, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Jul 13, 2023
@stale stale bot closed this as completed Jul 28, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 28, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@adleong adleong

@mateiidavid mateiidavid

Labels
bug wontfix
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

iptables: support ignore root traffic daixiang0/linkerd2-proxy-init
3 participants
@adleong @mateiidavid @daixiang0