-
Notifications
You must be signed in to change notification settings - Fork 160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
400 Error when logging in from AzureAd with OIDC #3783
Comments
Do you know how many groups you would expect to see? It might be that the cookie we're setting is too large. |
For my user I think it is about 8 groups (I'm not an AD admin at my org so I can't check directly). But for some other users it could be up to 20 groups I suppose (just guessing based on my own). |
Are you authenticating your kubectl clients to Kubernetes successfully? I can see this: Which strongly suggests that the browser is dropping the cookie for some reason, I have seen this when it was "too large" for the browser, i.e. around 4096bytes, but based on "8 groups" this feels unlikely. |
So, based on the error I added in the context section, I believe the authentication is happening successfully somewhere, because when I specify the |
I've pushed a change to log out the sizes of the tokens as we're setting them.
There seem to be two possible issues, and my sense is that the first |
Cool! Helm controller should pick up the latest image tag from the oci repository and deploy it automatically right? Or do I need to specify a specific image tag on my helm repo now for testing? |
Sorry, it'll need to get reviewed and then merged and released. Might want to track #3785 |
@bigkevmcd i notice that the PR got merged yesterday but my pod is still running the previous release from a few days ago
even after deleting and recreating the pod. The helm repo looks like it didnt pick up a change yet
am I waiting for something else to happen first? |
We haven't yet "released" the change, it's merged but not yet released, you can however build an image from the source if you want to. We are organising a release which would include this fix. |
Hey hey, any ETA on that release yet? I found something which might be relavent today, after clicking around a bit in the UI and then checking the logs, in-amongst the cookie errors, i also got a
|
The logging went out in yesterday's release https://github.com/weaveworks/weave-gitops/releases/tag/v0.26.0 If it turns out to be the size of the cookie, then we can prioritise a fix, if not, I'm not entirely sure what's going on, except for trying replicate it. |
So it did! i actually didnt notice the new version on the bottom of my UI. So currently it's running on version 0.26.0. So my understanding is that it should now be logging the size of the cookie somewhere for me to check, is that correct? |
OK so it actually looks like things are working as expected? The missing cookie error seems to have dissapeared however i still dont see any data in the dashboard when i log in with OIDC:
Only thing that confuses me is the token length says 0? |
You should be seeing https://github.com/weaveworks/weave-gitops/pull/3785/files The thinking there is that we log out as we send it, and if the browser drops it, we'll be able to tell? Those are parsing it, perhaps earlier in the logs when you're authing that will be logged? |
Yep, just found it. And also, i found some other errors which seemed to go away magically.. originally it seemed like it was looking for the wrong provider and then all of a sudden it kicked into gear and worked... Logs
|
I wonder if these log entries are one of the issues here coreos/go-oidc#344
Gives us something to go on, at least we can rule out the size of the token! |
Ok cool, so i'll leave this with you guys for now then? I guess it really is a possibility that AzureAD just isnt supported right now. In the mean-time we will figure out an alternative auth method to front the dashboard with. Is it possible to disable all auth? So you don't even see the login screen? So that we can put our own auth method in front of the URL instead? |
Not currently, please feel free to open an issue. Support for something like https://github.com/oauth2-proxy/oauth2-proxy doesn't feel unreasonable. |
Another one having the same issue with Azure.
|
@angelbarrera92 thanks, that looks like the cookie is too big, it likely includes all the groups that you're a member of. We need to shift to a session-store based approach for storing cookies, which is non-trivial, but it would mean that we'd issue shorter cookies, and store the details in-browser, it's not currently scheduled, but this is an indication that we need to bump it up the priority list. |
Hi @bigkevmcd I can confirm we are having the same issue and the cookie is definitely too big. From the devtools in MS Edge:
logs from our deployment:
We didn't see this on "version": "v0.22.0". Not sure if something changed with the app or the token was just smaller at that stage. Using Azure AD with these extra secret configs:
|
Hi @adamshawvipps ahh...ok, I can't think of anything that would've changed in between v0.22.0 other than we added debug logging to try and find out if this was the issue. We'll discuss it internally, and see what we can do to prioritise a solution to this.
Thanks for providing the data for this. |
@bigkevmcd thanks for looking into it |
We are seeing the exact same issue (size between 5k and 10k) |
We have a fix that's being tested just now before release that will hopefully get this solved. |
@oarset @angelbarrera92 @m477r1x We have released a change that should solve this. https://github.com/weaveworks/weave-gitops/releases/tag/v0.31.0 Please feel free to reopen this issue if you are still experiencing the problem. We are looking to add support for external session storage, so if this is something that you want support for, please feel free to open an issue. |
Quick test, it works :) I'll keep you posted! |
Describe the bug
This might be related (or not) to issue #2955. We have configured OIDC auth from AzureAD for use with the gitops dashboard, and are receiving the following error on the pod when logging in with OIDC:
Environment
v0.25.0
To Reproduce
Steps to reproduce the behavior:
oidc-auth
and populate with OIDC connection details which match what are configured on the OIDC app in AzureADExpected behavior
Logs in with OIDC and sucessfully impersonate the weave-gitops-admin-cluster-role to see resources
Actual Behavior
No resouerces are shown in the UI and the pod logs the error in the above description
Additional Context (screenshots, logs, etc)
I also tried setting the
.impersonationResourceNames
to include the exact user I was logging in with but that lead to a different error:This error kept happening for every group that the user is a member of (ideally we do not want to have to create bindings in kubernetes fvor all groups because AD group memberships can change frequently and you don't want to have to keep updating the dashboard manifests every time).
The text was updated successfully, but these errors were encountered: