ci(google-auth): use workload identity federation instead of json

[cpcloud]

No description provided.

[tswast]

Love it! Still working on getting approvals to enable this in our ibis-gbq test project.

[cpcloud]

@tswast Should we just give this a go and merge it?

[tswast]

Sorry, I misspoke at PyData NYC. Looks like I'm still waiting for approval on one more internal change (issue 375475414 for Googlers' reference) before this can work.

[cpcloud]

@tswast Any updates here?

[tswast]

Thanks for the reminder. I'm still waiting on a thumbs up from our security team. I think I went through the right process, but it's been stuck on the last LGTM for a couple months now. 😭 I'll ping them again today. Maybe I can get another reviewer assigned?

[tswast]

Approving, but consider resolving the comments on .github/workflows/ci-data.yml before merging.

[cpcloud]

@tswast Thanks for pushing this through, merging on green!

[cpcloud]

@tswast It looks like two tests are failing on main (which is still a win since everything else is passing 🥳) see here: https://github.com/ibis-project/ibis/actions/runs/12507938156/job/34896929010

Can you help debug the permissions here?

[cpcloud]

Looks like step 5 here might be required (giving permissions to specific cloud resources).

[tswast]

Thanks for the heads up. Taking a look now. Step 5 does have a concerning warning, which I wonder if applies in this case?

However, not all Google Cloud resources support principalSet identities

For example, the failure to read gs://ibis-testing-libraries/lodash.min.js from BigQuery is concerning, as I have granted the GitHub provider access to that bucket in two different ways. I'm not seeing any notes about principalSet in the BigQuery + GCS or BigQuery + JS libraries docs. I'll do some experimentation and see if I can narrow down what's going on.

[cpcloud]

Thanks. If it's helpful, writing data to a bucket (the benchmark data upload) is working, so accessing a bucket directly is not a problem.

Just to clarify, from my POV I see two things that might be an issue:

1. Reading a bucket from within BigQuery (perhaps specific to JS libraries)
2. Accessing another project's data, even if it's publicly accessible, like the one we're testing against.

[cpcloud]

If you can't find anything in the next few business days, we can xfail those tests until we understand whether there's an issue.

[tswast]

I tried out GitHub Actions on another repo (tswast/code-snippets#3) to see if I could reproduce the BigQuery + JS library failure, but I was able to call bqutil.fn.xml_to_json in https://github.com/tswast/code-snippets/actions/runs/12521512469/job/34928670087

I suspect it might be a permission issue rather than a BigQuery bug with such credentials. I'll try to take a look at the ibis setup a bit more next week.