Introduce OpenAI action

[AliSoftware]

What does it do?

Introduce an openai_generate openai_ask action to allow some CI automation tasks in particular to generate release notes blurbs for new versions.

This came up as part of internal feedback from p91TBi-cBb-p2:

There is a note that we do manually thing which can be replaced with an API call

Create a PR to update fastlane/metadata/android/en-US/changelogs/default.txt, fastlane/metadata/wear/en-US/changelogs/default.txt (if applicable) and CHANGELOG.md in the release/%version% branch, providing a more user-friendly copy for the release notes (see details)
Feel free to use ChatGPT and the following prompt to generate nice user-facing notes from the bullet points:

The main goal of this PR is thus to introduce an action that could take the content of the CHANGELOG.md / RELEASE-NOTES.txt file of one of our mobile apps repos, call this action, and use the result to generate a PR with the response saved into the proper changelogs/default.txt file(s).

items = extract_release_notes_for_version(version: app_version, release_notes_file_path: 'RELEASE-NOTES.txt')
nice_changelog = openai_ask(
  prompt: :release_notes, # Uses the pre-crafted prompt for App Store / Play Store release notes
  question: "Help me write release notes for the following items:\n#{items}",
  api_token: get_required_env('OPENAI_API_TOKEN')
)
File.write(File.join('fastlane', 'metadata', 'android', en-US', 'changelogs', 'default.txt'), nice_changelog)
# Then open a PR to have that change reviewed by the Release Manager (and tweaked if needed) and merged

Status: Proof-of-Concept

TBH this was just a proof of concept I wanted to quickly test today to answer the Release Managers' feedback on WCAndroid from their last rotations and validate if this was easily possible. Now that I've seen this is possible, I'll consider creating a Project Thread at some point to start discussing about it and plan on adopting it for apps that might be interested.

For the case of WooCommerce this will still require to split the items starting with [WEAR] from the rest and make 2 different calls to store the 2 separate blurbs in the two different changelogs/default.txt files, so there'll probably still be a bit of tweaking before we actually start to use that action. Probably similar for PocketCasts which also have mobile+automotive+wear apps. For other mobile apps which only have mobile and for which there'd be no need to split the entries based on app variant, this should be as straightforward as the self.example provided in the action's code though 🙂

To Test

I've asked access to the a8c org in OpenAI (ref: p1732898401785999-slack-C06CJ3U621X) and have generated an API token for us to experiment with this action (stored in SecretStore, ?secret_id=12591).

You can thus experiment with pointing a repo to this branch of the release-toolkit (or even hack this repo's Fastfile directly) then add a call to this openai_ask method, using the api_token: found in the Secret Store, and validate it gives nice single <650 character paragraph as a response for prompts using :release_notes

This is how the response stubs used in the unit tests have been generated btw.

Checklist before requesting a review

• Run bundle exec rubocop to test for code style violations and recommendations
• Add Unit Tests (aka specs/*_spec.rb) if applicable
• Run bundle exec rspec to run the whole test suite and ensure all your tests pass
• Make sure you added an entry in the CHANGELOG.md file to describe your changes under the appropriate existing ### subsection of the existing ## Trunk section.
• If applicable, add an entry in the MIGRATION.md file to describe how the changes will affect the migration from the previous major version and what the clients will need to change and consider.

[iangmaia]

💭 A similar idea could work with a Dangermattic plugin to auto-review PRs. Of course it would take some fine tuning and it would be nice to clearly separate this generated feedback from the rest of the comments, but perhaps we could get to something useful.

[AliSoftware]

💭 A similar idea could work with a Dangermattic plugin to auto-review PRs. Of course it would take some fine tuning and it would be nice to clearly separate this generated feedback from the rest of the comments, but perhaps we could get to something useful.

I'm not sure I follow your idea with Dangermattic auto-reviewing PRs in this context…

1. Was you idea to have Dangermattic auto-approve those PRs that are auto-generated to create the release notes from OpenAI, so that we bring the automation of that step even further?
2. Or was your idea to have Dangermattic use the OpenAI API to comment on some specific code changes of various PRs with OpenAI responses?

If that was about (1), I'd actually vote against it.

• I was already a bit reluctant to have the release notes automated by OpenAI/ChatGPT transparently via a ReleasesV2 task because that would mean that those would not be proof-read by a human. I liked the fact that this step was manual so far because it allowed a human (the Release Manager) to ensure ChatGPT didn't generate wrong/misleading/etc release notes or use unfortunate phrasing that would require tweaking (to account for our app's specific vocabulary of a feature, etc).
• But then I realized that it would still be proof-read due to the fact that we would do a PR to merge those, so suddenly I was back to being OK to use OpenAI to generate those blurbs, because we'd have the PR review process to give the opportunity for a human to proof-read and review those… which is the whole point of a PR
• But if we decide to also automate the review of the PR via Dangermattic, that kinda defeats that last point

If that was (2), I'm not sure what type of comments you had in mind; but we might also want to be mindful of our OpenAPI usage and the billing that goes with it, having it used on every single run of Danger (run 1 or more times for each PR) might become a bit much. And we might also be mindful of what we send to OpenAI and its model (e.g. code and context from private repos…)

[AliSoftware]

Alternative naming idea:

• Name the action openai_prompt(context_prompt:, message:) or something similar, to highlight that this is focused on text prompts (as opposed to "generating" as in Dall-E and generating images or similar). The thing is, prompt here would be used as a verb, while it could be confusing for the name of the action if someone were interpreting it as a noun instead… (is it supposed to return the prompt?!)
• Name the action openai_ask(prompt:, question:) instead of openai_generate
• Use message: instead of question:…

I think I like openai_ask(prompt:, question:) after all, but open to brainstorming and bikeshedding 😅

[iangmaia]

Hmm I'd also vote for openai_ask(prompt:, question:)! IMO it sounds more natural and better reflects the "conversation" aspect of the API.

[AliSoftware]

Sold! Action renamed in c829b8d

[iangmaia]

2. Or was your idea to have Dangermattic use the OpenAI API to comment on some specific code changes of various PRs with OpenAI responses?

More like this -- sorry I wasn't very specific and just thinking out loud, but I wasn't thinking in implementation details (e.g. how often to run it to save on OpenAI API calls) and more on the concept (and perhaps as a potential experiment) of having a LLM review a diff, PR description, title and so on and post back its comments.

[iangmaia]

Giving the option of using predefined prompts as symbols is a neat solution 👍

[iangmaia]

Looking great 👍
Just posted a couple of small non-blocking comments.

[AliSoftware]

FYI: I forgot to update the CHANGELOG before merging this PR, but I've fixed it directly in trunk afterwards, via 1a58ba0