Release flow improvements (#597)

* Some improvements to the release flow after starkiller submodule removal

* use changelog-path

* remove todos

* combine sponsors changelog for release notes

* update docs

* update readme

* add commit to sponsors changelog

.github/actions/update-starkiller/action.yml

@@ -1,25 +1,21 @@
 name: 'Update Starkiller'
-description: 'Updates the config.yaml and starkiller submodule to the specified version and commits the change.'
+description: 'Updates the config.yaml to the specified version and commits the change.'
 inputs:
   starkiller-version:
-    description: 'The branch to merge from'
+    description: 'The starkiller ref to use'
     required: true
   app-version:
     description: 'The changelog section to update'
     required: true
   repo:
     description: "The starkiller repo to use. If not set, doesn't update the field."
     required: false
+  changelog-path:
+    description: "The changelog path"
+    default: CHANGELOG.md
 runs:
   using: "composite"
   steps:
-    - name: Install Python
-      uses: actions/setup-python@v2
-      with:
-        python-version: '3.9'
-    - name: Install minimal python packages
-      shell: bash
-      run: python -m pip install pyyaml
     # This is technically only needed for the first time we update the public repo, as a bootstrap.
     # After that, the submodule is already initialized to the right repo.
     - name: Update config.yaml repo
@@ -29,28 +25,15 @@ runs:
         sed -i "s|repo:.*|repo: ${{ inputs.repo }}|" empire/server/config.yaml
     - name: Update config.yaml ref
       shell: bash
-      # this depends on there not being any other keys in teh config named ref
+      # this depends on there not being any other keys in the config named ref
       # if there are, this will break. We could be safer to load/dump the yaml,
       # but would lose the comments.
       run: |
         sed -i "s/ref: .*/ref: ${{ inputs.starkiller-version }}/" empire/server/config.yaml
-    # If use_temp_dir is true, Starkiller is cloned into a temp directory, the CI will fail,
-    # and the submodule will not be updated. So set it to false, make the changes, then set it back.
-    - name: Update config.yaml use_temp_dir
-      shell: bash
-      run: |
-        sed -i'.bak' "s/use_temp_dir: .*/use_temp_dir: false/" empire/server/config.yaml
-    - name: Run starkiller update script
-      shell: bash
-      run: python empire.py sync-starkiller
-    - name: Reset use_tmp_dir
-      shell: bash
-      run: |
-        mv empire/server/config.yaml.bak empire/server/config.yaml
     - name: Update changelog
       shell: bash
       run: |
-        sed -i '/## \[${{ inputs.app-version }}\]/a -   Updated Starkiller to ${{ inputs.starkiller-version }}' CHANGELOG.md
+        sed -i '/## \[${{ inputs.app-version }}\]/a -   Updated Starkiller to ${{ inputs.starkiller-version }}' ${{ inputs.changelog-path }}
     - name: Commit changes
       shell: bash
       run: |

.github/ci-and-release.md

@@ -121,23 +121,24 @@ the `private-main` branch.
 Once the `release/` pull request is merged, the `Private - Tag Release` workflow will automatically run.
 The workflow will create a tag and release on the `HEAD` of `private-main` using the release notes from `CHANGELOG.md` for the body of the release.
 
-### 6. Start Sponsor/Kali Release
+### 6. Repeat Step 2 - Prerelease Merge
+Repeat step 2 to merge `private-main` into `sponsors-main` and `kali-main`.
+
+### 7. Start Sponsor/Kali Release
 Start the release by running the `Sponsors & Kali - Create Release` manual workflow.
 If starkiller needs to be updated, provide a `starkillerVersion` input. The value provided should be a git tag minus the `-kali` or `-sponsors` suffix.
 
-This will first attempt to merge the `private-main` branch into `sponsors-main` and `kali-main` with the new release changes. Most likely, if there is a merge conflict here it is caused by `CHANGELOG.md` and should be minor. If that occurs, the merge conflict can be resolved in the pull request via the GitHub editor, or locally. 
-
-If a Starkiller tag was provided, it will update the Starkiller submodule and the changelog accordingly. It does this on the `sponsors-main` and `kali-main` release branches separately.
+If a Starkiller tag was provided, it will update the Starkiller config and the changelog accordingly.
 
 A release PR will then be opened for each branch and the test suite will run.
 
 
-#### 7. Manual Step - Merge sponsor/kali release PRs
+#### 8. Manual Step - Merge sponsor/kali release PRs
 Once the workflow runs, it will open two pull requests from the `release/v{version}-sponsors` and `release/v{version}-kali` branches to `sponsors-main` and `kali-main` respectively.
 
 Check the changelog on these branches, this will be the changelog that is used for the release notes.
 
-If there are sponsor/kali specific changelog entries that need to be added, add them to the `CHANGELOG.md` file on the release branch.
+If there are sponsor specific changelog entries that need to be added, add them to the `CHANGELOG-SPONSORS.md` file on the release branch.
 
 You can get a list of the new commits that are in this release by using the following command. Replace `v4.9.0-sponsors` with whatever the previous release was.
 ```
@@ -151,7 +152,7 @@ the `sponsors-main` branch or `kali-main` branch.
 
 **Potential Enhancement** We could add automation that copies the `unreleased` section from the target branch to the version section in the `head` branch.
 
-### 7. Tag and Release
+### 9. Tag and Release
 Once the pull requests are merged, the `Sponsors - Tag Release` and `Kali - Tag Release` workflows will automatically run.
 The workflows will create a tag and release on the `HEAD` of `sponsors-main` and `kali-main`, using the release notes from `CHANGELOG.md` for the body of the release.
 

.github/workflows/release-sponsor-kali-start.yml

@@ -19,7 +19,7 @@ jobs:
       - name: Check out code
         uses: actions/checkout@v3
         with:
-          ref: private-main
+          ref: sponsors-main
           submodules: 'recursive'
           fetch-depth: 0
           token: ${{ secrets.RELEASE_TOKEN }}
@@ -37,61 +37,22 @@ jobs:
         with:
           starkiller-version: ${{ env.STARKILLER_TAG }}
           app-version: ${{ env.APP_VERSION }}
-      - name: Get release branch name
-        run: |
-            echo "RELEASE_BRANCH=release/$APP_VERSION-sponsors" >> $GITHUB_ENV
-      - name: Create release branch
-        run: git checkout -b ${{ env.RELEASE_BRANCH }}
-      - name: Push new branch
-        run: git push origin ${{ env.RELEASE_BRANCH }}
-      - name: Create pull request into ${{ env.TARGET_BRANCH }}
-        uses: thomaseizinger/[email protected]
+          changelog-path: CHANGELOG-SPONSORS.md
+      - name: Update Sponsors Changelog
+        uses: vinnybod/[email protected]
         with:
-          GITHUB_TOKEN: ${{ secrets.RELEASE_TOKEN }}
-          head: ${{ env.RELEASE_BRANCH }}
-          base: ${{ env.TARGET_BRANCH }}
-          title: v${{ env.APP_VERSION }} into ${{ env.TARGET_BRANCH }}
-          reviewers: ${{ github.event.issue.user.login }}
-          body: |
-            This PR was automatically generated by the `release-sponsor-kali-start` workflow.
-            I've updated the version name and code commit: ${{ steps.make-commit.outputs.commit }}.
-            This PR should be merged with a merge commit, **not a squash commit.**
-            Merging this PR will trigger a tag and release automatically.
-  # this is basically copy/paste from the above.
-  create_kali:
-    # disable for now.
-    if: false
-#    if: ${{ github.repository == 'BC-Security/Empire-Sponsors' }}
-    runs-on: ubuntu-latest
-    steps:
-      - name: Set env vars
+          tag: v${{ env.APP_VERSION }}
+          changelogPath: CHANGELOG-SPONSORS.md
+          skipDiff: false
+      - name: Commit changelog and manifest files
+        id: make-commit
         run: |
-          echo "TARGET_BRANCH=kali-main" >> $GITHUB_ENV
-          echo "STARKILLER_TAG=${{ github.event.inputs.starkillerVersion }}-kali" >> $GITHUB_ENV
-      - name: Check out code
-        uses: actions/checkout@v3
-        with:
-          ref: private-main
-          submodules: 'recursive'
-          fetch-depth: 0
-          token: ${{ secrets.RELEASE_TOKEN }}
-      - name: Initialize mandatory git config
-        run: |
-          git config user.name "GitHub Actions"
-          git config user.email [email protected]
-      - name: Get app version
-        run: |
-          # Get the version from pyproject.toml using sed
-          echo "APP_VERSION=$(sed -n 's/^version = "\(.*\)"/\1/p' pyproject.toml)" >> $GITHUB_ENV
-      - name: Update Starkiller Version
-        if: ${{ github.event.inputs.starkillerVersion }}
-        uses: ./.github/actions/update-starkiller
-        with:
-          starkiller-version: ${{ env.STARKILLER_TAG }}
-          app-version: ${{ env.APP_VERSION }}
+          git add -A
+          git commit --message "Prepare release ${{ env.APP_VERSION }} sponsors"
+          echo "commit=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT
       - name: Get release branch name
         run: |
-            echo "RELEASE_BRANCH=release/$APP_VERSION-kali" >> $GITHUB_ENV
+            echo "RELEASE_BRANCH=release/$APP_VERSION-sponsors" >> $GITHUB_ENV
       - name: Create release branch
         run: git checkout -b ${{ env.RELEASE_BRANCH }}
       - name: Push new branch

.github/workflows/release-sponsor-kali-tag.yml

@@ -5,7 +5,6 @@ on:
       - closed
     branches:
       - sponsors-main
-#      - kali-main
 
 jobs:
   tag_release:
@@ -27,6 +26,11 @@ jobs:
       - name: Extract release notes
         id: extract_release_notes
         uses: vinnybod/extract-release-notes@v2
+      - name: Extract release notes - sponsors
+        id: extract_release_notes_sponsors
+        uses: vinnybod/extract-release-notes@v2
+        with:
+          changelog_file: CHANGELOG-SPONSORS.md
       - name: Create Release
         id: create_release
         uses: actions/create-release@v1
@@ -35,6 +39,10 @@ jobs:
         with:
           tag_name: v${{ env.TAG_NAME }}
           release_name: v${{ env.TAG_NAME }}
-          body: ${{ steps.extract_release_notes.outputs.release_notes }}
+          body: |
+            ${{ steps.extract_release_notes.outputs.release_notes }}
+            
+            ### Sponsors Only
+            ${{ steps.extract_release_notes_sponsors.outputs.release_notes }}
           draft: false
           prerelease: false

CHANGELOG.md

@@ -9,7 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 -   Remove Starkiller as a submodule, treat it as a normal directory (@Vinnybod)
     - Everything should 'just work', but if you have issues after pulling these latest changes, try deleting the Starkiller directory before running the server `rm -r empire/server/api/v2/starkiller`. 
-
+-   Some improvements to the release flow after starkiller submodule removal (@Vinnybod)
+-   
 ## [5.3.0] - 2023-05-17
 
 -   Add the ability to specify a module option as a file (@Vinnybod)