From 3e9dfe49350b39687c6f08c00c2e91bfe919316d Mon Sep 17 00:00:00 2001 From: John Mertic Date: Sat, 10 Aug 2024 08:48:59 -0400 Subject: [PATCH 1/4] Update README.md Signed-off-by: John Mertic --- README.md | 56 ++++++++++--------------------------------------------- 1 file changed, 10 insertions(+), 46 deletions(-) diff --git a/README.md b/README.md index a381f9a..1565b98 100644 --- a/README.md +++ b/README.md @@ -1,57 +1,21 @@ -# Landscape Tools +# LFX Landscape Tools [![License](https://img.shields.io/github/license/jmertic/landscape-tools)](LICENSE) [![CI](https://github.com/jmertic/landscape-tools/workflows/CI/badge.svg)](https://github.com/jmertic/landscape-tools/actions?query=workflow%3ACI) -[![codecov](https://codecov.io/gh/jmertic/landscape-tools/graph/badge.svg?token=A05TDQO69V)](https://codecov.io/gh/jmertic/landscape-tools) -[![CodeQL](https://github.com/jmertic/landscape-tools/actions/workflows/codeql.yml/badge.svg)](https://github.com/jmertic/landscape-tools/actions/workflows/codeql.yml) +[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=jmertic_lfx-landscape-tools&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=jmertic_lfx-landscape-tools) +[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=jmertic_lfx-landscape-tools&metric=coverage)](https://sonarcloud.io/summary/new_code?id=jmertic_lfx-landscape-tools) -This project contains some tools for making it easier to build and maintain a [landscape](https://github.com/cncf/landscapeapp). +This project contains tools that make building and maintaining a [landscape](https://github.com/cncf/landscapeapp) easier by pulling data from LFX on projects and members. -Current tools are: +It is an evolution of the former [landscape-tools](https://github.com/jmertic/landscape-tools), with a refactor to add the ability to pull projects from LFX and generate text logos when a pure SVG does not exist. Additional differences include: -- [landscapemembers.py](landscapemembers.py) - Creates a new landscape.yml or updates an existing landscape.yml file with members populated. Leverages LF SFDC as the primary data source for members include, and uses Crunchbase and [other landscapes](https://github.com/cncf/landscapeapp/blob/master/landscapes.yml) as secondary data sources for data enrichment. -- [makememberprojectlogos.sh](makememberprojectlogos.sh) - Creates uniform project/product logos for an organization where they don't have a specific logo to use. -- [downloadcrunchbasedata.sh](downloadcrunchbasedata.sh) - Fetches the latest CSV dump of organizations from crunchbase -- [buildimages.sh](buildimages.sh) - Creates images for all the projects of a given company, using the given image as base and adds the project name underneath. +- Only leverage LFX data for members and projects; no longer looks up in CrunchBase or other landscapes. +- Pull review data from a TAC repo using a specific project format, if used. +- More verbose error messages that improve debugging. -## Installation +## Installation and Configuration instructions -```bash -git clone https://github.com/jmertic/landscape-tools -cd landscape-tools -pip install -r requirements.txt -``` - -If you wish to use crunchbase as a data source, add this command. - -```bash -./downloadcrunchbasedata.sh -``` - -## Configuration - -All of the Python scripts depend on a `config.yaml` file being present in the same directory as the script to provide any configuration variables, or passing a `-c` option to the script with a path to the config file. Settings are below. - -```yaml -landscapeName: # short name of your landscape - matches entry at https://github.com/cncf/landscapeapp/blob/master/landscapes.yml -landscapeMemberClasses: # classes of membership; name matches how it's listed in LF SFDC, and category how it will be listed in the landscape. Example below... - - name: Associate Membership - category: Associate - - name: Gold Membership - category: Gold - - name: Platinum Membership - category: Platinum - - name: Silver Membership - category: Silver -project: # project slug -landscapeMemberCategory: # category name of the members section in the landscape.yml file -landscapefile: # filename to use for the outputted landscape.yml file -missingcsvfile: # filename to use for the list of entries with missing parts ( such as a logo, website, or crunchbase entry ) -``` - -### Environment variables - -This depends on `CRUNCHBASE_KEY` being set to a valid key if you wish to use that as a data source ( required by [downloadcrunchbasedata.sh](downloadcrunchbasedata.sh) ). +See [INSTRUCTIONS.md](./INSTRUCTIONS.md). ## Contributing From fddc97ca725193c6683723dd5b575a29ecf8213b Mon Sep 17 00:00:00 2001 From: John Mertic Date: Sun, 11 Aug 2024 08:29:24 -0400 Subject: [PATCH 2/4] Update INSTRUCTIONS.md Signed-off-by: John Mertic --- INSTRUCTIONS.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/INSTRUCTIONS.md b/INSTRUCTIONS.md index d645641..8200dd7 100644 --- a/INSTRUCTIONS.md +++ b/INSTRUCTIONS.md @@ -1,6 +1,16 @@ -# How this all works? +# How does this all work? -The main script [landscapemembers.py](landscapemembers.py) is designed to pull a list of all current members of the given project from LF Members endpoint, using that as the primary source of data for the landscape. Data pulled includes... +There is one main command, `landscape,` which has several different sub-commands depending on what you are looking to pull. + +- `landscape build_members` pulls LFX member data, replacing what is currently in the landscape. +- `landscape build_projects` pulls LFX project data, replacing what is currently in the landscape. +- `landscape sync_projects` pulls LFX project data, adding new entries that are not in the landscape and syncing entries that are. + +There are two primary script paths used to pull data from LFX. + +## `build_members` + +The primary - Member name - Website From 1fd7e5f025d59ea05828111318af277ebcc3607a Mon Sep 17 00:00:00 2001 From: John Mertic Date: Mon, 11 Nov 2024 14:49:58 -0500 Subject: [PATCH 3/4] Update README.md Signed-off-by: John Mertic --- README.md | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 82 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 1565b98..6d03bce 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # LFX Landscape Tools -[![License](https://img.shields.io/github/license/jmertic/landscape-tools)](LICENSE) -[![CI](https://github.com/jmertic/landscape-tools/workflows/CI/badge.svg)](https://github.com/jmertic/landscape-tools/actions?query=workflow%3ACI) +[![License](https://img.shields.io/github/license/jmertic/lfx-landscape-tools)](LICENSE) +[![CI](https://github.com/jmertic/lfx-landscape-tools/workflows/CI/badge.svg)](https://github.com/jmertic/lfx-landscape-tools/actions?query=workflow%3ACI) [![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=jmertic_lfx-landscape-tools&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=jmertic_lfx-landscape-tools) [![Coverage](https://sonarcloud.io/api/project_badges/measure?project=jmertic_lfx-landscape-tools&metric=coverage)](https://sonarcloud.io/summary/new_code?id=jmertic_lfx-landscape-tools) @@ -13,9 +13,87 @@ It is an evolution of the former [landscape-tools](https://github.com/jmertic/la - Pull review data from a TAC repo using a specific project format, if used. - More verbose error messages that improve debugging. -## Installation and Configuration instructions +## Setting up the GitHub Action -See [INSTRUCTIONS.md](./INSTRUCTIONS.md). +1) Add [secrets](https://docs.github.com/en/actions/reference/encrypted-secrets) for `PAT`, which is a [GitHub Personal Authorization Token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) set for the `repo` scope. +2) [Add a new label](https://docs.github.com/en/github/managing-your-work-on-github/managing-labels#creating-a-label) - `automated-build`. This is for this workflow to work and shouldn't be used for anything else. +3) Add the following code to a `build.yml` file in your landscape repo's `.github/workflows/` directory. + +```yaml +name: Build Landscape from LFX + +on: + workflow_dispatch: + schedule: + - cron: "0 4 * * *" + +jobs: + build: + runs-on: ubuntu-24.04 + steps: + - name: Harden Runner + uses: step-security/harden-runner@91182cccc01eb5e619899d80e4e971d6181294a7 # v2.10.1 + with: + egress-policy: audit + - uses: jmertic/lfx-landscape-tools@main + with: + project_processing: skip # see options in action.yml + env: + token: ${{ secrets.PAT }} + repository: ${{ github.repository }} + ref: ${{ github.ref }} +``` +4) Add the following code to a `validate.yml` file in your landscape repo's `.github/workflows/` directory. + +```yaml +name: Validate Landscape + +on: + merge_group: + pull_request: + branches: + - main + - master + +jobs: + validate-landscape: + runs-on: ubuntu-latest + name: "Validate landscape.yml file" + steps: + - name: Harden Runner + uses: step-security/harden-runner@91182cccc01eb5e619899d80e4e971d6181294a7 # v2.10.1 + with: + egress-policy: audit + + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 + - uses: cncf/landscape2-validate-action@7f299c46e9b03b4e8bc2896882734fb0b0756b37 # v2.0.0 + with: + target_kind: data + target_path: ./landscape.yml + - uses: pascalgn/automerge-action@7961b8b5eec56cc088c140b56d864285eabd3f67 # v0.16.4 + if: success() + env: + GITHUB_TOKEN: "${{ secrets.PAT }}" + MERGE_LABELS: "automated-build" + MERGE_RETRY_SLEEP: 300000 + MERGE_METHOD: "squash" +``` + +## Local install + +You can install this tool on your local computer via `pipx` + +```bash +pipx install git+https://github.com/jmertic/lfx-landscape-tools.git +``` + +Similarly, you can use the command below to upgrade your local install. + +```bash +pipx upgrade lfx-landscape-tools +``` + +You can then use the `lfxlandscape` command to run the various commands. Use `lfxlandscape --help` for the options. ## Contributing From 1852c1a72aed1b44723e7219dcf87ce1d50dcc1c Mon Sep 17 00:00:00 2001 From: John Mertic Date: Mon, 11 Nov 2024 14:51:14 -0500 Subject: [PATCH 4/4] Delete INSTRUCTIONS.md Signed-off-by: John Mertic --- INSTRUCTIONS.md | 202 ------------------------------------------------ 1 file changed, 202 deletions(-) delete mode 100644 INSTRUCTIONS.md diff --git a/INSTRUCTIONS.md b/INSTRUCTIONS.md deleted file mode 100644 index 8200dd7..0000000 --- a/INSTRUCTIONS.md +++ /dev/null @@ -1,202 +0,0 @@ -# How does this all work? - -There is one main command, `landscape,` which has several different sub-commands depending on what you are looking to pull. - -- `landscape build_members` pulls LFX member data, replacing what is currently in the landscape. -- `landscape build_projects` pulls LFX project data, replacing what is currently in the landscape. -- `landscape sync_projects` pulls LFX project data, adding new entries that are not in the landscape and syncing entries that are. - -There are two primary script paths used to pull data from LFX. - -## `build_members` - -The primary - -- Member name -- Website -- Crunchbase URL -- Twitter handle -- Logo URL - -The pull will look for active members based on the Purchase History, and respect the 'Don't show logo on website' item if selected. - -With the data for things like logos and crunchbase entries, there is often more accurate data in other [landscapes](https://landscapes.dev). The script will attempt to look for this, and overlay data from other landscapes if the data from the LF Members endpoint is empty. This also pulls in useful data such as `stock_ticker`, which often has to be set to `null` as the Crunchbase data is inaccurate. - -# How to use - -You can use this script in a few different ways, but generally one of the below works best - -## Recommended - Automatic build with GitHub Actions - -Easiest option to do a build is leveraging GitHub Actions, which requires nothing for you to setup on your local machine. You can have it run on demand, or set the `cron` option in the `update_members.yml` action to have it run on a schedule ( this is how it's done for the LF Landscape, which runs nightly at 9:00pm-ish EST ). - -If it's all setup, goto the landscape repo under [Actions](actions). You should see a job called 'Update members' on the right side under 'Workflows'. Click that, then on the next screen click 'Run workflow' - -There is some prerequiste setup in GitHub. - -1) Add [secrets](https://docs.github.com/en/actions/reference/encrypted-secrets) for `PAT`, which is a [GitHub Personal Authorization Token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) set for the `repo` scope. -2) [Add two new labels](https://docs.github.com/en/github/managing-your-work-on-github/managing-labels#creating-a-label) - `automerge` and `automated-build`. These are for this workflow to all work and shouldn't be used for anything else. - -Actions are stored in the `.github/workflows` directory. There are three to create if you want to do the entire workflow, including having it autocommit if the build all works successfully. - -First one is `update_members.yml` - -```yaml -name: Update members - -on: - workflow_dispatch: - # you can add a cron option here as well if you want it fully hands off - -jobs: - build: - runs-on: ubuntu-latest - - steps: - - name: Checkout Landscape - uses: actions/checkout@v4 - with: - path: landscape - - name: Checkout landscape-tools - uses: actions/checkout@v4 - with: - repository: jmertic/landscape-tools - path: landscape-tools - - name: Set up Python 3.x - uses: actions/setup-python@v5 - with: - python-version: '3.x' - - name: Install dependencies - run: | - pip install --no-deps --require-hashes -r landscape-tools/requirements.txt - - name: Run build - working-directory: ./landscape - run: | - ../landscape-tools/landscapemembers.py - - name: Save missing.csv file - uses: actions/upload-artifact@v4 - with: - name: missing-members - path: ./landscape/missing.csv - - name: Checkout landscapeapp - uses: actions/checkout@v4 - with: - repository: cncf/landscapeapp - path: landscapeapp - - name: Setup node - uses: actions/setup-node@v4 - with: - node-version: '18' - - name: Cleanup YAML files - working-directory: ./landscapeapp - run: | - node tools/removePuppeteer - npm install - PROJECT_PATH=../landscape node tools/removeQuotes - PROJECT_PATH=../landscape node tools/pruneExtraEntries - - name: Create Pull Request - uses: peter-evans/create-pull-request@v6 - with: - token: ${{ secrets.PAT }} - branch-suffix: timestamp - path: ./landscape - title: Update members - labels: automated-build - commit-message: Update members -``` - -Next is `marksuccessfulbuild.yml`, which looks for when the Netlify preview build is done and labels the pull request to be merged. Change the string `omp-landscape` for the name of your landscape in Netlify - -```yaml -name: "Set Issue Label on successful build" -on: - workflow_dispatch: - issue_comment: - types: [created] - -jobs: - test: - runs-on: ubuntu-latest - steps: - - uses: Amwam/issue-comment-action@v1.3.1 - if: ${{ github.event.issue.pull_request && contains(github.event.issue.labels.*.name, 'automated-build') }} - with: - keywords: '["Deploy preview for *omp-landscape* ready"]' - labels: '["automerge"]' - github-token: "${{ secrets.PAT }}" -``` - -Finally, `automerge.yaml` will merge the pull request moments after the issue us labeled `automerge` - -```yaml -name: Autocommit pull requests - -on: - workflow_dispatch: - pull_request: - types: [labeled] - -jobs: - build: - runs-on: ubuntu-latest - - steps: - - name: Automerge Pull Request if possible - uses: "pascalgn/automerge-action@v0.15.5" - env: - GITHUB_TOKEN: "${{ secrets.PAT }}" - MERGE_LABELS: "automerge" - MERGE_RETRY_SLEEP: 300000 - MERGE_METHOD: "squash" -``` - -## Optional - Running a full build locally - -To run things locally, you'll want to have both this repo and the given landscape repo checked out. The following steps should do that... - -```zsh -git clone https://github.com/jmertic/landscape-tools.git -git clone [landscape repo url] landscape # if you have multiple landscapes pick a new name here ;-) -git clone https://github.com/cncf/landscapeapp.git -cd landscapeapp -npm install -g yarn@latest -yarn -cd ../landscape-tools -chmod +x *.py -pip install -r requirements.txt -./downloadcrunchbasedata.sh -``` - -You'll want to add the [API keys](https://github.com/cncf/landscapeapp#api-keys) and [.bash_profile or .zshrc](https://github.com/cncf/landscapeapp#installing-locally) bits here as well. - -You will also want to create the `config.yaml` file in the `landscape` directory with the [proper config settings for your landscape](https://github.com/jmertic/landscape-tools#configuration) - -Now to run a full build, do the following... - -```zsh -cd landscape # or whatever the directory name you chose above for the landscape clone itself -../landscape-tools/landscapemembers.py -yf -yo -``` - -`../landscape-tools/landscapemembers.py` will take about a minute, and will update the `landscape.yml` file in your local clone, as well as adding the logos under `hosted_logos/`. - -`yf` likely will take a few minutes on the first run, but should go faster after that. This will update `processed_landscape.yml` - -`yo` will run and you can preview the landscape. Once you are happy with thing, commit the changes and push to Github. - -```zsh -git add . -git commit -m 'Updated member logos' # or whatever you want to set here -git push -``` - -# Troubleshooting errors - -There are a few common issues that pop up that require intervention. - -- Bad logo, either the 'SVG embeds a PNG' error or 'Convert text to glyph'. Tips on how to resolve these errors and get a proper logo are at https://github.com/cncf/landscapeapp#logos, and then refer to the internal instructions for updating logos. -- Bad crunchbase entry, for this just add the right one in SF for the member under 'Crunchbase URL' -- `Build failed because of: no headquarter addresses for xxx`, here you need to update the CrunchBase entry to add a location ( details at https://support.crunchbase.com/hc/en-us/articles/360019601394-Updating-a-Company-Profile ) -- `No cached entry, and can not fetch: xxx. Can't resolve stock ticker XXXX; please manually add a "stock_ticker" key to landscape.yml or set to null`. Usually this means either the stock ticker is set wrong ( for example, the company is merged with another and it changed ) or it's pointing to some foreign stock market that the landscape doesn't know how to handle. In either case the easiest thing to do is set `stock_ticker` to `null`.