feat: add GitLab build pipelines (#12)

Co-authored-by: simoscarano <[email protected]>

README.md

@@ -1 +1,22 @@
-# Public Pipelines
+# Mia-Platform Public Pipelines
+
+![GitLab CI/CD Supported]  
+![Node.js with NPM Supported] ![Node.js with Yarn Supported] ![Go Supported]
+
+The goals of this repository is to provide to user reusable pipelines for different languages for building libraries
+and Docker images to use in their projects and follow the best practicies for setting up comprehensive pipelines
+for building a project, assessing its security, and keep track of its dependencies. You can see this as a starting
+step for your Platform Engeneering team for setting up a comprehensive suite of tools for their pipelines that can be
+easy to use, and meet regulatory compliance for your company.
+
+## How to Contribute
+
+You can contribute to this repository in adding support for new languages or improve the default steps for the
+exsisting ones, improve the documentation, improve the Docker images and so on. We welcome everybody!
+
+You can find more reading the [documentation](./docs/10_starting-guide.md).
+
+[Gitlab CI/CD Supported]: https://img.shields.io/badge/GitLab-CI%2FCD-orange?logo=gitlab&style=for-the-badge
+[Node.js with NPM Supported]: https://img.shields.io/badge/18%2C20-black?style=for-the-badge&logo=npm&label=NPM%20Node.js&color=3C873A
+[Node.js with Yarn Supported]: https://img.shields.io/badge/18%2C20-black?style=for-the-badge&logo=yarn&label=Yarn%20Node.js&color=3C873A
+[Go Supported]: https://img.shields.io/badge/1.20%2C1.21-black?style=for-the-badge&logo=go&label=Go&color=007E9A

bin/gitlab/config.json

@@ -0,0 +1,6 @@
+{
+	"instance_url": "https://gitlab.com",
+	"group_id": "",
+	"pipeline_images_base_name": "ghrc.io/mia-platform",
+	"default_visibility": "internal"
+}

bin/gitlab/install

@@ -0,0 +1,151 @@
+#! /usr/bin/env bash
+
+set -o pipefail
+set -o errexit
+
+## This script will setup a repository in your chosen GitLab group or in a predefined root groups.
+## Then all the public pipelines yaml files will be commited in the newly created project and a pipeline
+## will be set to automatically open merge requests with eventual updates.
+
+## The needed values from your self-maanged instance or GitLab SaaS are the following:
+## - a valid token that has the 'api' and 'write_repository' scopes on the target GitLab instance
+## - the username of the user that has created the token
+## - a different path where to find the configuration file for the script
+
+# local configuration
+readonly temp_project_folder_path=/tmp/tmp-pipelines
+basedir=$(dirname "$0")
+
+readonly basedir
+
+# parameters read from the user
+TOKEN=""
+USERNAME=""
+CONFIG_PATH="${basedir}/config.json"
+
+# original values that the final user can change
+readonly ORIGINAL_CONTAINER_PATH="ghcr.io/mia-platform"
+readonly ORIGINAL_INCLUDE_PATH="mia-platform/pipeline-templates"
+readonly DEFAULT_GROUP_NAME=""
+
+function print_usage() {
+	cat <<EOF
+Usage: $0 --token token
+
+Options:
+	-u, --username: the username of the user associated with the access token
+	-t, --token:    the access token for the user that has api and write_repository scopes
+	-c, --config:   the path for the configuration file, default to directory of the script file
+EOF
+}
+
+### Check for variables and provide usage if nothing is present
+if [[ $# -eq 0 ]]; then
+	echo "Missing flags. Please see usage below."
+	print_usage
+	exit 1
+fi
+
+while [[ -n "$1" ]]; do
+	case $1 in
+	-u | --username)
+		shift
+		readonly USERNAME=$1
+	;;
+	-t | --token)
+		shift
+		readonly TOKEN=$1
+	;;
+	-c | --config)
+		shift
+		readonly CONFIG_PATH=$1
+	;;
+	-h | --help)
+		print_usage
+		exit 0
+	;;
+	*)
+		print_usage
+		exit 1
+	;;
+	esac
+	shift
+done
+
+# validate user inpunts
+if [[ -z "${TOKEN}" ]]; then
+	echo "You must provide a token."
+	echo "Try '$0 --help' for more information."
+	exit 1
+elif [[ -z "${USERNAME}" ]]; then
+	echo "You must provide a username."
+	echo "Try '$0 --help' for more information."
+	exit 1
+fi
+config=$(cat "${CONFIG_PATH}")
+
+# create a temporary folder, ensure that it will be empty, and copy all the needed files into it
+echo "Creating temp project locally..."
+rm -fr "${temp_project_folder_path}"
+mkdir -p "${temp_project_folder_path}"
+
+cp -rT "${basedir}/../../gitlab-ci/base/" "${temp_project_folder_path}/"
+cp -r "${basedir}/../../gitlab-ci/marketplace" "${temp_project_folder_path}/gitlab-ci"
+
+############################### Start to make API calls ###############################
+gitlab_url="$(jq -c -r .instance_url <<<"${config}")/api/v4"
+readonly gitlab_url
+
+group_id=$(jq -c -r .group_id <<<"${config}")
+visibility=$(jq -c -r .default_visibility <<<"${config}")
+
+# create a default group where to save the pipeline project if is not given by the user
+if [[ -z ${group_id} ]]; then
+	echo "Creating default base group: ${DEFAULT_GROUP_NAME}"
+	data="{\"path\":\"mia-platform\",\"name\":\"Mia-Platform\",\"visibility\": \"${visibility}\"}"
+	group_response=$(curl -q -H "PRIVATE-TOKEN: ${TOKEN}" -H "Content-type: application/json" --data "${data}" "${gitlab_url}/groups")
+	group_id=$(jq -c -r .id <<<"${group_response}")
+fi
+
+# create the project where to save the pipeline files
+echo "Creating project..."
+data="{\"path\":\"pipeline-templates\",\"name\":\"Pipeline Templates\",\"visibility\": \"${visibility}\",\"namespace_id\":${group_id}}"
+project_response=$(curl -q -H "PRIVATE-TOKEN: ${TOKEN}" -H "Content-type: application/json" --data "${data}" "${gitlab_url}/projects")
+
+############################### End to make API calls ###############################
+
+############################### Start to update pipelines data ###############################
+
+# change pipeline images base name if necessary
+project_path=$(jq -c -r .path_with_namespace <<<"${project_response}")
+pipeline_image_name=$(jq -c -r .pipeline_images_base_name <<<"${config}")
+if [[ "${pipeline_image_name}" != "${ORIGINAL_CONTAINER_PATH}" ]]; then
+	echo "Changing pipeline CONTAINER_PATH variable content to: ${pipeline_image_name}"
+	perl -i -pe "s|${ORIGINAL_CONTAINER_PATH}|${pipeline_image_name}|g" "${temp_project_folder_path}/Application.gitlab-ci.yml"
+fi
+
+# change include path for the pipeline templates
+
+if [[ "${project_path}" != "${ORIGINAL_INCLUDE_PATH}" ]]; then
+	echo "Changing templates include path to: ${project_path}"
+	find "${temp_project_folder_path}" -type f -name "*.gitlab-ci.yml" -exec perl -i -pe "s|project: ${ORIGINAL_INCLUDE_PATH}|project: ${project_path}|g" {} \;
+fi
+
+############################### End to update pipelines data ###############################
+
+############################### Push Repository ###############################
+
+project_http_url=$(jq -c -r .http_url_to_repo <<<"${project_response}")
+echo "Push repository to ${project_http_url} using provided username and token..."
+http_sep="://"
+project_with_auht="${project_http_url%%"${http_sep}"*}${http_sep}${USERNAME}:${TOKEN}@${project_http_url##*"${http_sep}"}"
+git init -q -b main "${temp_project_folder_path}"
+git -C "${temp_project_folder_path}" add .
+git -C "${temp_project_folder_path}" commit --quiet --message "Initial commit"
+git -C "${temp_project_folder_path}" push --quiet --set-upstream "${project_with_auht}" main
+echo "Push completed"
+
+############################### Cleanup ###############################
+
+echo "Remove temporary files..."
+rm -fr "${temp_project_folder_path}"

containers/README.md

@@ -0,0 +1,8 @@
+# Pipelines Dedicated Containers
+
+This folders contains all the dedicated containers used inside the GitLab pipelines. Relying on official images of the
+various tools and languages is sometimes limiting on the things that we can do inside a single job so we build these
+images.
+
+All images will start from the `base-pipeline` image that contains syft, cosing and basic utilities for easier scripting
+during the job execution.

docs/10-starting-guide.md

@@ -0,0 +1,28 @@
+# Mia-Platform Pipelines
+
+In case you don‘t have a unified pipeline strategy for building libraries and Docker images, you can start with
+our pipelines. They are a starting step for quickly setting them up following best practictes and trying to
+add useful steps that can match regulatory compliance for your company.
+
+## Supported Tecnologies
+
+Here you can find all the languages and CI/CD systems currently supported:
+
+| Language  | [GitLab CI/CD]
+| --- | --- |
+| [Node.js] with [NPM] | Yes |
+| [Node.js] with [Yarn] | Yes |
+| [Go] | Yes |
+
+## GitLab CI/CD
+
+For GitLab CI/CD you can follow the following guides:
+
+- [Installation](./20-gitlab-installation.md): how to setup the templates in your SaaS or Self-managed instance
+- [Templates](./gitlab-ci/10-templates.md): a rundown of all the templates files and what they contains
+
+[GitLab CI/CD]: https://docs.gitlab.com/ee/ci/ (GitLab CI documentation site)
+[Node.js]: https://nodejs.org (Node.js® is a JavaScript runtime built on Chrome’s V8 JavaScript engine)
+[NPM]: https://www.npmjs.com (npm is the world’s largest software registry)
+[Yarn]: https://yarnpkg.com (Yarn is a package manager that doubles down as project manager)
+[Go]: https://go.dev (Go is an open source programming language that makes it simple to build secure, scalable systems)

docs/20-gitlab-installation.md

@@ -0,0 +1,80 @@
+# How to Use the GitLab CI/CD in your Organization
+
+To use effectively the GitLab CI/CD pipelines templates in your organizations you have to save them in your
+GitLab instance if you use self-service or inside the group of your organization inside GitLab SaaS.
+
+To do so we provide a [script](../bin/gitlab/install) you can run that will copy all the required files
+in a dedicated project.
+
+## Requirements
+
+- [git] installed and configured with username and email
+- [jq]
+- a user on the target GitLab instance that can create groups
+- a personal access token of that user with `api` and `write_repository` permissions
+
+## Configuration File
+
+To run correctly you will need to setup the configuration file with the appropriate values. You can find it in this
+repository under [`bin` > `gitlab > config.json`](../bin/gitlab/config.json) and below you can find the default content:
+
+```json
+{
+	"instance_url": "https://gitlab.com",
+	"group_id": "",
+	"pipeline_images_base_name": "ghrc.io/mia-platform",
+	"default_visibility": "internal"
+}
+```
+
+For an explanation of the various keys and possible values reference the following table:
+
+| key | value |
+| --- | --- |
+| **instance_url** | The url of the GitLab instance, default to the GitLab SaaS installation |
+| **group_id** | The id of the parent group where to create the pipeline template project, if left empty a `mia-platform` group will be created on the root of the instance, for GitLab SaaS this values cannot be empty |
+| **pipeline_images_base_name** | If you cannot access directly the public repositories where the images for the jobs are hosted you can change this to the mirror where they will be available |
+| **default_visibility** | The visibility set for the project and optional group that will be created, by default they will be set as internal, but you can set `public` or `private` if you need to |
+
+## Install the Templates
+
+1. clone the repository on your local machine
+
+	```sh
+	git clone https://github.com/mia-platform/public-pipelines.git
+	```
+
+1. edit the default values of the confiugration file with your favorite editor
+
+	```sh
+	${EDITOR} ./public-pipelines/bin/gitlab/config.json
+	```
+
+1. set the GitLab username and token as env variables
+
+	```sh
+	export GITLAB_USERNAME="<username>"
+	export GITLAB_TOKEN="<user-token>"
+	```
+
+1. run the script
+
+	```sh
+	./public-pipelines/bin/gitlab/install -u "${GITLAB_USERNAME}" \
+		-t "${GITLAB_TOKEN}" -c \
+		"./public-pipelines/bin/gitlab/config.json"
+	```
+
+1. after the run is complete you can delete the cloned repository if you want
+
+	```sh
+	rm -fr ./public-pipelines
+	```
+
+## Setting Pipeline Templates
+
+If you are a Premium subscriber you can use the newly created project as an instance or group template repository.
+All pipelines used for the public marketplace can also be used as templates for your projects that do not use the console.
+
+[git]: https://git-scm.com (Git is a free and open source distributed version control system )
+[jq]: https://jqlang.github.io/jq/ (jq is a lightweight and flexible command-line JSON processor)

docs/gitlab-ci/10-templates.md

@@ -0,0 +1,65 @@
+# GitLab CI/CD Templates
+
+The GitLab CI/CD templates are composed by multiple files and here you can find a brief overview of what
+they contains, and how, the various jobs that they will add, works and can be customized.
+
+## Main Files
+
+### Application.gitlab-ci.yml
+
+The `Application.gitlab-ci.yml` file is the file that will import
+every other files. It will not declare any jobs, but will contain imports of all the other files and the GitLab
+templates that will form the base for your pipelines. This will allow you to only import this file in your
+`.gitlab-ci.yaml` file inside your repository.
+
+### Governance.gitlab-ci.yml
+
+In the `Governance.gitlab-ci.yml` file there will be added all the jobs
+and templates that add security and compliance steps to the pipelines.  
+This is a separeted file so if you are an Ultimate subscriber you can set it as a [Compliance pipelines] in your
+GitLab environment groups.
+
+The pipeline is conditionally included inside the `Application.gitlab-ci.yml` file if we detect that the Compliance
+pipelines feature is turned off.
+
+Once set to be enforced at the Group and Project level. This file will be the file GitLab CI processes and runs first.
+Everything set and defined in this file will be immutable and unable to be changed or overwritten by the development
+teams. This file will invoke the `.gitlab-ci.yml`` file in a team's repository; there the team can define any number
+of jobs or variables. However, what's defined in this file will take precedent over anything defined in the
+individual projects.
+
+In this file will also explicited the standard stages for the pipelines:
+
+- `build`
+- `sast`
+- `test`
+- `container-build`
+- `compliance`
+- `deploy`
+
+### GovernanceOverrides.gitlab-ci.yml
+
+In the `GovernanceOverrides.gitlab-ci.yml`file there will be added overrides for the security steps included inside
+the Governance file.
+
+### CustomOverrides.gitlab-ci.yml
+
+In the `CustomOverrides.gitlab-ci.yml` file the maintainers of the templates for your organization can add all the
+overrides needed by them to conform the pipeline templates to their GitLab environment.  
+We reccomend to use only this file to ad your overrides to simplify eventual upgrades of the templates after the
+first installation.
+
+## Jobs Folder
+
+The main part of the pipeline templates are implemented inside the `jobs` folder, inside that folder you can find
+various files that create the jobs you can use to compose your pipelines.
+
+- [npm](./20-npm.md)
+- [yarn](./30-yarn.md)
+- [golang](./40-golang.md)
+- [docker](./50-docker.md)
+- [licenses](./60-licenses.md)
+- [sysdig](./70-sysdig.md)
+
+[Compliance pipelines]: https://docs.gitlab.com/ee/user/group/compliance_frameworks.html#compliance-pipelines
+	(GitLab compliance pipelines documentation site)