K8s docs/alpine fixes (#58)

* script +x permissions, fix skaffold, doc fixes

* change waiter to use debian slim

* new pull image scripts

* add presentations

README.md

@@ -34,26 +34,37 @@ Built on Kubernetes with scale in mind, our goal with Nemesis was to create a ce
 Nemesis aims to automate a number of repetitive tasks operators encounter on engagements, empower operators’ analytic capabilities and collective knowledge, and create structured and unstructured data stores of as much operational data as possible to help guide future research and facilitate offensive data analysis.
 
 ## Setup / Installation
-Follow the [quickstart guide](docs/quickstart.md)
+Follow the [quickstart guide](docs/quickstart.md).
 
-Or see the full [setup instructions](docs/setup.md)
+Or see the full [setup instructions](docs/setup.md).
 
 ## Usage
 See the [Nemesis Usage Guide](docs/usage_guide.md).
 
 ## Contributing / Development Environment Setup
-See [development.md](./docs/development.md)
-
-## Further Reading
-
-| Post Name                                   | Publication Date | Link                                                                               |
-|---------------------------------------------|------------------|------------------------------------------------------------------------------------|
-| *Nemesis 1.0.0*                             | Apr 25, 2024     | https://posts.specterops.io/nemesis-1-0-0-8c6b745dc7c5                             |
-| *Summoning RAGnarok With Your Nemesis*      | Mar 13, 2024     | https://posts.specterops.io/summoning-ragnarok-with-your-nemesis-7c4f0577c93b      |
-| *Shadow Wizard Registry Gang: Structured Registry Querying* | Sep 5, 2023 | https://posts.specterops.io/shadow-wizard-registry-gang-structured-registry-querying-9a2fab62a26f |
-| *Hacking With Your Nemesis*                 | Aug 9, 2023      | https://posts.specterops.io/hacking-with-your-nemesis-7861f75fcab4                 |
-| *Challenges In Post-Exploitation Workflows* | Aug 2, 2023      | https://posts.specterops.io/challenges-in-post-exploitation-workflows-2b3469810fe9 |
-| *On (Structured) Data*                      | Jul 26, 2023     | https://posts.specterops.io/on-structured-data-707b7d9876c6                        |
+See [development.md](./docs/development.md).
+
+## Additional Information
+Blog Posts:
+
+| Title                                                                                                                                                            | Date         |
+|------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|
+| [*Nemesis 1.0.0*](https://posts.specterops.io/nemesis-1-0-0-8c6b745dc7c5)                                                                                        | Apr 25, 2024 |
+| [*Summoning RAGnarok With Your Nemesis*](https://posts.specterops.io/summoning-ragnarok-with-your-nemesis-7c4f0577c93b)                                          | Mar 13, 2024 |
+| [*Shadow Wizard Registry Gang: Structured Registry Querying*](https://posts.specterops.io/shadow-wizard-registry-gang-structured-registry-querying-9a2fab62a26f) | Sep 5, 2023  |
+| [*Hacking With Your Nemesis*](https://posts.specterops.io/hacking-with-your-nemesis-7861f75fcab4)                                                                | Aug 9, 2023  |
+| [*Challenges In Post-Exploitation Workflows*](https://posts.specterops.io/challenges-in-post-exploitation-workflows-2b3469810fe9)                                | Aug 2, 2023  |
+| [*On (Structured) Data*](https://posts.specterops.io/on-structured-data-707b7d9876c6)                                                                            | Jul 26, 2023 |
+
+
+Presentations:
+
+| Title                                                                      | Date         |
+|----------------------------------------------------------------------------|--------------|
+| [*SAINTCON 2023*](https://www.youtube.com/watch?v=0q9u2hDcpIo)             | Oct 24, 2023 |
+| [*BSidesAugusta 2023*](https://www.youtube.com/watch?v=Ug9r7lCF_FA)        | Oct 7, 2023  |
+| [*44CON 2023*](https://www.youtube.com/watch?v=tjPTLBGI7K8)                | Sep 15, 2023 |
+| [*BlackHat Arsenal USA 2023*](https://www.youtube.com/watch?v=Ms3o8n6aS0c) | Sep 15, 2023 |
 
 
 ## Acknowledgments

dockerfiles/waiter.Dockerfile

@@ -1,3 +1,3 @@
-FROM alpine:3.17.0
+FROM debian:11-slim
 
-RUN apk --update add jq curl postgresql-client
+RUN apt-get update && apt-get install -y jq curl postgresql-client

docs/odr/README.md

@@ -24,7 +24,7 @@ The goal of Nemesis is to store operational data that may be useful to do automa
 All data must be encoded as UTF-8. Data must be formated as JSON to be parsed by Nemesis. The `metadata` and `data` sections of the submission will both contain information of different formats such as timestamps, booleans, and binary data. Conventions for each are defined here. Any information format that is unique to an ODR will be specified within that specific ODR. Any string value may include hex escape sequences. Escape sequences for UTF-8 is not needed because all submissions are already in UTF-8.
 
 | Base Information Formats | Standard and encodings                                                    |
-| ------------------------ | ------------------------------------------------------------------------- |
+|--------------------------|---------------------------------------------------------------------------|
 | UUID                     | Nemesis UUID string referencing a binary data already uploaded to Nemesis |
 | bool                     | True or False booleans                                                    |
 | datetime                 | ISO 8601 data timestamp in UTC                                            |
@@ -43,16 +43,17 @@ The metadata must include an experation date for when the data should be removed
 Finally, the metadata must include a type name to define which ODR the body of the data submission must conform to.
 
 Values:
-| Name       | Format   | Description                                                                    |
-| ---------- | -------- | ------------------------------------------------------------------------------ |
-| agent_id   | string   | Name or unique identifier for an agent                                         |
-| agent_type | string   | Name of the type of agent (ex. beacon/apollo/etc.)                             |
-| automated  | bool     | True if the submission was submitted automatically                             |
-| data_type  | string   | Name of the ODR the body must conform to                                       |
-| expiration | datetime | Time the data should be removed from Nemesis                                   |
-| source     | string   | Info about the source of the data. See each data type for possible values      |
-| project    | string   | Name or unique identifier for a project                                        |
-| timestamp  | datetime | Time the C2 platform sent the data to Nemesis                                  |
+
+| Name       | Format   | Description                                                               |
+|------------|----------|---------------------------------------------------------------------------|
+| agent_id   | string   | Name or unique identifier for an agent                                    |
+| agent_type | string   | Name of the type of agent (ex. beacon/apollo/etc.)                        |
+| automated  | bool     | True if the submission was submitted automatically                        |
+| data_type  | string   | Name of the ODR the body must conform to                                  |
+| expiration | datetime | Time the data should be removed from Nemesis                              |
+| source     | string   | Info about the source of the data. See each data type for possible values |
+| project    | string   | Name or unique identifier for a project                                   |
+| timestamp  | datetime | Time the C2 platform sent the data to Nemesis                             |
 
 
 Json Example:

docs/requirements_minikube.md

@@ -111,9 +111,13 @@ You will need to install two services in k8s before getting started. Helm makes
 ```bash
 # Add Bitnami repository
 helm repo add bitnami https://charts.bitnami.com/bitnami
-# Install Traefik ingress
-helm install traefik traefik --repo https://traefik.github.io/charts --namespace kube-system
+
 # Install ElasticSearch operator to manage "default" namespace. The managedNamespaces field will need to be configured if you desire to install Nemesis in a different namespace
 helm install elastic-operator eck-operator --repo https://helm.elastic.co --namespace elastic-system --create-namespace --set managedNamespaces='{default}'
+
+# Install Traefik v2
+helm repo add traefik https://traefik.github.io/charts
+helm install traefik traefik/traefik -n kube-system --version 27.0.2
+
 ```
 </details>

helm/nemesis/templates/elasticsearch.yaml

@@ -67,7 +67,7 @@ spec:
               command: ["sh", "-c", "sysctl -w vm.max_map_count=262144"]
             {{- if eq .Values.operation.environment "production" }}
             - name: volume-mount-hack
-              image: busybox
+              image: busybox:1.36.1
               imagePullPolicy: IfNotPresent
               securityContext:
                 privileged: true

scripts/pull_images_k3s.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+pushd "$SCRIPT_DIR/../" > /dev/null
+
+if [ $# -gt 0 ]; then
+    # Grab the images required for building Nemesis
+    IMAGES=$(skaffold render -m nemesis -m enrichment --digest-source=tag | grep 'image:' | sed 's/^[ \t]*image: //' | sed 's/"//g' | grep -v '^nemesis-' | sort -u)
+else
+    # Grab the images from the helm chart - this will pull the images from our Docker image registry
+    IMAGES=$(helm template helm/nemesis/ | grep 'image:' | sed 's/^[ \t]*image: //' | sed 's/"//g' | sort -u)
+fi
+
+# Pull each image
+for image in $IMAGES; do
+    only_slashes="${image//[^\/]}"
+    slash_count="${#only_slashes}"
+
+    if ((slash_count == 0)); then
+        # Format: Just single a library image name provided (e.g. debian)
+        domain="docker.io"
+        image="library/${image}"
+    elif ((slash_count == 1)); then
+        # Format: REPO/NAME (e.g. specterops/nemesis)
+        domain="docker.io"
+        image="${image}"
+    elif ((slash_count == 2)); then
+        # Format: DOMAIN.LOCAL/REPO/NAME (e.g., docker.io/specterops/nemesis)
+        domain=$(echo -n $image | awk -F[/] '{print $1}')
+        image=$(echo -n $image | sed "s/^${domain}\///")
+    else
+        echo "ERROR: too many slashes!"
+        exit
+    fi
+
+    # Check if we need to add the tag
+    if [[ ! "$image" =~ : ]]; then
+        image="${image}:latest"
+    fi
+
+    k3s ctr image pull "${domain}/${image}"
+done
+
+popd > /dev/null

scripts/pull_images.sh → scripts/pull_images_minikube.sh

@@ -1,10 +1,10 @@
-#/bin/bash
+#!/bin/bash
 
 SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 pushd "$SCRIPT_DIR/../" > /dev/null
 
 
-DOCKER_IMAGES=$(cat dockerfiles/* | grep -o 'FROM .* ' | grep -Ev 'base|dependencies|debcommon|build|cobaltstrike' | sed -r 's/FROM (.*) as/\1/i' | sort -ui)
+DOCKER_IMAGES=$(cat dockerfiles/* | grep -o 'FROM .* ' | grep -Ev 'base|dependencies|debcommon|build|cobaltstrike|model_download|yara-rules' | sed -r 's/FROM (.*) as/\1/i' | sort -ui)
 DEPLOYMENT_IMAGES=$(skaffold render | grep 'image: ' | sed -r 's/.*image: (.*)/\1/' | sort -u | grep -vE ':[a-z0-9]{64}$')
 
 # combine and unique the two lists

skaffold.yaml

@@ -69,30 +69,14 @@ deploy:
         jupyter.image.pullPolicy: IfNotPresent
         nlp.image.pullPolicy: IfNotPresent
         passwordcracker.image.pullPolicy: IfNotPresent
-    - name: nemesis-monitoring
-      chartPath: helm/monitoring
-      setValueTemplates:
-        nemesisWaiter.image.repository: "{{.IMAGE_REPO_nemesis_waiter}}"
-        nemesisWaiter.image.tag: "{{.IMAGE_TAG_nemesis_waiter}}@{{.IMAGE_DIGEST_nemesis_waiter}}"
-      setValues:
-        nemesisWaiter.image.pullPolicy: IfNotPresent
-    flags:
-      upgrade:
-        ["--timeout=30m"]
-      install:
-        ["--timeout=45m"]
+
 portForward:
   - resourceType: service
-    resourceName: ingress-nginx-controller
-    namespace: ingress-nginx
+    resourceName: traefik
+    namespace: kube-system
     port: 80
     localPort: 8080
     address: 0.0.0.0
-  - resourceType: service
-    resourceName: jupyter
-    namespace: default
-    port: 8888
-    address: 0.0.0.0
 ---
 apiVersion: skaffold/v4beta6
 kind: Config
@@ -125,4 +109,33 @@ deploy:
         enrichment.image.repository: "{{.IMAGE_REPO_enrichment_dev}}"
         enrichment.image.tag: "{{.IMAGE_REPO_enrichment_dev}}@{{.IMAGE_DIGEST_enrichment_dev}}"
       setValues:
-        enrichment.image.pullPolicy: IfNotPresent
+        enrichment.image.pullPolicy: IfNotPresent
+---
+apiVersion: skaffold/v4beta6
+kind: Config
+metadata:
+  name: monitoring
+build:
+  tagPolicy:
+    sha256: {}
+  local:
+    push: false
+    tryImportMissing: false
+    useBuildkit: true
+    concurrency: 0
+  artifacts: []
+deploy:
+  helm:
+    releases:
+    - name: nemesis-monitoring
+      chartPath: helm/monitoring
+      setValueTemplates:
+        nemesisWaiter.image.repository: "{{.IMAGE_REPO_nemesis_waiter}}"
+        nemesisWaiter.image.tag: "{{.IMAGE_TAG_nemesis_waiter}}@{{.IMAGE_DIGEST_nemesis_waiter}}"
+      setValues:
+        nemesisWaiter.image.pullPolicy: IfNotPresent
+    flags:
+      upgrade:
+        ["--timeout=30m"]
+      install:
+        ["--timeout=45m"]