-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #182 from flanksource/fix/catalog-improvements
Fix/catalog improvements
- Loading branch information
Showing
39 changed files
with
976 additions
and
677 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,16 @@ | ||
| { | ||
| "cSpell.words": [ | ||
| "Unmarshaler" | ||
| "apiserver", | ||
| "kubeconfig", | ||
| "omitempty", | ||
| "organisation", | ||
| "pgdata", | ||
| "Replicaset", | ||
| "struct", | ||
| "Suceeded", | ||
| "templatable", | ||
| "trivy", | ||
| "Unmarshaler", | ||
| "vuln" | ||
| ] | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,109 @@ | ||
| import React from 'react' | ||
| import Admonition from '@theme/Admonition' | ||
| import useDocusaurusContext from '@docusaurus/useDocusaurusContext' | ||
| import Link from '@docusaurus/Link' | ||
|
|
||
| export function ConfigTransform() { | ||
| const { siteConfig } = useDocusaurusContext() | ||
| const tableRows = [ | ||
| { | ||
| field: 'gotemplate', | ||
| description: 'Go template to transform the scraped config item', | ||
| scheme: 'string', | ||
| required: '' | ||
| }, | ||
| { | ||
| field: 'javascript', | ||
| description: 'Javascript to transform the scraped config item', | ||
| scheme: 'string', | ||
| required: '' | ||
| }, | ||
| { | ||
| field: 'jsonpath', | ||
| description: 'Specify JSONPath', | ||
| scheme: 'jsonpath', | ||
| schemaLink: siteConfig.customFields.links['jsonpath'], | ||
| required: '' | ||
| }, | ||
| { | ||
| field: 'expr', | ||
| description: 'Specify Cel expression', | ||
| scheme: 'cel-expression', | ||
| schemaLink: siteConfig.customFields.links['cel'], | ||
| required: '' | ||
| }, | ||
| { | ||
| field: 'changes', | ||
| description: 'Apply transformation on the scraped changes', | ||
| scheme: '[]Changes', | ||
| schemaLink: '../concepts/changes', | ||
| required: '' | ||
| }, | ||
| { | ||
| field: 'exclude', | ||
| description: | ||
| 'Fields to remove from the config, useful for removing sensitive data and fields that change often without a material impact i.e. Last Scraped Time', | ||
| scheme: '[]Exclude', | ||
| schemaLink: '../concepts/exclusion', | ||
| required: '' | ||
| }, | ||
| { | ||
| field: 'mask', | ||
| description: | ||
| 'Specify configurations to replace sensitive fields with hash functions or static string.', | ||
| scheme: '[]Mask', | ||
| schemaLink: '../concepts/masking', | ||
| required: '' | ||
| }, | ||
| { | ||
| field: 'relationship', | ||
| description: 'Form relationships between config items using selectors', | ||
| scheme: '[]RelationshipConfig', | ||
| schemaLink: '../concepts/relationship#relationship-config', | ||
| required: '' | ||
| } | ||
| ] | ||
|
|
||
| return ( | ||
| <> | ||
| <table> | ||
| <thead> | ||
| <tr> | ||
| <th>Field</th> | ||
| <th>Description</th> | ||
| <th>Scheme</th> | ||
| <th>Required</th> | ||
| </tr> | ||
| </thead> | ||
| <tbody> | ||
| {tableRows.map((row, index) => ( | ||
| <tr key={index}> | ||
| <td> | ||
| <code>{row.field}</code> | ||
| </td> | ||
| <td>{row.description}</td> | ||
| <td> | ||
| {row.schemaLink ? ( | ||
| <Link to={row.schemaLink}> | ||
| <code>{row.scheme}</code> | ||
| </Link> | ||
| ) : ( | ||
| <code>{row.scheme}</code> | ||
| )} | ||
| </td> | ||
| <td>{row.required}</td> | ||
| </tr> | ||
| ))} | ||
| </tbody> | ||
| </table> | ||
|
|
||
| <Admonition type="info"> | ||
| <p> | ||
| Both the go template and javascript receive{' '} | ||
| <a href="/config-db/references/scrape-result">ScrapeResult</a> as the | ||
| template variable. | ||
| </p> | ||
| </Admonition> | ||
| </> | ||
| ) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,134 @@ | ||
| # Changes | ||
|
|
||
| When you save a config item to ConfigDB, it also tracks its changes. These changes may come from external sources like Kubernetes events or Azure activities. Alternatively, they can be automatically detected by comparing the old config with the newly changed one. | ||
|
|
||
| ## Types | ||
|
|
||
| Changes are of two types | ||
|
|
||
| - Diff change | ||
| - Event based change | ||
|
|
||
| ### Diff Change | ||
|
|
||
| These are changes generated by ConfigDB by simply comparing the old and new config values. The image below shows a change where the port value was modified from 8080 to 3000. | ||
|
|
||
|  | ||
|
|
||
| ### Event based change | ||
|
|
||
| These are changes provided by an external source example: Kubernetes Events & AWS CLoudtrail. Event based changes have a type associated to it. | ||
|
|
||
|  | ||
|
|
||
| ## Change Transformation | ||
|
|
||
| You can transform a change before it's saved to the database. | ||
|
|
||
| | Field | Description | Scheme | Required | Templatable | | ||
| | --------- | ----------------------------------------------------------------------------- | ------------------------------------------- | -------- | ----------- | | ||
| | `mapping` | Mapping is a list of CEL expressions that maps a change to the specified type | [_`[]ChangeMapping`_](#mapping) | | | | ||
| | `exclude` | Select what changes to exclude | <CommonLink to="cel">_`[]CEL`_</CommonLink> | | `true` | | ||
|
|
||
| ### Exclusions | ||
|
|
||
| Some configs can have changes in high volume that may not be relevant. Example: A kubernetes Node config changes frequently as the pods in the cluster update their images. From the node's perspective the image changes are irrelevant. | ||
|
|
||
| This is where exclusions can become handy. Here's an example that ignore all image changes in a kubernetes node config: | ||
|
|
||
| ```yaml title="kubernetes-scraper.yaml" | ||
| apiVersion: configs.flanksource.com/v1 | ||
| kind: ScrapeConfig | ||
| metadata: | ||
| name: kubernetes-scraper | ||
| spec: | ||
| kubernetes: | ||
| - clusterName: local-kind-cluster | ||
| transform: | ||
| changes: | ||
| exclude: | ||
| - 'config_type == "Kubernetes::Node" && details.message == "status.images"' | ||
| ``` | ||
| #### Template Variables | ||
| | Field | Description | Scheme | | ||
| | -------------------- | ---------------------------------------------------------------------- | ---------------- | | ||
| | `action` | Action of the change | `delete\|ignore` | | ||
| | `change_type` | Type of the change | `string` | | ||
| | `config_type` | Type of the config | `string` | | ||
| | `config` | The corresponding json config of the change | `map[string]any` | | ||
| | `created_at` | Creation time of the change | `time.Time` | | ||
| | `created_by` | User who created the change | `string` | | ||
| | `details` | Details of the change. Example: it could be a kubernetes event object. | `map[string]any` | | ||
| | `diff` | Diff of the change | `string` | | ||
| | `external_change_id` | ID of the change in the external system | `string` | | ||
| | `external_id` | ID of the change | `string` | | ||
| | `patches` | Patches of the change | `string` | | ||
| | `severity` | Severity of the change | `string` | | ||
| | `source` | Source of the change | `string` | | ||
| | `summary` | Summary of the change | `string` | | ||
|
|
||
| ### Mapping | ||
|
|
||
| When you encounter a diff change, unlike an event based change, it can sometimes appear cryptic. The summary of the change may not immediately indicate what the change is about. For example, the change 'status.images' might not be self-explanatory. To address this issue, we can assign types to these diff changes using mapping. | ||
|
|
||
| ```yaml title="kubernetes-scraper.yaml" | ||
| apiVersion: configs.flanksource.com/v1 | ||
| kind: ScrapeConfig | ||
| metadata: | ||
| name: kubernetes-scraper | ||
| spec: | ||
| kubernetes: | ||
| - clusterName: local-kind-cluster | ||
| transform: | ||
| changes: | ||
| mapping: | ||
| - filter: > | ||
| change.change_type == 'diff' && change.summary == "status.containerStatuses" && | ||
| patch != null && has(patch.status) && has(patch.status.containerStatuses) && | ||
| patch.status.containerStatuses.size() > 0 && | ||
| has(patch.status.containerStatuses[0].restartCount) | ||
| type: PodCrashLooping | ||
| - filter: > | ||
| change.change_type == 'diff' && change.summary == "status.images" && config.kind == "Node" | ||
| type: ImageUpdated | ||
| ``` | ||
|
|
||
| | Field | Description | Scheme | Required | | ||
| | --------- | ------------------------------------ | ----------------------------------------- | -------- | | ||
| | `filter` | Selects changes to apply the mapping | <CommonLink to="cel">_`CEL`_</CommonLink> | `true` | | ||
| | `exclude` | Type to be set on the change | `string` | `true` | | ||
|
|
||
| #### Template Variables | ||
|
|
||
| | Field | Description | Scheme | | ||
| | ------------- | ------------------------- | ---------------------------------------------- | | ||
| | `config_type` | Type of the config | `string` | | ||
| | `config` | Access the scraped config | [`ScrapedResult`](../references/scrape-result) | | ||
|
|
||
| ## Change Retention | ||
|
|
||
| Changes can quickly accumulate and grow large over time. While it's important to retain some changes, others can be discarded after a period. Retention enables you to discard changes based on their type. | ||
|
|
||
| Example: The scraper config below only retains one week of PullSuceeded changes. | ||
|
|
||
| ```yaml title="kubernetes-scraper.yaml" | ||
| apiVersion: configs.flanksource.com/v1 | ||
| kind: ScrapeConfig | ||
| metadata: | ||
| name: kubernetes-scraper | ||
| spec: | ||
| retention: | ||
| changes: | ||
| - name: PullSuceeded | ||
| age: 7d # Only keep one week of PullSuceeded changes | ||
| kubernetes: | ||
| clusterName: local | ||
| ``` | ||
|
|
||
| | Field | Description | Scheme | Required | | ||
| | ------- | ------------------------------------------------------- | -------- | -------- | | ||
| | `name` | Name of the change type | `string` | `true` | | ||
| | `age` | Maximum age of the change type to retain (`12h`, `30d`) | `string` | | | ||
| | `count` | Maximum count to retain the change type | `int` | | |
46 changes: 46 additions & 0 deletions
46
mission-control/docs/config-db/concepts/custom-scrapers.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| # Custom Scrapers | ||
|
|
||
| Custom scrapers allow you to scrape from sources that are not well defined eg: scraping a file of an arbitrary structure. It is upon the user to define how the configs should be scraped from the source. | ||
|
|
||
| ```yaml title="file-scraper.yaml" | ||
| apiVersion: configs.flanksource.com/v1 | ||
| kind: ScrapeConfig | ||
| metadata: | ||
| name: file-scraper | ||
| spec: | ||
| file: | ||
| - type: $.Config.InstanceType | ||
| class: $.Config.InstanceType | ||
| id: $.Config.InstanceId | ||
| path: | ||
| - config*.json | ||
| - test*.json | ||
| ``` | ||
| ## Mapping | ||
| Custom scrapers need to define the id, type & class for each items that are scraped. For example: if you're scraping a file with a JSON Array where each element on the array is to be scraped as a config item, you need to define what the id, type & config class of the items should be. This can be done using mappings. | ||
| | Field | Description | Scheme | Required | | ||
| | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | -------- | | ||
| | `id` | ID for the config item | <CommonLink to="jsonpath">_`jsonpath`_</CommonLink> | `true` | | ||
| | `type` | Type for the config item | <CommonLink to="jsonpath">_`jsonpath`_</CommonLink> | `true` | | ||
| | `class` | Class for the config item. _(Defaults to `type`)_ | <CommonLink to="jsonpath">_`jsonpath`_</CommonLink> | | | ||
| | `name` | Name for the config item | <CommonLink to="jsonpath">_`jsonpath`_</CommonLink> | | | ||
| | `createFields` | JSONPath expression and are used to extract the created time of the config item from the scraped configuration. _(If multiple fields are specified, the first non-empty value will be used)_ | <CommonLink to="jsonpath">_`jsonpath`_</CommonLink> | | | ||
| | `deleteFields` | JSONPath expression and are used to extract the deleted time of the config item from the scraped configuration. _(If multiple fields are specified, the first non-empty value will be used)_ | <CommonLink to="jsonpath">_`jsonpath`_</CommonLink> | | | ||
| | `timestampFormat` | Timestamp format for fields specified in `createFields` & `deleteFields`. _(Default 2006-01-02T15:04:05Z07:00)_ | [`timestampFormat`](#timestamp-format) | | | ||
|
|
||
| ### Date Mapping | ||
|
|
||
| In Mission Control, config items possess created, updated, and deleted dates. While Kubernetes and Cloud resources receive these dates from the API, custom config items require extraction of these values from the config item itself. If no date mapping is provided, the scrape time will be used instead. | ||
|
|
||
| #### Timestamp Format | ||
|
|
||
| The format is specified using the [Go time format](https://golang.org/pkg/time/#Time.Format). | ||
|
|
||
| In the above example if the value of `made_at` was `2017/03/06 21:04:11Z`, then the `timestampFormat` file would look like this | ||
|
|
||
| ```yaml | ||
| timestampFormat: '2006/01/02 15:04:05Z' | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| # Exclusions | ||
|
|
||
| Exclusions allow you to remove certain fields from the scraped configuration. This is useful when you want to remove sensitive or just useless data from the scraped configuration. | ||
|
|
||
| ## Exclude | ||
|
|
||
| | Field | Description | Scheme | Required | | ||
| | ---------- | -------------------------------------------------------------------------- | ---------- | -------- | | ||
| | `jsonpath` | Specify JSONPath expression for the fields | `string` | `true` | | ||
| | `types` | specify the config types from which the JSONPath fields need to be removed | `[]string` | | | ||
|
|
||
| The `types` field is optional and if left empty, the filter will apply to all config items. [See Example](../examples/exclude-fields) |
Oops, something went wrong.