Upgrade binary test fixtures management #2444

spiffcs · 2023-12-18T20:56:11Z

Changes include:

adds PR validation to ensure that snippets are below 1000 bytes and are not executable. This additionally adds a script that verifies all binary test fixture snippets that this remains true on changes/additions.
adds a new test fixture helper for the binary cataloger fixtures (to manage snippets and full binaries)
ports all existing dynamic and positive test cases to use the new helper

The new test fixture helper can:

list binaries and snippets that are used by tests
manage downloading binaries only needed for testing (not already covered by a snippet) or all binaries. There are exceptions where original binaries could not be found.
helper command to capture snippets from larger binaries (manager add-snippet and capture-snippet.sh -- see the README in the test-fixtures directory for more info)
has a test helper function SnippetOrBinary to help select from available test fixtures (can be forced to only use full binaries). Responds to a new -must-use-full-binaries test flag (see the README added in the test-fixtures directory). Since there are cases where we have snippets but not the original binary, when using -must-use-full-binares these cases are skipped.

For listing what fixtures exist and what coverage there is between full binaries and snippets:

# at: syft/pkg/cataloger/binary/test-fixtures 

❯ make list
go run ./manager list

┌─────────────────────────────┬───────────────┬─────────────┬───────────────────────────────┬─────────────┬────────┬─────────┐
│ GROUP                       │ VERSION       │ PLATFORM    │ NAME                          │ CONFIGURED? │ BINARY │ SNIPPET │
├─────────────────────────────┼───────────────┼─────────────┼───────────────────────────────┼─────────────┼────────┼─────────┤
│ bash                        │ 5.1.16        │ linux/amd64 │ bash                          │ yes         │ yes    │ yes     │
│ busybox                     │ 1.36.1        │ linux/amd64 │ [                             │ yes         │ yes    │ yes     │
│ busybox                     │ 1.36.1        │ linux/amd64 │ busybox                       │             │        │ yes     │
│ consul                      │ 1.15.2        │ linux/amd64 │ consul                        │ yes         │ yes    │ yes     │
│ erlexec                     │ 25.3.2.6      │ linux/amd64 │ erlexec                       │ yes         │ yes    │ yes     │
│ erlexec                     │ 26.2.0.0      │ linux/amd64 │ erlexec                       │ yes         │ yes    │ yes     │
│ go                          │ 1.21.3        │ linux/amd64 │ go                            │ yes         │ yes    │ yes     │
│ go-version-hint             │ 1.15          │ any         │ VERSION                       │             │        │ yes     │
│ haproxy                     │ 1.5.14        │ linux/amd64 │ haproxy                       │ yes         │ yes    │ yes     │
│ haproxy                     │ 1.8.22        │ linux/amd64 │ haproxy                       │ yes         │ yes    │ yes     │
│ haproxy                     │ 2.7.3         │ linux/amd64 │ haproxy                       │ yes         │ yes    │ yes     │
│ helm                        │ 3.10.3        │ linux/amd64 │ helm                          │ yes         │ yes    │ yes     │
│ helm                        │ 3.11.1        │ linux/amd64 │ helm                          │ yes         │ yes    │ yes     │
│ httpd                       │ 2.4.54        │ linux/amd64 │ httpd                         │ yes         │ yes    │ yes     │
│ java-jre-ibm                │ 1.8.0_391     │ linux/amd64 │ java                          │ yes         │ yes    │ yes     │
│ java-jre-openjdk            │ 1.8.0_352-b08 │ linux/amd64 │ java                          │ yes         │ yes    │ yes     │
│ java-jre-openjdk            │ 11.0.17       │ linux/amd64 │ java                          │ yes         │ yes    │ yes     │
│ java-jre-oracle             │ 19.0.1        │ darwin      │ java                          │             │        │ yes     │
│ java-jre-oracle             │ 19.0.1        │ linux/amd64 │ java                          │             │        │ yes     │
│ mariadb                     │ 10.6.15       │ linux/amd64 │ mariadb                       │ yes         │ yes    │ yes     │
│ memcached                   │ 1.6.18        │ linux/amd64 │ memcached                     │ yes         │ yes    │ yes     │
│ mysql                       │ 5.6.51        │ linux/amd64 │ mysql                         │ yes         │ yes    │ yes     │
│ mysql                       │ 8.0.34        │ linux/amd64 │ mysql                         │ yes         │ yes    │ yes     │
│ nginx                       │ 1.25.1        │ linux/amd64 │ nginx                         │ yes         │ yes    │ yes     │
│ nginx-openresty             │ 1.21.4.3      │ linux/amd64 │ nginx                         │ yes         │ yes    │ yes     │
│ node                        │ 19.2.0        │ linux/amd64 │ node                          │ yes         │ yes    │ yes     │
│ openssl                     │ 3.1.4         │ linux/amd64 │ openssl                       │             │        │ yes     │
│ perl                        │ 5.12.5        │ linux/amd64 │ perl                          │             │        │ yes     │
│ perl                        │ 5.20.0        │ linux/amd64 │ perl                          │             │        │ yes     │
│ perl                        │ 5.37.8        │ linux/amd64 │ perl                          │             │        │ yes     │
│ php-apache                  │ 8.2.1         │ linux/amd64 │ libphp.so                     │             │        │ yes     │
│ php-cli                     │ 8.2.1         │ linux/amd64 │ php                           │             │        │ yes     │
│ php-fpm                     │ 8.2.1         │ linux/amd64 │ php-fpm                       │             │        │ yes     │
│ postgres                    │ 15.1          │ linux/amd64 │ postgres                      │ yes         │ yes    │ yes     │
│ postgres                    │ 15beta4       │ linux/amd64 │ postgres                      │ yes         │ yes    │         │
│ postgres                    │ 9.5alpha1     │ linux/amd64 │ postgres                      │             │        │ yes     │
│ postgres                    │ 9.6.24        │ linux/amd64 │ postgres                      │ yes         │ yes    │ yes     │
│ python                      │ 3.6.3         │ linux/amd64 │ python3.6                     │             │        │ yes     │
│ python-alpine-shared-libs   │ 3.4           │ linux/amd64 │ libpython3.4m.so.1.0          │ yes         │ yes    │         │
│ python-alpine-shared-libs   │ 3.4           │ linux/amd64 │ python3.4                     │ yes         │ yes    │         │
│ python-duplicates           │ 3.8.16        │ linux/amd64 │ dir/python3.8                 │             │        │ yes     │
│ python-duplicates           │ 3.8.16        │ linux/amd64 │ libpython3.8.so               │             │        │ yes     │
│ python-duplicates           │ 3.8.16        │ linux/amd64 │ patchlevel.h                  │             │        │ yes     │
│ python-duplicates           │ 3.8.16        │ linux/amd64 │ python3.8                     │             │        │ yes     │
│ python-rhel-shared-libs     │ 3.9           │ linux/amd64 │ libpython3.9.so.1.0           │ yes         │ yes    │         │
│ python-rhel-shared-libs     │ 3.9           │ linux/amd64 │ python3.9                     │ yes         │ yes    │         │
│ python-shared-lib           │ 3.7.4         │ linux/amd64 │ libpython3.7m.so.1.0          │ yes         │ yes    │ yes     │
│ python-slim-shared-libs     │ 3.11          │ linux/amd64 │ libpython3.11.so.1.0          │ yes         │ yes    │         │
│ python-slim-shared-libs     │ 3.11          │ linux/amd64 │ python3.11                    │ yes         │ yes    │         │
│ python-with-incorrect-match │ 3.5.3         │ linux/amd64 │ python3.5                     │             │        │ yes     │
│ python3.9                   │ 3.9.16        │ linux/amd64 │ python3.9                     │ yes         │ yes    │         │
│ redis-server                │ 2.8.23        │ linux/amd64 │ redis-server                  │ yes         │ yes    │         │
│ redis-server                │ 4.0.11        │ linux/amd64 │ redis-server                  │ yes         │ yes    │         │
│ redis-server                │ 5.0.0         │ linux/amd64 │ redis-server                  │ yes         │ yes    │         │
│ redis-server                │ 6.0.16        │ linux/amd64 │ redis-server                  │ yes         │ yes    │         │
│ redis-server                │ 7.0.0         │ linux/amd64 │ redis-server                  │ yes         │ yes    │         │
│ redis-server                │ 7.0.14        │ linux/amd64 │ redis-server                  │ yes         │ yes    │         │
│ redis-server                │ 7.2.3         │ linux/amd64 │ redis-server                  │ yes         │ yes    │         │
│ redis-server                │ 7.2.3         │ linux/arm64 │ redis-server                  │ yes         │ yes    │         │
│ ruby                        │ 1.9.3p551     │ linux/amd64 │ ruby                          │ yes         │ yes    │ yes     │
│ ruby-bullseye-shared-libs   │ 2.7.7         │ linux/amd64 │ libruby.so.2.7                │ yes         │ yes    │         │
│ ruby-bullseye-shared-libs   │ 2.7.7         │ linux/amd64 │ libruby.so.2.7.7              │ yes         │ yes    │         │
│ ruby-bullseye-shared-libs   │ 2.7.7         │ linux/amd64 │ ruby                          │ yes         │ yes    │         │
│ ruby-bullseye-shared-libs   │ 3.2.1         │ linux/amd64 │ libruby.so.3.2                │ yes         │ yes    │         │
│ ruby-bullseye-shared-libs   │ 3.2.1         │ linux/amd64 │ libruby.so.3.2.1              │ yes         │ yes    │         │
│ ruby-bullseye-shared-libs   │ 3.2.1         │ linux/amd64 │ ruby                          │ yes         │ yes    │         │
│ ruby-shared-libs            │ 2.6.10        │ linux/amd64 │ libruby.so.2.6                │ yes         │ yes    │         │
│ ruby-shared-libs            │ 2.6.10        │ linux/amd64 │ libruby.so.2.6.10             │ yes         │ yes    │         │
│ ruby-shared-libs            │ 2.6.10        │ linux/amd64 │ ruby                          │ yes         │ yes    │         │
│ rust-libstd                 │ 1.50.0        │ darwin      │ libstd-f6f9eec1635e636a.dylib │             │        │ yes     │
│ rust-libstd                 │ 1.50.0        │ linux/amd64 │ libstd-6f77337c1826707d.so    │ yes         │ yes    │ yes     │
│ rust-libstd                 │ 1.67.1        │ darwin      │ libstd-16f2b65e77054c42.dylib │             │        │ yes     │
│ rust-libstd                 │ 1.67.1        │ linux/amd64 │ libstd-c6192dd4c4d410ac.so    │ yes         │ yes    │ yes     │
│ rust-libstd-musl            │ 1.67.1        │ linux/amd64 │ libstd-86aefecbddda356d.so    │ yes         │ yes    │ yes     │
│ traefik                     │ 1.7.34        │ linux/amd64 │ traefik                       │ yes         │ yes    │ yes     │
│ traefik                     │ 2.9.6         │ linux/amd64 │ traefik                       │ yes         │ yes    │ yes     │
└─────────────────────────────┴───────────────┴─────────────┴───────────────────────────────┴─────────────┴────────┴─────────┘

To download fixtures:

# at: syft/pkg/cataloger/binary/test-fixtures 

❯ make download
go run ./manager download  --skip-if-covered-by-snippet

postgres@15beta4                      ⏭  already exists
[email protected]                   ⏭  already exists
[email protected]                   ⏭  already exists
[email protected]                    ⏭  already exists
[email protected]                   ⏭  already exists
[email protected]                    ⏭  already exists
[email protected]                   ⏭  already exists
[email protected]                    ⏭  already exists
[email protected]           ⏭  already exists
[email protected]          ⏭  already exists
[email protected]                      ⏭  already exists
[email protected]         ⏭  already exists
[email protected]       ⏭  already exists
[email protected]       ⏭  already exists
[email protected]               ⏭  already exists

... or all available binaries:

# at: syft/pkg/cataloger/binary/test-fixtures 

❯ make download-all
go run ./manager download

[email protected]                        ⏭  already exists
[email protected]                           ⏭  already exists
[email protected]                      ⏭  already exists
[email protected]                      ⏭  already exists
[email protected]                             ⏭  already exists
[email protected]                        ⏭  already exists
[email protected]                        ⏭  already exists
[email protected]                         ⏭  already exists
[email protected]                          ⏭  already exists
[email protected]_391                ⏭  already exists
[email protected]                       ⏭  already exists
[email protected]                      ⏭  already exists
[email protected]                          ⏭  already exists
[email protected]                          ⏭  already exists
[email protected]                          ⏭  already exists
[email protected]              ⏭  already exists
[email protected]                           ⏭  already exists
[email protected]_352-b08        ⏭  already exists
[email protected]              ⏭  already exists
[email protected]                         ⏭  already exists
postgres@15beta4                      ⏭  already exists
[email protected]                       ⏭  already exists
[email protected]               ⏭  already exists
[email protected]                   ⏭  already exists
[email protected]                   ⏭  already exists
[email protected]                    ⏭  already exists
[email protected]                   ⏭  already exists
[email protected]                    ⏭  already exists
[email protected]                   ⏭  already exists
[email protected]                    ⏭  already exists
[email protected]                        ⏭  already exists
[email protected]                    ⏭  already exists
[email protected]                    ⏭  already exists
[email protected]               ⏭  already exists
[email protected]                        ⏭  already exists
[email protected]                         ⏭  already exists
[email protected]           ⏭  already exists
[email protected]          ⏭  already exists
[email protected]                      ⏭  already exists
[email protected]         ⏭  already exists
[email protected]       ⏭  already exists
[email protected]       ⏭  already exists
[email protected]               ⏭  already exists
[email protected]                           ⏭  already exists
[email protected]                           ⏭  already exists
[email protected]                         ⏭  already exists

The add-snippet command helps select from managed binaries how to create snippets

Deferred work

add additional mechanisms for downloading binaries (e.g. simple download from a URL)

Signed-off-by: Christopher Phillips <[email protected]>

Signed-off-by: Alex Goodman <[email protected]>

.github/scripts/check_binary_fixture_size.sh

syft/pkg/cataloger/binary/test-fixtures/get-fixture-snippet.sh

Signed-off-by: Christopher Phillips <[email protected]>

Signed-off-by: Alex Goodman <[email protected]>

…pets Signed-off-by: Alex Goodman <[email protected]>

Signed-off-by: Alex Goodman <[email protected]>

…pets Signed-off-by: Alex Goodman <[email protected]>

Signed-off-by: Alex Goodman <[email protected]>

kzantow

🎉

syft/pkg/cataloger/binary/README.md

Signed-off-by: Alex Goodman <[email protected]>

…pets Signed-off-by: Alex Goodman <[email protected]>

* test: strip fixtures of any execution permissions Signed-off-by: Christopher Phillips <[email protected]> * chore: add lint check for large files Signed-off-by: Christopher Phillips <[email protected]> * add helper script to capture binary snippets Signed-off-by: Alex Goodman <[email protected]> * chore: update scripts and add new dir output for snippets Signed-off-by: Christopher Phillips <[email protected]> * test: update erlang test to new generated format Signed-off-by: Christopher Phillips <[email protected]> * test: update memcached to new generator pattern Signed-off-by: Christopher Phillips <[email protected]> * test: update openjdk to named version Signed-off-by: Christopher Phillips <[email protected]> * test: move openjdk lts to versioned folder Signed-off-by: Christopher Phillips <[email protected]> * test: rename unversioned java to versioned folders Signed-off-by: Christopher Phillips <[email protected]> * test: migrate bash fixture to new snippet workflow Signed-off-by: Christopher Phillips <[email protected]> * test: update script to size 600 bytes Signed-off-by: Christopher Phillips <[email protected]> * test: update go classifier to new snippet workflow Signed-off-by: Christopher Phillips <[email protected]> * test: move haproxy new new snippet Signed-off-by: Christopher Phillips <[email protected]> * test: add flatter haproxy example Signed-off-by: Christopher Phillips <[email protected]> * test: update tests to new pattern Signed-off-by: Christopher Phillips <[email protected]> * test: final version of snippet script Signed-off-by: Christopher Phillips <[email protected]> * [wip] download bin helpers Signed-off-by: Alex Goodman <[email protected]> * add manager for binary cataloger test fixtures Signed-off-by: Alex Goodman <[email protected]> * add remaining binary cataloger patterns and snippets Signed-off-by: Alex Goodman <[email protected]> * adjust gitignore to be more permissive to snippets Signed-off-by: Alex Goodman <[email protected]> * add rust darwin snippets Signed-off-by: Alex Goodman <[email protected]> * skip tests that are missing full binaries Signed-off-by: Alex Goodman <[email protected]> * address PR feedback Signed-off-by: Alex Goodman <[email protected]> * add tests for binary test fixture manager Signed-off-by: Alex Goodman <[email protected]> * highlight rows that do not have binaries or snippets Signed-off-by: Alex Goodman <[email protected]> * bump fixture limit to 1K (found exceptions when adding snippets) Signed-off-by: Alex Goodman <[email protected]> * add redis and postgres snippets Signed-off-by: Alex Goodman <[email protected]> * improve formating of fixture listing Signed-off-by: Alex Goodman <[email protected]> --------- Signed-off-by: Christopher Phillips <[email protected]> Signed-off-by: Alex Goodman <[email protected]> Co-authored-by: Alex Goodman <[email protected]>

spiffcs and others added 3 commits December 18, 2023 15:55

test: strip fixtures of any execution permissions

59eef37

Signed-off-by: Christopher Phillips <[email protected]>

chore: add lint check for large files

f194de3

Signed-off-by: Christopher Phillips <[email protected]>

add helper script to capture binary snippets

de87abf

Signed-off-by: Alex Goodman <[email protected]>

wagoodman reviewed Dec 18, 2023

View reviewed changes

.github/scripts/check_binary_fixture_size.sh Outdated Show resolved Hide resolved

wagoodman reviewed Dec 18, 2023

View reviewed changes

syft/pkg/cataloger/binary/test-fixtures/get-fixture-snippet.sh Outdated Show resolved Hide resolved

spiffcs added 10 commits December 19, 2023 11:16

chore: update scripts and add new dir output for snippets

2bf9971

Signed-off-by: Christopher Phillips <[email protected]>

test: update erlang test to new generated format

9099b98

Signed-off-by: Christopher Phillips <[email protected]>

test: update memcached to new generator pattern

a782fd2

Signed-off-by: Christopher Phillips <[email protected]>

test: update openjdk to named version

d289325

Signed-off-by: Christopher Phillips <[email protected]>

test: move openjdk lts to versioned folder

b945968

Signed-off-by: Christopher Phillips <[email protected]>

test: rename unversioned java to versioned folders

5e55530

Signed-off-by: Christopher Phillips <[email protected]>

test: migrate bash fixture to new snippet workflow

d1f3d11

Signed-off-by: Christopher Phillips <[email protected]>

test: update script to size 600 bytes

0b4f9ad

Signed-off-by: Christopher Phillips <[email protected]>

test: update go classifier to new snippet workflow

b92ccf8

Signed-off-by: Christopher Phillips <[email protected]>

test: move haproxy new new snippet

fd26fc1

Signed-off-by: Christopher Phillips <[email protected]>

spiffcs force-pushed the least-permissive-snippets branch from 33fcff0 to fd26fc1 Compare December 19, 2023 17:30

spiffcs and others added 5 commits December 19, 2023 12:36

test: add flatter haproxy example

7b86f23

Signed-off-by: Christopher Phillips <[email protected]>

test: update tests to new pattern

48cfc60

Signed-off-by: Christopher Phillips <[email protected]>

test: final version of snippet script

6d89a2b

Signed-off-by: Christopher Phillips <[email protected]>

[wip] download bin helpers

bab4142

Signed-off-by: Alex Goodman <[email protected]>

add manager for binary cataloger test fixtures

e516eb4

Signed-off-by: Alex Goodman <[email protected]>

wagoodman marked this pull request as draft December 21, 2023 23:07

wagoodman added 5 commits December 22, 2023 12:48

add remaining binary cataloger patterns and snippets

d849719

Signed-off-by: Alex Goodman <[email protected]>

adjust gitignore to be more permissive to snippets

8006e9d

Signed-off-by: Alex Goodman <[email protected]>

Merge remote-tracking branch 'origin/main' into least-permissive-snip…

a8b0f23

…pets Signed-off-by: Alex Goodman <[email protected]>

add rust darwin snippets

76619b2

Signed-off-by: Alex Goodman <[email protected]>

skip tests that are missing full binaries

8d9f1a8

Signed-off-by: Alex Goodman <[email protected]>

wagoodman changed the title ~~test: strip fixtures of any execution permissions~~ test: upgrade binary test fixtures management Jan 4, 2024

wagoodman added changelog-ignore Don't include this issue in the release changelog and removed changelog-ignore Don't include this issue in the release changelog labels Jan 4, 2024

Merge remote-tracking branch 'origin/main' into least-permissive-snip…

f422fc9

…pets Signed-off-by: Alex Goodman <[email protected]>

wagoodman changed the title ~~test: upgrade binary test fixtures management~~ Upgrade binary test fixtures management Jan 5, 2024

address PR feedback

ac1c35e

Signed-off-by: Alex Goodman <[email protected]>

wagoodman marked this pull request as ready for review January 5, 2024 17:06

wagoodman requested a review from a team January 5, 2024 17:06

kzantow approved these changes Jan 5, 2024

View reviewed changes

syft/pkg/cataloger/binary/README.md Outdated Show resolved Hide resolved

add tests for binary test fixture manager

1860b35

Signed-off-by: Alex Goodman <[email protected]>

wagoodman force-pushed the least-permissive-snippets branch from 7feff62 to 1860b35 Compare January 5, 2024 20:00

wagoodman added 5 commits January 5, 2024 15:06

highlight rows that do not have binaries or snippets

28aea2b

Signed-off-by: Alex Goodman <[email protected]>

bump fixture limit to 1K (found exceptions when adding snippets)

6d1a72f

Signed-off-by: Alex Goodman <[email protected]>

add redis and postgres snippets

4f0a5ae

Signed-off-by: Alex Goodman <[email protected]>

improve formating of fixture listing

83109d0

Signed-off-by: Alex Goodman <[email protected]>

Merge remote-tracking branch 'origin/main' into least-permissive-snip…

42c5308

…pets Signed-off-by: Alex Goodman <[email protected]>

wagoodman enabled auto-merge (squash) January 5, 2024 21:29

wagoodman merged commit 7182f5b into main Jan 5, 2024
10 checks passed

wagoodman deleted the least-permissive-snippets branch January 5, 2024 21:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade binary test fixtures management #2444

Upgrade binary test fixtures management #2444

spiffcs commented Dec 18, 2023 •

edited by wagoodman

Loading

kzantow left a comment

Upgrade binary test fixtures management #2444

Upgrade binary test fixtures management #2444

Conversation

spiffcs commented Dec 18, 2023 • edited by wagoodman Loading

Deferred work

kzantow left a comment

Choose a reason for hiding this comment

spiffcs commented Dec 18, 2023 •

edited by wagoodman

Loading