DAOS-16501 build: Add libsanitize #15105

knard38 · 2024-09-09T15:15:42Z

Description

Add scons build option SANITIZERS allowing to use the libasan. This new option takes a list of sanitizer tool such as: AddressSanitizer (i.e. -fsanitize=address), ThreadSanitizer(i.e. -fsanitize=thread), LeakSanitizer (i.e. -fsanitize=leak), etc.

A list of the available santizer tools and their compatibility could be found in the gcc man page.

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

TODO Required-githooks: true Signed-off-by: Cedric Koch-Hofer <[email protected]>

github-actions · 2024-09-09T15:15:59Z

Ticket title is 'LRZ: m02r01s10dao coredump - invalid free'
Status is 'Resolved'
Labels: 'lrz'
https://daosio.atlassian.net/browse/DAOS-16501

daosbuild1 · 2024-09-09T15:20:54Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/1/execution/node/328/log

daosbuild1 · 2024-09-09T15:21:03Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/1/execution/node/336/log

daosbuild1 · 2024-09-09T15:21:20Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/1/execution/node/396/log

daosbuild1 · 2024-09-09T15:22:52Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/1/execution/node/340/log

daosbuild1 · 2024-09-09T15:47:40Z

Test stage Unit Test with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/1/display/redirect

daosbuild1 · 2024-09-09T15:48:08Z

Test stage Unit Test bdev on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/1/display/redirect

daosbuild1 · 2024-09-09T15:48:09Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/1/display/redirect

daosbuild1 · 2024-09-09T15:48:19Z

Test stage Unit Test on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/1/display/redirect

daosbuild1 · 2024-09-09T15:55:16Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/1/display/redirect

utils/rpms/daos.spec

…/daos-16501 Required-githooks: true

Miscelleneaous fixe\s. Required-githooks: true Signed-off-by: Cedric Koch-Hofer <[email protected]>

daosbuild1 · 2024-09-09T18:42:05Z

Test stage Unit Test with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/2/display/redirect

daosbuild1 · 2024-09-09T18:42:13Z

Test stage Unit Test bdev on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/2/display/redirect

daosbuild1 · 2024-09-09T18:42:19Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/2/display/redirect

daosbuild1 · 2024-09-09T18:42:56Z

Test stage Unit Test on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/2/display/redirect

daosbuild1 · 2024-09-09T18:44:53Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/2/display/redirect

daosbuild1 · 2024-09-09T22:36:54Z

Test stage Unit Test with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/3/display/redirect

daosbuild1 · 2024-09-09T22:36:56Z

Test stage Unit Test bdev on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/3/display/redirect

daosbuild1 · 2024-09-09T22:37:18Z

Test stage Unit Test on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/3/display/redirect

daosbuild1 · 2024-09-09T22:37:23Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/3/display/redirect

daosbuild1 · 2024-09-09T22:39:30Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/3/display/redirect

daosbuild1 · 2024-09-10T07:07:17Z

Test stage Unit Test with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/4/display/redirect

daosbuild1 · 2024-09-10T07:07:27Z

Test stage Unit Test bdev on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/4/display/redirect

daosbuild1 · 2024-09-10T07:07:31Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/4/display/redirect

daosbuild1 · 2024-09-10T07:07:33Z

Test stage Unit Test on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/4/display/redirect

daosbuild1 · 2024-09-10T07:10:07Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15105/4/display/redirect

daosbuild1 · 2024-09-10T09:34:46Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/6/execution/node/360/log

daosbuild1 · 2025-01-07T13:58:24Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/38/execution/node/349/log

daosbuild1 · 2025-01-07T13:58:57Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/38/execution/node/341/log

daosbuild1 · 2025-01-07T13:59:54Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/38/execution/node/339/log

Fixing invalid compilation errors reported by gcc on el9. Signed-off-by: Cedric Koch-Hofer <[email protected]>

daosbuild1 · 2025-01-07T14:05:53Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/40/execution/node/340/log

daosbuild1 · 2025-01-07T14:14:56Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/41/execution/node/352/log

daosbuild1 · 2025-01-07T14:18:48Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/41/execution/node/308/log

daosbuild1 · 2025-01-07T14:20:10Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/41/execution/node/355/log

daosbuild1 · 2025-01-07T14:23:59Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/41/execution/node/329/log

Fixing invalid compilation errors reported by gcc on el9. Signed-off-by: Cedric Koch-Hofer <[email protected]>

daosbuild1 · 2025-01-07T14:43:12Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/43/execution/node/358/log

daosbuild1 · 2025-01-07T14:48:07Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/43/execution/node/359/log

daosbuild1 · 2025-01-07T14:48:24Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/43/execution/node/345/log

mchaarawi · 2025-01-07T14:49:20Z

TBH, I have not done some performance test as my initial aim was not to use it in production but for helping to identify thread doing some memory corruptions (such as buffer overflow).

yes for sure this is just for dev, but when i tried enabling sanitizer on the server side a while ago, the server just hangs. maybe i did something wrong and it works for you.
does this work also for the dependencies? like does it also report things for mercury for example?

Fixing invalid compilation errors reported by gcc on el9. Signed-off-by: Cedric Koch-Hofer <[email protected]>

daosbuild1 · 2025-01-07T14:56:02Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/45/execution/node/346/log

daosbuild1 · 2025-01-07T15:00:51Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/45/execution/node/344/log

daosbuild1 · 2025-01-07T15:01:52Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/45/execution/node/345/log

Fixing invalid compilation errors reported by gcc on el9. Signed-off-by: Cedric Koch-Hofer <[email protected]>

daosbuild1 · 2025-01-07T15:04:00Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/45/execution/node/337/log

daosbuild1 · 2025-01-07T15:13:18Z

Test stage Build RPM on EL 9 completed with status UNSTABLE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/47/execution/node/341/log

daosbuild1 · 2025-01-07T15:15:00Z

Test stage Build RPM on EL 8 completed with status UNSTABLE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/47/execution/node/353/log

daosbuild1 · 2025-01-07T15:16:13Z

Test stage Build RPM on Leap 15.5 completed with status UNSTABLE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15105/47/execution/node/354/log

knard38 · 2025-01-07T16:06:46Z

TBH, I have not done some performance test as my initial aim was not to use it in production but for helping to identify thread doing some memory corruptions (such as buffer overflow).

yes for sure this is just for dev, but when i tried enabling sanitizer on the server side a while ago, the server just hangs. maybe i did something wrong and it works for you. does this work also for the dependencies? like does it also report things for mercury for example?

I was not planning to manage the direct DAOS dependencies such as mercury.
I have planned look at it in a second time... if I am able to obtain an acceptable result with this PR 🤞

liw · 2025-01-08T00:55:42Z

@mchaarawi, ASan requires an LD_LIBRARY_PATH for engine modules and a larger default Argobots stack size in your engine YAML file(s). Without them, engines either fail to start because they can't find librdb.so, or overrun stacks during pool creation. The performance has been amazing compared to Valgrind---I barely notice a difference when running daos_test on Wolf.

Mercury already has the support; @soumagne must have already fixed the issues that are easily found. Same for Argobots.

I've used ASan with DAOS plus daos_test in a rush before 2.6.1 and managed to fix the following product issues.

0e52fa5 DAOS-16458 object: fix invalid DRAM access in obj_bulk_transfer (DAOS-16458 object: fix invalid DRAM access in obj_bulk_transfer #15026)
226e283 DAOS-16251 object: Fix obj_ec_singv_split overflow (DAOS-16466 object: Fix obj_ec_singv_split overflow #15045)
663237c DAOS-16251 mgmt: Fix use-after-free in pool_list (DAOS-16251 mgmt: Fix use-after-free in pool_list #15014)

And the following test issues.

0dfe144 DAOS-16251 tests: Fix various memory issues (DAOS-16251 tests: Fix various memory issues #15147)
407199f DAOS-16251 dtx: Fix dtx_req_send user-after-free (DAOS-16251 dtx: Fix dtx_req_send user-after-free #15035)
2ac6f52 DAOS-16251 tests: Fix various buffer overflows (DAOS-16251 tests: Fix various buffer overflows #15003)

In that process, @knard38 and I discovered each other's work on ASan. We planned to land the support (this PR), present an introduction, and see if this is robust enough for certain regular CI runs. Beyond that, our future plan is to look into the memory leak reports (ignored so far), TSan, etc.

mchaarawi · 2025-01-08T01:02:04Z

@mchaarawi, ASan requires an LD_LIBRARY_PATH for engine modules and a larger default Argobots stack size in your engine YAML file(s). Without them, engines either fail to start because they can't find librdb.so, or overrun stacks during pool creation. The performance has been amazing compared to Valgrind---I barely notice a difference when running daos_test on Wolf.

Mercury already has the support; @soumagne must have already fixed the issues that are easily found. Same for Argobots.

I've used ASan with DAOS plus daos_test in a rush before 2.6.1 and managed to fix the following product issues.

0e52fa5 DAOS-16458 object: fix invalid DRAM access in obj_bulk_transfer (DAOS-16458 object: fix invalid DRAM access in obj_bulk_transfer #15026)

226e283 DAOS-16251 object: Fix obj_ec_singv_split overflow (DAOS-16466 object: Fix obj_ec_singv_split overflow #15045)

663237c DAOS-16251 mgmt: Fix use-after-free in pool_list (DAOS-16251 mgmt: Fix use-after-free in pool_list #15014)

And the following test issues.

0dfe144 DAOS-16251 tests: Fix various memory issues (DAOS-16251 tests: Fix various memory issues #15147)

407199f DAOS-16251 dtx: Fix dtx_req_send user-after-free (DAOS-16251 dtx: Fix dtx_req_send user-after-free #15035)

2ac6f52 DAOS-16251 tests: Fix various buffer overflows (DAOS-16251 tests: Fix various buffer overflows #15003)

In that process, @knard38 and I discovered each other's work on ASan. We planned to land the support (this PR), present an introduction, and see if this is robust enough for certain regular CI runs. Beyond that, our future plan is to look into the memory leak reports (ignored so far), TSan, etc.

thanks for the detailed reply!
you obviously spent more time getting it to work than i did :-)

liw · 2025-01-08T01:08:13Z

thanks for the detailed reply! you obviously spent more time getting it to work than i did :-)

You're very welcome to work with Cedric and I, if you like. We still don't know whether ASan will turn out to be robust enough or not, because I believe Argobots should cause some false positives, but I've seen zero, which puzzles me.

Fixing invalid compilation errors reported by gcc on el9. Signed-off-by: Cedric Koch-Hofer <[email protected]>

…/daos-16501

Fix libasan version for ubuntu image used with GHA Fix invalid copyright Signed-off-by: Cedric Koch-Hofer <[email protected]>

DAOS-16501 build: Add libsanitize

2d97760

TODO Required-githooks: true Signed-off-by: Cedric Koch-Hofer <[email protected]>

brianjmurrell reviewed Sep 9, 2024

View reviewed changes

utils/rpms/daos.spec Outdated Show resolved Hide resolved

kanard38 added 2 commits September 9, 2024 18:10

Merge remote-tracking branch 'origin/master' into ckochhof/dev/master…

8d233f5

…/daos-16501 Required-githooks: true

DAOS-16501 build: Add libsanitize

ce48861

Miscelleneaous fixe\s. Required-githooks: true Signed-off-by: Cedric Koch-Hofer <[email protected]>

DAOS-16501 build: Add libsanitize

df2675b

Fixing invalid compilation errors reported by gcc on el9. Signed-off-by: Cedric Koch-Hofer <[email protected]>

DAOS-16501 build: Add libsanitize

6b25da9

Fixing invalid compilation errors reported by gcc on el9. Signed-off-by: Cedric Koch-Hofer <[email protected]>

DAOS-16501 build: Add libsanitize

192f66d

Fixing invalid compilation errors reported by gcc on el9. Signed-off-by: Cedric Koch-Hofer <[email protected]>

DAOS-16501 build: Add libsanitize

9ef5a08

Fixing invalid compilation errors reported by gcc on el9. Signed-off-by: Cedric Koch-Hofer <[email protected]>

kanard38 added 3 commits January 8, 2025 06:44

DAOS-16501 build: Add libsanitize

2566572

Fixing invalid compilation errors reported by gcc on el9. Signed-off-by: Cedric Koch-Hofer <[email protected]>

Merge remote-tracking branch 'origin/master' into ckochhof/dev/master…

0011ba7

…/daos-16501

DAOS-16501 build: Add libsanitize

d81556c

Fix libasan version for ubuntu image used with GHA Fix invalid copyright Signed-off-by: Cedric Koch-Hofer <[email protected]>

DAOS-16501 build: Add libsanitize #15105

Are you sure you want to change the base?

DAOS-16501 build: Add libsanitize #15105

Conversation

knard38 commented Sep 9, 2024 • edited Loading

Description

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented Sep 9, 2024 • edited Loading

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 9, 2024

daosbuild1 commented Sep 10, 2024

daosbuild1 commented Sep 10, 2024

daosbuild1 commented Sep 10, 2024

daosbuild1 commented Sep 10, 2024

daosbuild1 commented Sep 10, 2024

daosbuild1 commented Sep 10, 2024

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

mchaarawi commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

daosbuild1 commented Jan 7, 2025

knard38 commented Jan 7, 2025

liw commented Jan 8, 2025 • edited Loading

mchaarawi commented Jan 8, 2025

liw commented Jan 8, 2025

knard38 commented Sep 9, 2024 •

edited

Loading

github-actions bot commented Sep 9, 2024 •

edited

Loading

liw commented Jan 8, 2025 •

edited

Loading