feat(infiniband): use events db to persist ibstat check thresholds

[gyuho]

• test(events): test purge disable
• feat(infiniband): use events db to persist ibstat check thresholds

[codecov]

Codecov Report

Attention: Patch coverage is 80.32787% with 12 lines in your changes missing coverage. Please review.

Project coverage is 23.01%. Comparing base (ab3785a) to head (1a5499a).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
...celerator/nvidia/infiniband/threshold/threshold.go	80.32%	8 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #370      +/-   ##
==========================================
+ Coverage   22.94%   23.01%   +0.07%     
==========================================
  Files         289      290       +1     
  Lines       25883    25944      +61     
==========================================
+ Hits         5938     5971      +33     
- Misses      19335    19364      +29     
+ Partials      610      609       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[eahydra]

It is better to use the write lock for the entire function.

[eahydra]

Do we support syncing node hardware spec like ib threshold from server? /cc @cardyok

[gyuho]

Yes

gpud/internal/session/serve.go

Lines 170 to 183 in 9fe57e6

	case "updateConfig":
	if payload.UpdateConfig != nil {
	for componentName, value := range payload.UpdateConfig {
	log.Logger.Infow("Update config received for component", "component", componentName, "config", value)
	switch componentName {
	case nvidia_infiniband_id.Name:
	var updateCfg nvidia_infiniband.ExpectedPortStates
	if err := json.Unmarshal([]byte(value), &updateCfg); err != nil {
	log.Logger.Warnw("failed to unmarshal update config", "error", err)
	} else {
	nvidia_infiniband.SetDefaultExpectedPortStates(updateCfg)
	}
	default:

(Current issue is this update will reset on GPUd restart)

[eahydra]

I'm curious that why not store these data to a separate file which stores all node hardware spec or node related thresholds?

[gyuho]

Yes, we can do that as well (separate table might be easier, since we already have all the persistence via sqlite file)

[eahydra]

OK, but it would be better to use a text file to store this configuration, as it would be more maintainable.

[gyuho]

Closing for now