Implement OpenScanHub initial reporting

[lbarcziova]

• in fedmsg repo, process the messages about OSH task states (see hub: add support for fedora messaging openscanhub/openscanhub#299 )
• create new event class for that and implement parsing of these
• create a model for a scan
• store the task in the DB somewhere here and instead of successful status there, report in-progress
• create a handler to report the results for the new event
  • obtain the task in our DB, update the state and report it to user
  • for now, just forward the URLs from the message (=> no additional processing)
  • take into consideration Add configurations for CI to fail on OSH scan failures and new findings #2515
  • by default don't fail the CI, have a look also into action_required conclusion from here, whether that could be used, or use neutral/successful
  • report neutral status for failed, cancelled or interrupted builds

---

Part of #2516

[siteshwar]

• by default don't fail the CI, have a look also into action_required conclusion from here, whether that could be used, or use neutral/successful
• report neutral status for failed, cancelled or interrupted builds

Please also see Code scanning results check failures.

[siteshwar]

by default don't fail the CI, have a look also into action_required conclusion from here, whether that could be used, or use neutral/successful

If you want to find out if new findings were detected, follow these steps:

• Download the added.js file.
• Run csgrep added.js and check if output is non-empty.
• If output is non-empty, it means there were new findings by the analzyers. Set the CI to action required state.
• Otherwise pass the CI.

EDIT: @kdudka Is there any other way to count number of findings in the reports?

[kdudka]

@siteshwar For simple scans, we can check whether the scan-results-summary.txt file is empty or not. Unfortunately, for differential scans, we do not provide such a file that would summarize the added.* results. We can still check whether added.err is empty or not.

[siteshwar]

And just to keep everybody in sync. The messages from OpenScanHub can be seen through:

fedora-messaging --conf /etc/fedora-messaging/fedora.toml consume --routing-key "org.fedoraproject.prod.openscanhub.task.*

[kdudka]

@siteshwar I can see that the URLs provided in Body: of messages about finished scans use the http:// scheme instead of https://. Consequently the scan results URLs, when used directly, return 302 Found instead of the real data.

[kdudka]

The problem may actually be more generic. I got an e-mail notification and the task URL also contained http:// URL. Something must be misconfigured in the Fedora deployment because the internal OSH instances use https:// URLs.

[majamassarini]

Hi! I want just to let you know I am starting working on this.

[siteshwar]

The problem may actually be more generic. I got an e-mail notification and the task URL also contained http:// URL. Something must be misconfigured in the Fedora deployment because the internal OSH instances use https:// URLs.

I would investigate it separately through openscanhub/fedora-infra#73

[siteshwar]

There is a confusion about task state changes in OSH and how they should be reflected in the reporting. I would summarize it again for clarification:

• When a task is submitted to OSH, but has not started, it is in FREE state. The status in Packit should say, it is waiting to be started.

• When a worker picks up the task and starts running it, it moves into OPEN state. OSH would emit this message with task id in the body.

org.fedoraproject.prod.openscanhub.task.open

Packit reporting should say the task is running.

• If the scan fails to finish successfully, there would be one of these messages with task id in the body:

org.fedoraproject.prod.openscanhub.task.cancel
org.fedoraproject.prod.openscanhub.task.fail
org.fedoraproject.prod.openscanhub.task.interrupt

Packit reporting should put the CI in neutral state.

• If the scan finishes successfully, you would get org.fedoraproject.prod.openscanhub.task.finish with url to added and removed findings in JSON format along with task id. Follow the steps from previous comment to process these results.

[siteshwar]

@lachmanfrantisek @lbarcziova

I am following up from #2580 (comment)

Please tell me exactly the message topics you would like to receive on Packit side and what should be the content of the message body.

[lbarcziova]

hi @siteshwar ! I talked about that with @majamassarini and am aligned with the way how she proposed it, so having 2 types of messages: one for start/in-progress task and one for finished, while there should be the state or result attribute describing the actual status.

[siteshwar]

openscanhub/openscanhub#301

[siteshwar]

openscanhub/openscanhub#301

These changes are in production now. Example messages:

[INFO fedora_messaging.twisted.consumer] Successfully consumed message from topic org.fedoraproject.prod.openscanhub.task.started (message id f6561b12-a1d3-4567-9f72-2ef3b941210a)
[INFO fedora_messaging.twisted.consumer] Consuming message from topic org.fedoraproject.prod.openscanhub.task.started (message id d1a83e8e-c9d0-4a1f-adad-562bc38026a4)
Id: d1a83e8e-c9d0-4a1f-adad-562bc38026a4
Topic: org.fedoraproject.prod.openscanhub.task.started
Headers: {
    "fedora_messaging_schema": "base.message",
    "fedora_messaging_severity": 20,
    "priority": 0,
    "sent-at": "2024-10-18T15:44:43+00:00",
    "x-received-from": [
        {
            "cluster-name": "[email protected]",
            "exchange": "amq.topic",
            "redelivered": false,
            "uri": "amqps://rabbitmq01.iad2.fedoraproject.org/%2Fpubsub"
        }
    ]
}
Body: {
    "task_id": 18203
}
[INFO fedora_messaging.twisted.consumer] Successfully consumed message from topic org.fedoraproject.prod.openscanhub.task.started (message id d1a83e8e-c9d0-4a1f-adad-562bc38026a4)
[INFO fedora_messaging.twisted.consumer] Consuming message from topic org.fedoraproject.prod.openscanhub.task.finished (message id 29b7d83b-2a79-46ab-8b9a-5b4f36794daa)
Id: 29b7d83b-2a79-46ab-8b9a-5b4f36794daa
Topic: org.fedoraproject.prod.openscanhub.task.finished
Headers: {
    "fedora_messaging_schema": "base.message",
    "fedora_messaging_severity": 20,
    "priority": 0,
    "sent-at": "2024-10-18T15:45:01+00:00",
    "x-received-from": [
        {
            "cluster-name": "[email protected]",
            "exchange": "amq.topic",
            "redelivered": false,
            "uri": "amqps://rabbitmq01.iad2.fedoraproject.org/%2Fpubsub"
        }
    ]
}
Body: {
    "added.js": "http://openscanhub.fedoraproject.org/task/18198/log/added.js?format=raw",
    "fixed.js": "http://openscanhub.fedoraproject.org/task/18198/log/fixed.js?format=raw",
    "scan-results.js": "http://openscanhub.fedoraproject.org/task/18198/log/crun-1.17-1.20241018152736619857.pr1579.39.g30fda6b/scan-results.js?format=raw",
    "status": "success",
    "task_id": 18198
}