feat: find purls for affected packages

[dejanb]

This is db implementation of feature needed for #994.

I would go with it to unlock work on UI (#1043) and see if and how to replace same functionality with graph in the future.

I'm keeping #1014 draft PR as a placeholder for this future work

[carlosthe19916]

@dejanb Thanks for the PR. I've just tested it.

• I imported all data from https://github.com/trustification/trustify/tree/main/etc/datasets/ds1
• If I see the Quarkus SBOM Using the UI I see the following result

• You see that the expanded area of the first row contains a full PURL therefore I can link the purl to the Package Details page in the UI
• The second row does not have a purl and only name. I guess this PR will help identifying some Purls but not all of them right? there will still be some package that won't contains IDs?

Also I tested with all the other SBOMS from https://github.com/trustification/trustify/tree/main/etc/datasets/ds1 and ALL sboms, except the quarkus-sbom are not able to generate the IDs. It looks like this PR enhanced things but only for the quarkus-sbom and not for the rest of them. But that might just be that the other SBOMs do not have enough data to achieve and make the magic?

[ctron]

We should be careful with terminology here and how we use data. "packages" (aka components) only make sense in the context of an SBOM. The same is true for "names of packages". PURLs (as well as CPEs) are different. Packages may have a PURL or CPE, which has a meaning outside of the SBOM.

At Red Hat, we may choose to name packages using names that make sense externally, but that's just our decision.

[dejanb]

@ctron This is the logic from v1 that made it possible to find these purls. It is the "best effort" kind of thing so it shouldn't affect others (non-RH advisories).
As for the performance, that's where in-mem graph search instead of db can come to play if needed (and/or is ready)

@carlosthe19916 so the large number of purls we don't find is consistent with v1, but in that place we don't show the vulnerability. Maybe it's the same logic we should apply here as well. We need to think more about what's more useful: showing that sbom is vulnerable although we can't find the component inside or not showing it to be affected in that case.

Anyways, I will come back to this next week. Please feel free to comment, change anything around this in the meantime.

[dejanb]

@carlosthe19916 @ctron @JimFuller-RedHat So scrapping the previous effort, this PR now tries to solve this all in a single query. As a result it will return only results for matching packages, but that's exactly v1 parity so that should good for now. Performances are also reasonable on the small dataset, but we need to check that further.

[carlosthe19916]

@dejanb sorry for my late reply.

I uploaded the files from D3 and then using the current UI go to the details of an SBOM page. and I see the following message in the log of the backend:

2024-12-04T12:04:34.055378Z  WARN ThreadId(49) sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT DISTINCT ON (\"product_status\".\"context_cpe_id\", …" db.statement="\n\nSELECT\n  DISTINCT ON (\n    \"product_status\".\"context_cpe_id\",\n    \"product_status\".\"status_id\",\n    \"product_status\".\"package\",\n    \"product_status\".\"vulnerability_id\"\n  ) \"advisory\".\"id\" AS \"advisory$id\",\n  \"advisory\".\"identifier\" AS \"advisory$identifier\",\n  \"advisory\".\"version\" AS \"advisory$version\",\n  \"advisory\".\"document_id\" AS \"advisory$document_id\",\n  \"advisory\".\"deprecated\" AS \"advisory$deprecated\",\n  \"advisory\".\"issuer_id\" AS \"advisory$issuer_id\",\n  \"advisory\".\"published\" AS \"advisory$published\",\n  \"advisory\".\"modified\" AS \"advisory$modified\",\n  \"advisory\".\"withdrawn\" AS \"advisory$withdrawn\",\n  \"advisory\".\"title\" AS \"advisory$title\",\n  \"advisory\".\"labels\" AS \"advisory$labels\",\n  \"advisory\".\"source_document_id\" AS \"advisory$source_document_id\",\n  \"vulnerability\".\"id\" AS \"vulnerability$id\",\n  \"vulnerability\".\"title\" AS \"vulnerability$title\",\n  \"vulnerability\".\"reserved\" AS \"vulnerability$reserved\",\n  \"vulnerability\".\"published\" AS \"vulnerability$published\",\n  \"vulnerability\".\"modified\" AS \"vulnerability$modified\",\n  \"vulnerability\".\"withdrawn\" AS \"vulnerability$withdrawn\",\n  \"vulnerability\".\"cwes\" AS \"vulnerability$cwes\",\n  \"base_purl\".\"id\" AS \"base_purl$id\",\n  \"base_purl\".\"type\" AS \"base_purl$type\",\n  \"base_purl\".\"namespace\" AS \"base_purl$namespace\",\n  \"base_purl\".\"name\" AS \"base_purl$name\",\n  \"versioned_purl\".\"id\" AS \"versioned_purl$id\",\n  \"versioned_purl\".\"base_purl_id\" AS \"versioned_purl$base_purl_id\",\n  \"versioned_purl\".\"version\" AS \"versioned_purl$version\",\n  \"qualified_purl\".\"id\" AS \"qualified_purl$id\",\n  \"qualified_purl\".\"versioned_purl_id\" AS \"qualified_purl$versioned_purl_id\",\n  \"qualified_purl\".\"qualifiers\" AS \"qualified_purl$qualifiers\",\n  \"qualified_purl\".\"purl\" AS \"qualified_purl$purl\",\n  \"sbom_package\".\"sbom_id\" AS \"sbom_package$sbom_id\",\n  \"sbom_package\".\"node_id\" AS \"sbom_package$node_id\",\n  \"sbom_package\".\"version\" AS \"sbom_package$version\",\n  \"sbom_node\".\"sbom_id\" AS \"sbom_node$sbom_id\",\n  \"sbom_node\".\"node_id\" AS \"sbom_node$node_id\",\n  \"sbom_node\".\"name\" AS \"sbom_node$name\",\n  \"status\".\"id\" AS \"status$id\",\n  \"status\".\"slug\" AS \"status$slug\",\n  \"status\".\"name\" AS \"status$name\",\n  \"status\".\"description\" AS \"status$description\",\n  \"cpe\".\"id\" AS \"cpe$id\",\n  \"cpe\".\"part\" AS \"cpe$part\",\n  \"cpe\".\"vendor\" AS \"cpe$vendor\",\n  \"cpe\".\"product\" AS \"cpe$product\",\n  \"cpe\".\"version\" AS \"cpe$version\",\n  \"cpe\".\"update\" AS \"cpe$update\",\n  \"cpe\".\"edition\" AS \"cpe$edition\",\n  \"cpe\".\"language\" AS \"cpe$language\"\nFROM\n  \"product_version\"\n  INNER JOIN \"sbom\" ON \"sbom\".\"sbom_id\" = \"product_version\".\"sbom_id\"\n  LEFT JOIN \"product\" ON \"product_version\".\"product_id\" = \"product\".\"id\"\n  LEFT JOIN \"cpe\" ON \"product\".\"cpe_key\" = \"cpe\".\"product\"\n  JOIN \"product_status\" ON \"cpe\".\"id\" = \"product_status\".\"context_cpe_id\"\n  JOIN \"status\" ON \"product_status\".\"status_id\" = \"status\".\"id\"\n  JOIN \"advisory\" ON \"product_status\".\"advisory_id\" = \"advisory\".\"id\"\n  JOIN \"vulnerability\" ON \"product_status\".\"vulnerability_id\" = \"vulnerability\".\"id\"\n  JOIN \"sbom_node\" ON \"sbom\".\"sbom_id\" = \"sbom_node\".\"sbom_id\"\n  JOIN \"sbom_package\" ON \"sbom_node\".\"sbom_id\" = \"sbom_package\".\"sbom_id\"\n  AND \"sbom_node\".\"node_id\" = \"sbom_package\".\"node_id\"\n  JOIN \"sbom_package_purl_ref\" ON \"sbom_package\".\"sbom_id\" = \"sbom_package_purl_ref\".\"sbom_id\"\n  AND \"sbom_package\".\"node_id\" = \"sbom_package_purl_ref\".\"node_id\"\n  JOIN \"qualified_purl\" ON \"sbom_package_purl_ref\".\"qualified_purl_id\" = \"qualified_purl\".\"id\"\n  JOIN \"versioned_purl\" ON \"qualified_purl\".\"versioned_purl_id\" = \"versioned_purl\".\"id\"\n  JOIN \"base_purl\" ON \"versioned_purl\".\"base_purl_id\" = \"base_purl\".\"id\"\nWHERE\n  \"sbom\".\"sbom_id\" = $1\n  AND (\n    \"product_status\".\"package\" IS NULL\n    OR \"product_status\".\"package\" = $2\n    OR \"product_status\".\"package\" LIKE CONCAT(\"base_purl\".\"namespace\", $3, \"base_purl\".\"name\")\n    OR \"product_status\".\"package\" = \"base_purl\".\"name\"\n  )\nORDER BY\n  \"product_status\".\"context_cpe_id\" ASC,\n  \"product_status\".\"status_id\" ASC,\n  \"product_status\".\"package\" ASC,\n  \"product_status\".\"vulnerability_id\" ASC\n" rows_affected=22 rows_returned=22 elapsed=1.469311664s elapsed_secs=1.4693116640000001 slow_threshold=1s

So there are mentions to query: slow statement: execution time

I am just reporting it, I am not saying it is wrong. Just wanted you to be aware of it in any case :)

• On the other hand, I don't know why. When I use this PR branch (backend) and then import the D1 or D3 files then I cannot see vulnerabilities in the UI (in the SBOM Details page - VUlnerabilities tab) for sboms different from "qurakus-sbom". I mean:
  • using this PR, the only sbom that has vulnerabilities with the status "affected" is the "quarkus-sbom"
  • using the main branch, all sboms have at least one vulnerability with status "affected".

I don't know if that is thing to consider or nor. I just wanted to point that out.

To me, as the person on the client side, I am happy with the result. I can approve this if my review is required :) . I can not judge the code :)

[dejanb]

@carlosthe19916 Thanks Carlos. Those both are valid and expected things.

The query is a bit slowish, but I can think we can further improve it.
As for the results, this is now in line with v1 parity. Currently in the main we show vulnerabilities for packages we can't find in the sbom. I think the new behaviour is correct.

[carlosthe19916]

LGTM! From the client side point of view. I'll let the others decide the rest.

[JimFuller-RedHat]

yes the query will need some optimisation, but for main LGTM.