Importer configuration design

[carlosthe19916]

I'd like to make one point in regards of the current status of the Importers (REST API)

Could we consider joining the osv and cve importers?

We currently have different configurations osv, cve based on the format of the file. But I think we should consider having the importer configurations based on the protocol/source-type rather than the format file. Let me expand on this:

This is how we currently configure a osv and a cve importer:

[
    {
        "name": "osv-oss-fuzz",
        "configuration": {
            "osv": {
                "disabled": true,
                "period": "5m",
                "description": "OSS-Fuzz vulnerabilities",
                "source": "https://github.com/google/oss-fuzz-vulns",
                "path": "vulns"
            }
        },
        "state": "waiting",
        "lastChange": "2024-06-03T08:24:29.949501Z"
    },
    {
        "name": "cve",
        "configuration": {
            "cve": {
                "disabled": true,
                "period": "5m",
                "description": "CVE List V5",
                "source": "https://github.com/CVEProject/cvelistV5"
            }
        },
        "state": "waiting",
        "lastChange": "2024-06-05T07:59:52.158285Z"
    }
]

Why don't we just have an importer for Git (Github) repositories? so rather than having configuration.osv and configuration.cve we replace it by configuration.git?

So the previous 2 importers could be defined as something like (I removed some fields to make the idea clearer):

[
    {
        "name": "osv-oss-fuzz",
        "configuration": {
            "git": {                
                "source": "https://github.com/google/oss-fuzz-vulns",
                "path": "vulns"
            }
        },     
    },
    {
        "name": "cve",
        "configuration": {
            "git": {              
                "source": "https://github.com/CVEProject/cvelistV5"
            }
        }
    }
]

• The advantage of doing it is that we can configure any git repository.
• Also, a single git repository could have mixed files "osv" and "cve". This should be possible as, as far as I know, the current Upload REST endpoint accepts a JSON file and it infers its format; therefore, technically a single git repository could have multiple formats stored

What do you think about it guys?

[carlosthe19916]

Expanding the original idea, I would even dare to suggest to also simplify the body structure of an importer. The current body follows this pattern:

{
  "name": "name",
  "configuration": {
    "sbom": {},
    "csaf": {},
    "osv": {},
    "cve": {}
  }
}

But it could be simplified as:

{
  "name": "name",
  "type": "sbom|csaf|osv|cve",
  "configuration": {}
}

I believe there are proper DTOs in place now, so the JSON response generated by the server should not necessarily be limited by how the Rust language defines enums

[ctron]

Why don't we just have an importer for Git (Github) repositories? so rather than having configuration.osv and configuration.cve we replace it by configuration.git?

Because they are actually different. The CVE importer allows to filter by year, OSV doesn't. OSV can have paths. For CVE that doesn't make sense. There's also some expectation/logic on which files get uploaded and which get ignored, based on the importer.

As we do have a JSON schema for this, I think we should just stick to that. And I understand that sometimes we need to double check the schema is generated correctly. But believe that's the best approach.

When it comes to serializing the enum that's backing this, we currently use the default serde representation (externally tagged: https://serde.rs/enum-representations.html#externally-tagged). We could use the "internally tagged" alternative too if there's a benefit to this. However, this would simpley move the discriminator to a different level. If it's easier for JS to handle it that way, we can change it.