Avoid cron and external processes where possible

[bobmcwhirter]

In an effort to reduce the number of processes, should we aim towards the goal of a single trustd with configuration, and when needed it internally has loops doing walker-y types of things?

Hopefully we can reach a point of simply being able to start and stop The Entire System, instead of remember to also run crawlers, walkers, importers. They become intrinsic into The System.

[jcrossley3]

I like this goal. If the command line arguments become too much, we might consider a config file at some point.

[ctron]

I am not having an answer here. But I bounce back and forth between config file and env-var/args.

Yes, the env-var approach is limited. The config-file approach has a huge downside that you can't inject simple values when deploying (like the chart provides 95% of the content, 5% get overridden by the user).

The idea of mixing config files with env-vars/args is a errors prone and complex beast in both trunk and zincati.

I would really wish there would be a better solution.

[bobmcwhirter]

Hey, configuration sounds like perhaps a good topic for another discussion.

[bobmcwhirter]

seeded: #34

[mrizzi]

Re loops usage, on trustification project, I found interesting collectorist's approach for managing packages to be evaluated for vulnerabilities with a pattern I think suitable also for trustify.
This approach could manage "communications" between trustify's components in a consistent way and, at the same time, being able to provide, in each moment, an always up-to-date status of each resource that trustify manages (which is a requirement).

[ctron]

I think the concept we had in Drogue with Ulf super-binary worked quite well. Collecting futures form "main" async functions and then calling join (or select?) on them. All those processed had been loops, or serve, run style functions.

[mrizzi]

It looks very interesting. I'm not that familiar with Drogue, @ctron could you share a reference to this super-binary please? I'm eager to have a look and learn it.

[ctron]

https://github.com/drogue-iot/drogue-cloud/blob/main/server/src/main.rs … it's also super big. But it also covers things like Kafka, Keycloak, and database. Maybe see it as a source of inspiration.

[mrizzi]

So, just for my benefit, after #49, does it mean that we should have a trustify-super-binary dedicated project for delivering single tool to run this locally out-of-the-box?

[bobmcwhirter]

I think we're (or at least I am) considering that to be the trustd binary crate thing.

[ctron]

So we would need some command in trustd for doing that. Like that?

trustd server # runs the HTTP API server
trustd importer(s) # runs the importer loops
trustd all # run all components

[carlosthe19916]

My two cents:
I believe Trustify should have one and only one command: trustd server. No importers, no extra commands, nothing else.
The REST API should rule everything, from starting, scheduling, monitoring, etc, tasks

My comments go in the direction that I think the current approach: trustd server + trustd importer is not ideal. But it is an opinion, not an affirmation at all.

What if we want to import data from https://access.redhat.com/security/data/csaf/v2/advisories/ ?

• trustd server starts the server
• The UI or a terminal initiates a REST POST request /importers/advisories with the body:

{
   "kind": "HTTP",
   "url": "https://access.redhat.com/security/data/csaf/v2/advisories/"
}

• trustd starts the import process based on the Body ingested in the POST Method

What if we want to import data from a FTP or SFTP?

POST /importers/advisories

{
   "kind": "FTP | SFTP",
   "url": "https://myServer.com",
   "username": "myuser",
   "password": "mypassword"
}

What if we want to import data from a GIT?

POST /importers/advisories

{
   "kind": "GIT",
   "url": "https://github.com/org/repo.git",
   "accessToken": "123456"
}

What do we gain making the API the ruler of tasks?

If the API is the one that rules all tasks, we can:

• Monitor the execution of the tasks (importer) through the API or UI
• Highly extensible way of supporting multiple ways and sources of data for them to be imported.
• Schedule the execution of heavy tasks according to resource consumption (CPU, RAM, etc)

User experience

Being able to fire the "import" task from the UI (through the REST API) allows the user to define multiple sources of data, configuring where that data is, and how to get access to that data (credentials required).

• User 1 knows that his company stores advisories & sboms in an s3 storage. Then he opens the UI. Opens a Form and fill the required data: "URL", "bucket name", "credentials". He clicks on save and schedule|run and that's it.
• USer 2 is from Red Hat and wants to import data from https://access.redhat.com/security/data/csaf/v2/advisories/ then he opens a Form, select the type or protocol over which the data will be pulled, then He clicks on save and schedule|run and that's it.

Users can by themselves configure multiple combinations of sources, run them when they want or even create CRONs like "I want this tasks to be executed every week at 12:00am"

[jcrossley3]

Here's how I'm seeing this communique so far:

Bob: "No cron or external processes"
Jens: "Agree! We'll build a super-binary"
Carlos: "No super-binary, just a single API server"
Bob: "I think that's what we're doing"
Jens: "Yeah, super-binary"

🤣

So there's a disconnect we should reconnect, I think.

Jens is suggesting:

trustd server # runs the HTTP API server
trustd importer(s) # runs the importer loops
trustd all # run all components

Carlos is suggesting only this:
trustd server

And then something is making HTTP invocations to our REST endpoints to say, fire up an importer. But I don't think Carlos is suggesting that starts a loop, only a single iteration of the loop Jens is describing. (Correct me if I'm wrong)

So when Carlos says this...

Users can by themselves configure multiple combinations of sources, run them when they want or even create CRONs like "I want this tasks to be executed every week at 12:00am"

We need to be very clear on what that involves. Are these curl commands in a crontab file? Or does our server have a built in scheduler that is configured via our API?

I just want to make sure y'all are hearing each other. 😄

What is the something that's firing up importers? I don't think the answer can be "users", at least not exclusively. Is the implementation of trustd importer just a bunch of POST requests to the process started by trustd server?

Let's agree on exactly what we're building.

[carlosthe19916]

Here's how I'm seeing this communique so far:

Bob: "No cron or external processes"
Jens: "Agree! We'll build a super-binary"
Carlos: "No super-binary, just a single API server"
Bob: "I think that's what we're doing"
Jens: "Yeah, super-binary"

I couldn't avoid laughing. Thanks for sharing it. I guess we are not connecting our ideas, or at least it does not look like it.

I'll add some points if it helps to clarify my points. I'm not saying my points are correct, just trying to expand what I had in mind regardless of whether or not it will be implemented.

What is the something that's firing up importers?

I believe is also important to define:

• How many times the importers will be fired? Are we talking about Jobs or a single fire and forget thing?
  • My assumption is that we want to continuously fetch data from somewhere. Something like "every Sunday pull all the CVE files from the source data I've configured"
  • Since my assumption is that we want to continuously fetch data. Then 2 things might be needed: a scheduler and a executor

This is what I had in mind:

1. There is a POST request to /importers/advisories to set up a task that will import files. This defines where to fetch the files from, what are the credentials to get access to those files (if applicable), and how often we want to fetch files.
2. trustd server gets the request and save it in a Scheduler.
3. The scheduler fires the executor. The executor is the thing that fetch data and import it into our system. We can have 2 types of executors
   • The executor can be an "internal" thing within out trustd server so no need to start anything else (everything runs insider trustd server) OR
   • The executor can also be an "external" process that has a CONTRACT with our server. This helps scaling the app when multiple importers need to be configured in the system.
4. The scheduler sends files through trustd server REST API.
   • The current approach, what we have in main branch, is the importer ingesting data directly to the DB and not through the API.

[ctron]

That's basically what I had in mind. With a slight difference: there is no external (standalone) executor. So there also no need for the executor (not the scheduler) to send documents through the external API, but through the internal API.

That "scheduler" (which is called "importer module") can run as part of trustd all or as part of trustd importer, which shouldn't make a difference.

[bobmcwhirter]

Correct me if I'm wrong, @jcrossley3 but really I think you'd be happy if day zero test-on-your-laptop instructions were limited to:

Execute the following:

$ trustd

And day two instructions would be "if you find yourself needing to split the process, see Chapter 28 of the documentation for those details".

[jcrossley3]

That would make me happy-er, yes. Should I have made that clear sooner? 😄