Replace jinja templates with class generation for parameterized jobs

[yurinnick]

Currently kbuild, kunit, fstests etc. jobs are based on python-heavy jinja2 templates. It makes it a whole lot difficult to understand and make changes. Let's discuss ways to move from templates to parameterized classes or class generators, or any other solution here.

[gctucker]

Yes this is entirely derived from the proof-of-concept and the whole thing needs to be rewritten, basically. I think one solution is to have all the common logic in kernelci-core to offload the templates. The first step was to have something that worked, now we can improve it incrementally.

I think this issue is too vague and not describing an actual issue or proposing an actual solution, so could you please rework it or maybe create specific ones that can be resolved with concrete changes?

[nuclearcat]

This is exactly what i am working on now, and exactly my thoughts @yurinnick .
And also i agree with @gctucker , i think it was partially just copied from legacy system and something that can be improved.

As i'm working now on improving kbuild i propose following (it is very early draft proposal):

When we generate job template, for any runtime, instead of going complex jinja2 magic we rather just execute "kci job generate --input-node-json nodefile.json jobname", which will generate compiling steps for next container (cross-compiler specific).

So for kubernetes it will looks like:

spec:
  initContainers:
  - name: prepare-job
    image: kernelci/api
    command: ['sh', '-c', "kci job generate ....  > /volume/compile.sh"]
# Basically it will fetch kernel sources, generate compile commands
  - name: compiler
    image: kernelci/gcc-10-arm64
    command: ['sh', '-c', "/volume/compile.sh"]
  containers:
  - name: finalize-job
    image: kernelci/api
    command: ['sh', '-c', 'cd /volume;kci job submit ....']
# It will create new nodeid, upload artifacts, etc

For Docker runtime it is a bit more complicated, but flow is about the same, first container will prepare everything, then spin "compiler" DinD, wait until it is completed, and do "finalize" job.

Such job templates will be much easier to troubleshoot manually as well.

[nuclearcat]

So maybe we can use something like this:

import sys

class BaseJob:
    def __init__(self, name):
        self.name = name

    def test(self):
        print("Base param: "+self.name)

class Kbuild(BaseJob):
    def __init__(self, name):
        super().__init__(name)

    def test(self):
        print("Kbuild param: "+self.name)

class Kunit(BaseJob):
    def __init__(self, name):
        super().__init__(name)

    def test(self):
        print("Kunit param: "+self.name)


def main():
    JobClass = getattr(sys.modules[__name__], "Kbuild")("param1")
    JobClass.test()
    JobClass = getattr(sys.modules[__name__], "Kunit")("param2")
    JobClass.test()

if __name__ == "__main__":
    main()

[gctucker]

What's the .test() method meant to be doing?

[nuclearcat]

I did short PoC of defining classes over variable and verified if it works, it is specific to this PoC.

[gctucker]

Right, but how is that meant to work in terms of separating the job definition and the Runtime implementations?

[nuclearcat]

Similar as before, just we don't generate new class over jinja2 template, we write for example KbuildJob, KunitJob, KverJob classes.
So when scheduler call to generate job for runtime, it will generate for first stage - kci command with job name and node information.
When this first stage will run inside runtime, by job name it will look up relevant configuration piece in yaml, figure out class name and use it, and use it for generating shell script for second stage (compilation inside "compiler container").
(I feel it is easier to write it than explain).