LMEvalJobs with Kueue suspend enabled start immediately

[ruivieira]

Tested with:

• ODH 2.19
• TrustyAI latest

DataScienceCluster:

kind: DataScienceCluster
metadata:
  name: default-dsc
  labels:
    app.kubernetes.io/created-by: opendatahub-operator
    app.kubernetes.io/instance: default
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: datasciencecluster
    app.kubernetes.io/part-of: opendatahub-operator
spec:
  components:
    codeflare:
      managementState: Removed
    kserve:
      serving:
        ingressGateway:
          certificate:
            type: OpenshiftDefaultIngress
        managementState: Managed
        name: knative-serving
      managementState: Removed
      defaultDeploymentMode: Serverless
    modelregistry:
      registriesNamespace: odh-model-registries
      managementState: Removed
    trustyai:
      devFlags:
        manifests:
          - contextDir: config
            sourcePath: ''
            uri: 'https://github.com/trustyai-explainability/trustyai-service-operator/tarball/main'
      managementState: Managed
    ray:
      managementState: Removed
    kueue:
      managementState: Managed
    workbenches:
      managementState: Removed
    dashboard:
      managementState: Managed   
    modelmeshserving:
      managementState: Managed
    datasciencepipelines:
      managementState: Removed
    trainingoperator:
      managementState: Removed

LMEvalJob CR:

apiVersion: trustyai.opendatahub.io/v1alpha1
kind: LMEvalJob
metadata:
  name: evaljob-sample
spec:
  suspend: true
  model: hf
  modelArgs:
  - name: pretrained
    value: google/flan-t5-base 
  taskList:
    taskRecipes:
    - card:
        name: "cards.wnli" 
      template: "templates.classification.multi_class.relation.default" 
  logSamples: true

[yhwang]

one quick question: is the JOB_MGR added to the enable-services here: https://github.com/trustyai-explainability/trustyai-service-operator/blob/main/config/manager/manager.yaml#L36

The JOB_MGR is not in the deployment yet. The overlays that enable the JOB_MGR is not there yet. Need to manually enable this feature.

[yhwang]

Let me re-post my comment on another PR here. It's easier to know all the needed settings for enabling the Kueue integration with the LMEvalJob:

When enabling the Kueue for LMES, one extra update in the kueue-manager-config configmap is needed. LMEvalJob needs to be added into the externalFrameworks like this:

integrations:
  .......
  externalFrameworks:
  - "trustyai.opendatahub.io/lmevaljob"