Error while using PBS on local machine

[XiaMinqi]

I tested the CH4 example provided in doc and tried to use PBS batch_type on local machine.
The job failed with output as follows:

INFO:dpgen:-------------------------iter.000000 task 01--------------------------
2022-12-06 09:02:25,076 - INFO : info:check_all_finished: False
Traceback (most recent call last):
  File "/root/.local/lib/python3.10/site-packages/dpdispatcher/submission.py", line 245, in handle_unexpected_submission_state
    job.handle_unexpected_job_state()
  File "/root/.local/lib/python3.10/site-packages/dpdispatcher/submission.py", line 624, in handle_unexpected_job_state
    self.submit_job()
  File "/root/.local/lib/python3.10/site-packages/dpdispatcher/submission.py", line 664, in submit_job
    job_id = self.machine.do_submit(self)
  File "/root/.local/lib/python3.10/site-packages/dpdispatcher/pbs.py", line 40, in do_submit
    stdin, stdout, stderr = self.context.block_checkcall('cd %s && %s %s' % (script_file_dir, 'qsub', script_file_name))
  File "/root/.local/lib/python3.10/site-packages/dpdispatcher/local_context.py", line 366, in block_checkcall
    raise RuntimeError("Get error code %d in locally calling %s with job: %s ", (code, cmd, self.submission.submission_hash))
RuntimeError: ('Get error code %d in locally calling %s with job: %s ', (175, 'cd /home/lijun/xmq/12.1/dpgen_example/work/f54e2f35d48f03ec7590796bda4ac430377c66f8 && qsub 195f749cebf85f26d09c3371b2e014d5da5f4fcb.sub', 'f54e2f35d48f03ec7590796bda4ac430377c66f8'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/.local/bin/dpgen", line 8, in <module>
    sys.exit(main())
  File "/root/.local/lib/python3.10/site-packages/dpgen/main.py", line 185, in main
    args.func(args)
  File "/root/.local/lib/python3.10/site-packages/dpgen/generator/run.py", line 3642, in gen_run
    run_iter (args.PARAM, args.MACHINE)
  File "/root/.local/lib/python3.10/site-packages/dpgen/generator/run.py", line 3607, in run_iter
    run_train  (ii, jdata, mdata)
  File "/root/.local/lib/python3.10/site-packages/dpgen/generator/run.py", line 610, in run_train
    submission.run_submission()
  File "/root/.local/lib/python3.10/site-packages/dpdispatcher/submission.py", line 184, in run_submission
    self.handle_unexpected_submission_state()
  File "/root/.local/lib/python3.10/site-packages/dpdispatcher/submission.py", line 248, in handle_unexpected_submission_state
    raise RuntimeError(
RuntimeError: Meet errors will handle unexpected submission state.
Debug information: remote_root==/home/lijun/xmq/12.1/dpgen_example/work/f54e2f35d48f03ec7590796bda4ac430377c66f8.
Debug information: submission_hash==f54e2f35d48f03ec7590796bda4ac430377c66f8.
Please check the dirs and scripts in remote_root. The job information mentioned above may help.

To check the origin of error, I changed directory to remote_root and submitted the PBS script generated by dpgen and get the following error:
qsub: Bad UID for job execution

And this is my machine.json file:

{
  "api_version": "1.0",
  "deepmd_version": "2.1.5",
  "train" :
    {
      "command": "mpirun -l -launcher=fork -hosts=localhost -np 56 dp",
      "machine": {
        "batch_type": "PBS",
        "context_type": "local",
        "local_root" : "./",
	"remote_root": "/home/lijun/xmq/12.1/dpgen_example/work",
        "_remote_profile":{
          "_email": "your lebesgue register email",
          "_password": "your lebesgue password",
          "_program_id": 0,
            "_input_data":{
                "_api_version":2,
                "_job_type": "indicate",
                "_log_file": "00*/train.log",
                "_grouped":true,
                "_job_name": "dpgen_train_job",
                "_disk_size": 100,
                "_scass_type":"c8_m60_1 * NVIDIA P100",
                "_platform": "ali",
                "_image_name":"deepmd-kit:2.0.1",
                "_on_demand":0
            }
        }
      },
      "resources": {
        "number_node": 1,
        "local_root":"./",
        "cpu_per_node": 56,
        "gpu_per_node": 0,
        "queue_name": "cpu56",
        "group_size": 1
      }
    },
  "model_devi":
    {
      "command": "mpirun -np 56 lmp -i input.lammps -v restart 0",
      "machine": {
        "batch_type": "PBS",
        "context_type": "local",
        "local_root" : "./",
	"remote_root": "/home/lijun/xmq/12.1/dpgen_example/work",
        "_remote_profile":{
          "_email": "your lebesgue register email",
          "_password": "your lebesgue password",
          "_program_id": 0,
            "_input_data":{
              "_api_version":2,
              "_job_type": "indicate",
              "_log_file": "*/model_devi.log",
              "_grouped":true,
              "_job_name": "dpgen_model_devi_job",
              "_disk_size": 200,
              "_scass_type":"c8_m60_1 * NVIDIA P100",
              "_platform": "ali",
              "_image_name":"deepmd-kit:2.0.1",
              "_on_demand":0
            }
        }
      },
      "resources": {
        "number_node": 1,
        "local_root":"./",
        "cpu_per_node": 56,
        "gpu_per_node": 0,
        "queue_name": "cpu56",
        "group_size": 5
      }
    },
  "fp":
    {
      "command": "mpirun -n 56 vasp_std",
      "machine": {
        "batch_type": "PBS",
        "context_type": "local",
        "local_root" : "./",
	"remote_root": "/home/lijun/xmq/12.1/dpgen_example/work",
        "_remote_profile":{
          "_email": "your lebesgue register email",
          "_password": "your lebesgue password",
          "_program_id": 0,
            "_input_data":{
              "_api_version":2,
              "_job_type": "indicate",
              "_log_file": "task*/fp.log",
              "_grouped":true,
              "_job_name": "dpgen_fp_job",
              "_disk_size": 100,
              "_scass_type":"c8_m8_cpu",
              "_platform": "ali",
              "_image_name":"vasp:5.4.4",
              "_on_demand":0
            }
        }
      },
      "resources": {
        "number_node": 1,
        "cpu_per_node": 56,
        "gpu_per_node": 0,
        "queue_name": "cpu56",
        "group_size": 20,
        "local_root":"./",
	"source_list": ["/opt/intel/oneapi/setvars.sh"]
      }
    }
}

BTW, when executing the workflow with shell instead, the CH4 example finished normally, but the 00.train step showed a quite low cpu usage. I have set the enviroment variables as:

export OMP_NUM_THREADS=8
export TF_INTRA_OP_PARALLELISM_THREADS=8
export TF_INTER_OP_PARALLELISM_THREADS=7

and didn't help.

Hoping on instructions on the problems above.

[njzjz]

Since you failed to submit the job manually, you may need to consult your cluster administrator.

[XiaMinqi]

Thank you for your kind reply. Perhaps I did not describe my issue clearly, the job was submitted and executed on local machine, and then the local machine PBS system failed. The PBS works well in other conditions, however.
Additionally, I'd appreciate if you could offer some advice on how to rasie the cpu usage in 00.train step. As I mentioned above, the cpu usage is still low after setting enviroment variables according to deepmd-kit doc.