/v1/job/statuses doesn't filter by namespace when using a non management token

[jorgemarey]

Nomad version

nomad v1.8.0

Issue

When ACLs are enabled and using a non management token, the namespace filter doesn't work correctly in the UI nor the API v1/jobs/statuses directly.

Reproduction steps

1. nomad agent -dev -acl-enabled
2. nomad acl bootstrap
3. nomad acl policy apply reader policy.hcl
   namespace "*" {
   policy = "read"
   }
4. nomad namespace apply test
5. nomad namespace apply dev
6. nomad run example.nomad.hcl (on namespace test)
7. nomad run example.nomad.hcl (on namespace dev)
8. nomad acl token create -ttl 1h -policy=reader -name test

Try to filter using the UI or the API by namespace using the token created in step 8.

Expected Result

• Only jobs of the filtered namespace should be shown

Actual Result

• All jobs for all allowed namespaces are shown.

[tgross]

Hi @jorgemarey! Strangely I wasn't able to confirm this behavior with Nomad 1.8.0 or the current tip of main. I ran thru the reproduction steps you've got here and despite having read access to both namespaces, I only see the job in the dev namespace in the API response:

$ nomad acl token self
Accessor ID  = 259b0161-16b6-7f99-6b2d-1197ccd30868
Secret ID    = <redacted>
Name         = test
Type         = client
Global       = false
Create Time  = 2024-06-21 19:32:17.162311689 +0000 UTC
Expiry Time  = <none>
Create Index = 21
Modify Index = 21
Policies     = [reader]

Roles
<none>


$ nomad operator api '/v1/jobs/statuses?filter=Namespace%20%3D%3D%20dev' | jq .
[
  {
    "ID": "example",
    "Namespace": "dev",
    "Name": "example",
    "Type": "service",
    "NodePool": "default",
    "Datacenters": [
      "*"
    ],
    "Priority": 50,
    "Version": 0,
    "SubmitTime": 1718998345549213000,
    "ModifyIndex": 35,
    "Allocs": [
      {
        "ID": "4fd3d076-ad79-b95d-7d74-3e7d90225c1d",
        "Group": "group",
        "ClientStatus": "running",
        "NodeID": "2982de71-599c-87da-dff7-ab2adf39b206",
        "DeploymentStatus": {
          "Canary": false,
          "Healthy": true
        },
        "JobVersion": 0,
        "FollowupEvalID": "",
        "HasPausedTask": false
      }
    ],
    "GroupCountSum": 1,
    "ChildStatuses": null,
    "ParentID": "",
    "LatestDeployment": {
      "ID": "d012d70a-a0b3-bc41-0311-60b3d0bfd1c4",
      "IsActive": false,
      "JobVersion": 0,
      "Status": "successful",
      "StatusDescription": "Deployment completed successfully",
      "AllAutoPromote": false,
      "RequiresPromotion": false
    }
  }
]

I see the same behavior in the UI. Maybe there's some code path in the UI that's sending the wrong querystring. Can you capture the path and querystring being sent from the devtools? But that wouldn't explain what you're seeing in the API, so can you confirm the exact API path and querystring you're sending for that as well?

[jorgemarey]

Hi @tgross I just reproduced it myself just now. Did you export the token or are you using a management token to make the statuses request? Using a management token it works fine.

Here is the history I just ran with nomad v1.8.0:

10004  ./nomad agent -dev -acl-enabled
10005  ./nomad acl bootstrap
10006  export NOMAD_TOKEN=63342f7a-378b-65ca-726a-b1d089c1f573
10007  vi policy.hcl
10008  ./nomad acl policy apply reader policy.hcl
10009  ./nomad namespace apply test
10010  ./nomad namespace apply dev
10011  ./nomad init
10012  vi example.nomad.hcl   # set namespace to "test"
10013  ./nomad run example.nomad.hcl
10014  vi example.nomad.hcl   # set namespace to "dev"
10015  ./nomad run example.nomad.hcl
10016  curl -H "X-Nomad-Token: ${NOMAD_TOKEN}" "http://localhost:4646/v1/jobs/statuses?namespace=*" | jq .    <- ok
10017  curl -H "X-Nomad-Token: ${NOMAD_TOKEN}" "http://localhost:4646/v1/jobs/statuses?namespace=dev" | jq .   <- ok
10018  ./nomad acl token create -ttl 1h -policy=reader -name test
10019  export NOMAD_TOKEN=4c05130a-45fb-1302-afda-d6fb9cd8dccb
10020  curl -H "X-Nomad-Token: ${NOMAD_TOKEN}" "http://localhost:4646/v1/jobs/statuses?namespace=dev" | jq .   <-  not ok  (shows all jobs)

[jorgemarey]

Just made a script that's able to reproduce it:

❯ cat run.sh 
#!/bin/bash
export NOMAD_TOKEN=$(./nomad acl bootstrap -json | jq -r .SecretID)
echo "namespace \"*\" {  policy = \"read\"}" |  ./nomad acl policy apply reader -
export NOMAD_READER_TOKEN=$(./nomad acl token create -json -ttl 1h -policy=reader -name test | jq -r .SecretID)
rm example.nomad.hcl
./nomad init
./nomad namespace apply test
./nomad namespace apply dev
./nomad job run -namespace=test -detach example.nomad.hcl
./nomad job run -namespace=dev -detach example.nomad.hcl
curl -s -H "X-Nomad-Token: ${NOMAD_TOKEN}" "http://localhost:4646/v1/jobs/statuses?namespace=*" | jq '. | length'
curl -s -H "X-Nomad-Token: ${NOMAD_READER_TOKEN}" "http://localhost:4646/v1/jobs/statuses?namespace=*" | jq '. | length' 
curl -s -H "X-Nomad-Token: ${NOMAD_TOKEN}" "http://localhost:4646/v1/jobs/statuses?namespace=dev" | jq '. | length'
curl -s -H "X-Nomad-Token: ${NOMAD_READER_TOKEN}" "http://localhost:4646/v1/jobs/statuses?namespace=dev" | jq '. | length' 

1. Download nomad: https://releases.hashicorp.com/nomad/1.8.1/nomad_1.8.1_linux_amd64.zip
2. ./nomad agent -dev -acl-enabled
3. ./nomad agent -dev -acl-enabled

The output that I got is allways:

❯ ./run.sh 
Successfully wrote "reader" ACL policy!
rm: cannot remove 'example.nomad.hcl': No such file or directory
Example job file written to example.nomad.hcl
Successfully applied namespace "test"!
Successfully applied namespace "dev"!
Job registration successful
Evaluation ID: 1ee59166-c5cb-0836-179e-4bbc471b6697
Job registration successful
Evaluation ID: c591a0a7-fbb4-e0a4-78be-780798c24e17
2
2
1
2

That final 2, should be a one. Hope this helps to find out where the problem is. Looking at the code, the problem could be arround here as allowableNamespaces is set to all namespaces the token has access and then thats used to filter.

[tgross]

Ah, thanks @jorgemarey! Now I see what the difference in the two requests we were making, and I can confirm what you're seeing.

You're sending the namespace query parameter, as in /v1/jobs/statuses?namespace=dev, whereas I was sending a filter parameter /v1/jobs/statuses?filter=Namespace+%3D%3D+dev. I was sending that request because when I look at the new UI there's no namespace dropdown, just a filter expression. (That being said, after I posted this I moved on to a different issue and did something that caused the namespace dropdown to reappear, so that may be an unrelated bug.)

The namespace query parameter absolutely should work here, and I think you're on the right track in terms of debugging. I'm getting towards the end of my day here but that should be a quick fix to pick up tomorrow.

[tgross]

Fix PR up here: #23456

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.