Package, hierarchy and divergent output

[rozaliev]

I've started experimenting with cue recently, and while working in a flat directory things seem to work as expected, but when I introduce any sort of hierarchy everything starts behaving weird, and I can't wrap my head around it.
Example:

cue version 0.7.0

go version go1.21.5
      -buildmode exe
       -compiler gc
       -trimpath true
  DefaultGODEBUG panicnil=1
     CGO_ENABLED 1
          GOARCH arm64
            GOOS darwin

.
├── cue.mod
│   ├── module.cue
│   ├── pkg
│   └── usr
├── main.cue
├── same.cue
├── sub
│   └── sub.cue
└── tester
   └── tester.cue

main.cue

package platform

resources: [ID=string]: #Spec & {
	name: ID
}

resources: [string]: #Spec

#Spec: {
	name:  string
	other: uint
	clusters: [string, ...string]
}

same.cue

package platform

// Uncomment me 
// tester: close({
// 	"dev": {
// 		"f1": 1
// 	}

// 	"prod": {
// 		"f2": 2
// 	}
// 	"infra": {
// 		"f3": 2
// 	}
// })

tester: [string]: _

#Allowlist: {
	allowed: [string]: _
	guard: allowed & {for _, r in resources for _, c in r.clusters {"\(c)": _}}
}

#Allowlist: {
	allowed: tester
}

tester/tester.cue

package platform

tester: close({
	"dev": {
		"f1": 1
	}

	"prod": {
		"f2": 2
	}
	"infra": {
		"f3": 2
	}
})

sub/sub.cue

package platform

resources: my: {
	name:  "my"
	other: 123
	clusters: ["prod", "other"]
}
resources: "yo": {
	name:  "yo"
	other: 444
	clusters: ["infra", "dev"]
}

cue vet ./... -c outputs nothing for this, but if I uncomment the block in same.cue it works, producing:

#Allowlist.guard.other: field not allowed:
    ./same.cue:3:15
    ./same.cue:16:9
    ./same.cue:19:11
    ./same.cue:20:9
    ./same.cue:20:19
    ./same.cue:20:20
    ./same.cue:20:65
    ./same.cue:20:66
    ./same.cue:24:1

My mental model was that everything inside a package is a giant set, every file will eventually merge. But in reality, it seems, that I'm getting a set of divergent outputs:

> cue export ./...
{
    "tester": {},
    "resources": {}
}
{
    "tester": {},
    "resources": {
        "my": {
            "name": "my",
            "other": 123,
            "clusters": [
                "prod",
                "other"
            ]
        },
        "yo": {
            "name": "yo",
            "other": 444,
            "clusters": [
                "infra",
                "dev"
            ]
        }
    }
}
{
    "tester": {
        "dev": {
            "f1": 1
        },
        "prod": {
            "f2": 2
        },
        "infra": {
            "f3": 2
        }
    },
    "resources": {}
}

So, it brings a bunch of questions: Does cue only work with flat file structure? If every file is marked as being part of the package, why is behavior different?

My guess is that files that are not a part of same hierarchy will never merge, and every subfolder produces its own merge with the root.

I was planning to build a validator with data at the leafs, schema validators and cross leaf checks at the top, but it doesn't work. Then, what is a proper way to build it? How do I express constraints that work between multiple different leafs? And how do I get a single unified output?

P.S. Merry Christmas!

[rozaliev]

I think I've found a similar issue: #1652 and slowly re-read https://cuelang.org/docs/concepts/packages/#file-organization about Instances.

As per @myitcv request I would like to mark it as “this really does fit/work”. I'll describe my use-case:

I'm trying to build a single source of truth for configs in Bazel driven monorepo. That means every cluster, app, Kubernetes resource should be there, in one big hermetic and consistent package. So that after I run cue vet every misconfig is visible, there should be no way to deploy to wrong cluster, use non-existing ENV variable or forget annotation for a pod. The result is going to be sliced accordingly and rendered for CD to deploy.

I think I can express most of the constraints with CUE, but everything package wise seems to work against me. It's definitely not an option to have hundreds or thousands files in a single flat directory. Creating a single vertical chain of directories is a bit weird, but also potentially bug prone, it requires everybody on the team to have perfect understanding of how Instances work, otherwise they can accidentally introduce a new one and be left with “why doesn't it work?” situation (like me before).

This leaves me with not that many options, either I use Go API to artificially flatten the hierarchy or create some Bazel rules to do the same. Both options seem to be more involved than I've expected. I might have missed some flag that makes everything work 🤷 It doesn't look like there are a lot of examples with similar cases in open source either.

I'd also like to point out that Instances behavior is unintuitive and potentially very bug prone. As a newcomer to CUE it leaves me with code that looks ok, passes vet, but shouldn't. Only after tinkering with a lot of reduced cases you can discover what's actually going on. I fully expect, that if I to introduce multi instanced version to our repo, someone will eventually write a policy that validates only a subset of data, and we will have an outage.

I hope that I described our use case well enough, and it would help smooth the experience for similar cases. Feel free to ask additional questions. I'm planning to check out Go API later this week, and hope that it's gonna work out.

[rozaliev]

Just an update: this line produces result I was look for:
cue vet $(find . -not -path "./cue.mod/**/*" -name "*.cue") -c

The same result can be archive using Go API https://cuetorials.com/go-api/loading/

[kkukshtel]

Looking for similar answers here but also — why can't cue eval/vet just look in subfolders for package files?