dynamic host volumes: account for other claims in capability check

[tgross]

When we feasibility check a dynamic host volume against a volume request, we check the attachment mode and access mode. This only ensures that the capabilities match, but doesn't enforce the semantics of the capabilities against other claims that may be made on the allocation.

Add support for checking the requested capability against other allocations that the volume claimed.

Ref: #24479

[tgross]

Moving back to draft as Daniel has pointed out a fairly serious design flaw in how this works with group.count > 1 on the first deploy. Fixed!

[tgross]

We're now using the scheduler's context's ProposedAllocs (requiring a trip back to the state store to get the Job structs, but that's no big deal), and that lets us remove the previously-proposed tracking of the previous alloc ID, counting writers, etc.

In addition to the unit tests, I've run this across deployments of a job with group.count=2 and the single-node-single-writer mode to verify everything is working as we'd expect.

[gulducat]

lgtm! just one bit of comment clarity

[gulducat]

I see what this is doing, but I don't follow the reasoning in this comment. I read it as meaning that different groups in the job will all have the same volume params, but this is more about allocs per group, specifically count, right?

Suggested change

	// all allocs for the same job will have the same read-only flag
	// and capabilities, so we only need to check a given job once
	continue
	// all allocs for the same task group will have the same read-only
	// flag and capabilities, so we only need to check a given job once
	continue

using the job's ns+id as the seen key, without any reference to group, threw me a bit, I think. not that you should change that, just that the comment is pretty important to clarify this.

[tgross]

Yeah, very good point. So that's actually a bug here because if a job has multiple allocs for different task groups on the same node, then we'd potentially miss volume requests to check. Ex. a job has alloc A for group A and alloc B for group B on the same node, and only group B has a volume request. If we happen to check alloc A first we'd miss that.

Will fix.