Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intel RDT: updates according to proposed spec changes. #3832

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 44 additions & 10 deletions libcontainer/intelrdt/intelrdt.go
Original file line number Diff line number Diff line change
Expand Up @@ -146,10 +146,11 @@ import (
*/

type Manager struct {
mu sync.Mutex
config *configs.Config
id string
path string
mu sync.Mutex
config *configs.Config
id string
path string
directoryCreated bool
}

// NewManager returns a new instance of Manager, or nil if the Intel RDT
Expand All @@ -170,9 +171,10 @@ func NewManager(config *configs.Config, id string, path string) *Manager {
// is actually available. Used by unit tests that mock intelrdt paths.
func newManager(config *configs.Config, id string, path string) *Manager {
return &Manager{
config: config,
id: id,
path: path,
config: config,
id: id,
path: path,
directoryCreated: false,
}
}

Expand Down Expand Up @@ -466,6 +468,14 @@ func (m *Manager) Apply(pid int) (err error) {
}
}

// If the directory doesn't exist we need to create it -> it means we also need
// to clean it up afterwards. Make a note to the manager.
if _, err := os.Stat(path); err != nil {
if os.IsNotExist(err) {
m.directoryCreated = true
}
}

if err := os.MkdirAll(path, 0o755); err != nil {
return newLastCmdError(err)
}
Expand All @@ -482,8 +492,9 @@ func (m *Manager) Apply(pid int) (err error) {
func (m *Manager) Destroy() error {
// Don't remove resctrl group if closid has been explicitly specified. The
// group is likely externally managed, i.e. by some other entity than us.
// There are probably other containers/tasks sharing the same group.
if m.config.IntelRdt != nil && m.config.IntelRdt.ClosID == "" {
// There are probably other containers/tasks sharing the same group. Also
// only remove the directory if it was created by us.
if m.config.IntelRdt != nil && m.config.IntelRdt.ClosID == "" && m.directoryCreated {
m.mu.Lock()
defer m.mu.Unlock()
if err := os.RemoveAll(m.GetPath()); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not really about this PR, but this can benefit from cgroups.RemovePath, as here we only want to remove directories, not files.

Expand Down Expand Up @@ -589,6 +600,28 @@ func (m *Manager) GetStats() (*Stats, error) {
return stats, nil
}

func combineSchemas(l3CacheSchema, memBwSchema string) string {
// If both l3CacheSchema and memBwSchema are set and
// l3CacheSchema contains a line beginning with "MB:", the
// value written to schemata file MUST be the non-"MB:"
// line(s) from l3CacheSchema and the line from memBWSchema.
Comment on lines +604 to +607
Copy link
Contributor

@kolyshkin kolyshkin May 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, can you elaborate on that requirement? In particular, why we allow a user to include MB: lines into l3CacheSchema? Would it be better to offload this requirement to a user of runc (whoever creates the OCI spec)? I mean, runc is a low level tool and should, theoretically, be as simple as possible.

My other question is -- what happens if we write both values as they are (i.e. without filtering)? Is this a security risk? Or is it just a convenience feature that allows memBwSchema to overwrite whatever is in l3CacheSchema?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One other thought. Let's say that we have

memBwSchema         = "MB:0=70;1=20"
l3CacheSchema       = "MB:0=80;1=10\nL3:0=f0;1=f"

Does it make a difference for the kernel, if we write

MB:0=80;1=10
L3:0=f0;1=f
MB:0=70;1=20

or just

L3:0=f0;1=f
MB:0=70;1=20

?

(I mean, except the additional line parsing done in the kernel)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To answer your other question: I tried this out now and having two MB values for the same domain in the same write won't work:

[root@wolfpass ipuustin]# uname -r
6.0.10-100.fc35.x86_64
[root@wolfpass ipuustin]# cat schemata-1.txt
MB:0=80;1=90
L3:0=ff;1=ff
[root@wolfpass ipuustin]# cat schemata-2.txt
MB:0=80;1=90
L3:0=ff;1=ff
MB:0=60;1=70
[root@wolfpass ipuustin]# cat schemata-1.txt > /sys/fs/resctrl/foo/schemata
[root@wolfpass ipuustin]# cat /sys/fs/resctrl/foo/schemata
    MB:0= 80;1= 90
    L3:0=0ff;1=0ff
[root@wolfpass ipuustin]# cat schemata-2.txt > /sys/fs/resctrl/foo/schemata
cat: write error: Invalid argument
[root@wolfpass ipuustin]# cat /sys/fs/resctrl/info/last_cmd_status
Duplicate domain 0

So we know that there is nobody trying to set overlapping MB: values in both memBwSchema and l3CacheSchema because that would now trigger an error in runc. There might be someone setting non-overlapping MB: values and not getting those filtered out however.


validLines := make([]string, 0)

// Split the l3CacheSchema to lines.
lines := strings.Split(l3CacheSchema, "\n")

// Remove the "MB:" lines.
for _, line := range lines {
if strings.HasPrefix(line, "MB:") {
continue
}
validLines = append(validLines, line)
}

return strings.Join(validLines, "\n") + "\n" + memBwSchema
Comment on lines +604 to +622
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not very efficient in terms of resources. We split the strings and then we combine them, creating two slices in the process. I'm not sure if there's a better solution (except than a slight optimization of reusing the same slice -- see https://github.com/golang/go/wiki/SliceTricks#filtering-without-allocating).

}

// Set Intel RDT "resource control" filesystem as configured.
func (m *Manager) Set(container *configs.Config) error {
// About L3 cache schema:
Expand Down Expand Up @@ -649,7 +682,8 @@ func (m *Manager) Set(container *configs.Config) error {

// Write a single joint schema string to schemata file
if l3CacheSchema != "" && memBwSchema != "" {
if err := writeFile(path, "schemata", l3CacheSchema+"\n"+memBwSchema); err != nil {
schemata := combineSchemas(l3CacheSchema, memBwSchema)
if err := writeFile(path, "schemata", schemata); err != nil {
return err
}
}
Expand Down
40 changes: 40 additions & 0 deletions libcontainer/intelrdt/intelrdt_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,46 @@ func TestIntelRdtSetMemBwSchema(t *testing.T) {
}
}

func TestIntelRdtSetCombinedSchema(t *testing.T) {
helper := NewIntelRdtTestUtil(t)

// Test filtering out the "MB:"" line in l3CacheSchema.

const (
schemaBefore = "MB:0=20;1=70"
memBwSchema = "MB:0=70;1=20"
l3CacheSchema = "MB:0=80;1=10\nL3:0=f0;1=f"
combinedSchemaAfter = "L3:0=f0;1=f\nMB:0=70;1=20"
)

helper.writeFileContents(map[string]string{
"schemata": schemaBefore + "\n",
})

helper.config.IntelRdt.MemBwSchema = memBwSchema
helper.config.IntelRdt.L3CacheSchema = l3CacheSchema
intelrdt := newManager(helper.config, "", helper.IntelRdtPath)
if err := intelrdt.Set(helper.config); err != nil {
t.Fatal(err)
}

tmpStrings, err := getIntelRdtParamString(helper.IntelRdtPath, "schemata")
if err != nil {
t.Fatalf("Failed to parse file 'schemata' - %s", err)
}

readValues := strings.Split(tmpStrings, "\n")
expectedValues := strings.Split(combinedSchemaAfter, "\n")

if readValues[0] != expectedValues[0] {
t.Fatal("Got the wrong value for L3 cache, set 'schemata' failed.")
}

if readValues[1] != expectedValues[1] {
t.Fatal("Got the wrong value for MemBW, set 'schemata' failed.")
}
}

func TestIntelRdtSetMemBwScSchema(t *testing.T) {
helper := NewIntelRdtTestUtil(t)

Expand Down