Programmer's Guide

It is an introduction to the general architecture of Themis codebase and describes how to extend it with new features.

Extending Themis

The section describes how to extend Themis with custom features like types, functions, algorithms and selectors.

Adding New Type

Themis provides a number of attribute types like boolean, string and so on. A type gathers information about possible attribute processing. The information is embedded in various places of PDP sourcecode. This section shows all the places which need modifications to add new type:

• type identification;
• value of the type;
• parsing type value in YAST and JAST;
• parsing type value in JCON;
• usage type value in content;
• marshaller and unmarshaller for PEP.

As an example of new type we are adding integer type here.

Type Constants

First of all one need to add Type* constant and name of new type to attribute.go file:

// Type* constants represent all data types PDP can work with.
const (
    ...
    // TypeInteger is integer data type.
    TypeInteger
    ...
)
...
// Type* collections bind type names and IDs.
var (
    // TypeNames is list of humanreadable type names. The order must be kept
    // in sync with Type* constants order.
    TypeNames = []string{
    ...
    "Integer",
    ...
    }
)

Position of name in TypeNames array should be the same as position of Type* constant. PDP automatically fills TypeKeys array with all lowercase versions of type names and TypeIDs map with mapping of the lowercase type identifiers to Type* constants.

Values of New Type

PDP stores attributes using AttributeValue structure. The structure can be created from native golang value, native can be obtained back from the structure and the structure supports bunch of other operations which bound to related native type. To make AttributeValue aware of new integer type we need to define related native type. Lets take int64 for example. Having this binding we need:

• to define MakeIntegerValue function which takes int64 value and creates instances of AttributeValue structure;
• to add related case to MakeValueFromString function;
• to add related case to Serialize method of AttributeValue structure;
• to add related case to describe method of AttributeValue structure;
• to add integer method to AttributeValue structure;

// MakeIntegerValue creates instance of integer attribute value.
func MakeIntegerValue(v int64) AttributeValue {
    return AttributeValue{
        t: TypeInteger,
        v: v}
}

// MakeValueFromString creates instance of attribute value by given type and
// string representation. The function performs necessary validation.
// No covertion defined for undefined type and collection types.
func MakeValueFromString(t int, s string) (AttributeValue, error) {
    switch t {
    ...
    case TypeInteger:
        n, err := strconv.ParseInt(s, 0, 64)
        if err != nil {
            return undefinedValue, newInvalidIntegerStringCastError(s, err)
        }

        return MakeIntegerValue(n), nil
    ...
    }

    return undefinedValue, newUnknownTypeStringCastError(t)
}

// Serialize converts attribute value to its string representation.
// No conversion defined for undefined value.
func (v AttributeValue) Serialize() (string, error) {
    switch v.t {
    ...
    case TypeInteger:
        return strconv.FormatInt(v.v.(int64), 10), nil
    ...
    }

    return "", newUnknownTypeSerializationError(v.t)
}

func (v AttributeValue) describe() string {
    switch v.t {
    ...
    case TypeInteger:
        return strconv.FormatInt(v.v.(int64), 10)
    ...
    }

    return "val(unknown type)"
}

func (v AttributeValue) integer() (int64, error) {
    err := v.typeCheck(TypeInteger)
    if err != nil {
        return 0, err
    }

    return v.v.(int64), nil
}

Functions new*Error are automatically generated from error descriptors in errors.yaml file. Error descriptor invalidIntegerStringCastError should be defined to get newInvalidIntegerStringCastError function in errors.go file (by go generate command).

Parsing New Type Values from YAST and JAST

Changes above introduce new type and values of the type to PDP. But we still need to make it accessible in policy files. YAST and JAST packages which are responsible for policy parsing contains special functions to parse immediate values. YAST contains the code at ast/yast/value.go file. As YAML has YAML native integer representation we also need to add specific method to get the value for our example with integer. This helper function should be added to ast/yast/context.go. YAML library can return numeric value as golang int, int64, uint64 or float64. To process all the cases following method can be added:

func (ctx context) validateInteger(v interface{}, desc string) (int64, boundError) {
    switch v := v.(type) {
    case int:
        return int64(v), nil

    case int64:
        return v, nil

    case uint64:
        if v > math.MaxInt64 {
            return 0, newUint64OverflowIntegerError(v, desc)
        }

        return int64(v), nil

    case float64:
        if v < -9007199254740992 || v > 9007199254740992 {
            return 0, newFloat64OverflowIntegerError(v, desc)
        }

        return int64(v), nil
    }

    return 0, newIntegerError(v, desc)
}

Here again uint64OverflowIntegerError, float64OverflowIntegerError and integerError descriptors should be defined in errors.yaml. Note that 2^53= 9007199254740992 is maximal integer which can be precisely represented as float64 (because float64 has exactly 53 bits for significand). Bigger integers can be represented with increasing precision loss from +/-1 for 2^53+1 to +/- 512 for MaxInt64 - 512. Numbers MaxInt64 - 511 and higher overflow int64 making it negative.

With the helper method unmarshaller for integer can be defined as following in ast/yast/value.go file:

func (ctx context) unmarshalIntegerValue(v interface{}) (pdp.AttributeValue, boundError) {
    n, err := ctx.validateInteger(v, "value of integer type")
    if err != nil {
        return pdp.AttributeValue{}, err
    }

    return pdp.MakeIntegerValue(n), nil
}

Now complete set of value unmarshallers:

func (ctx context) unmarshalValueByType(t int, v interface{}) (pdp.AttributeValue, boundError) {
    switch t {
    ...
    case pdp.TypeInteger:
        return ctx.unmarshalIntegerValue(v)

    ...
    }

    return pdp.AttributeValue{}, newNotImplementedValueTypeError(t)
}

For JAST parser we need as well to add helper method now to jparser/helpers.go file:

// GetNumber unmarshals number from JSON byte stream.
func GetNumber(d *json.Decoder, desc string) (float64, error) {
    t, err := d.Token()
    if err != nil {
        return 0, err
    }

    n, ok := t.(float64)
    if !ok {
        return 0, newNumberCastError(t, desc)
    }

    return n, nil
}

As golang encoding/json package represents any number as float64 we make only that type assertion here. With the helper we can put JAST unmarshaller to ast/jast/value.go:

func (ctx context) unmarshalIntegerValue(d *json.Decoder) (pdp.AttributeValue, error) {
    x, err := jparser.GetNumber(d, "value of integer type")
    if err != nil {
        return pdp.AttributeValue{}, err
    }

    if x < -9007199254740992 || x > 9007199254740992 {
        return pdp.AttributeValue{}, newIntegerOverflowError(x)
    }

    return pdp.MakeIntegerValue(int64(x)), nil
}

And call it:

func (ctx context) unmarshalValueByType(t int, d *json.Decoder) (pdp.AttributeValue, error) {
    switch t {
    ...
    case pdp.TypeInteger:
        return ctx.unmarshalIntegerValue(d)

    ...
    }

    return pdp.AttributeValue{}, newNotImplementedValueTypeError(t)
}

Using New Type Values in Content

The last step is to make content handlers and JCON parser aware of new type values. For content processing you need to update type checker and getter in content.go file:

func (c *ContentItem) typeCheck(path []AttributeValue, v interface{}) (ContentSubItem, error) {
    ...
    switch c.t {
    default:
        return nil, newUnknownContentItemResultTypeError(c.t)
    ...
    case TypeInteger:
        if _, ok := subItem.value.(int64); !ok {
            return nil, newInvalidContentValueTypeError(subItem.value, TypeInteger)
        }

    ...
    }

    return subItem, nil
}
...
func (v ContentValue) getValue(key AttributeValue, t int) (AttributeValue, error) {
    switch t {
    ...
    case TypeInteger:
        return MakeIntegerValue(v.value.(int64)), nil

    ...
    }

    panic(fmt.Errorf("can't convert to value of unknown type with index %d", t))
}

JCON parser requires unmarshaller for value of new type (jcon/item.go file):

func (c *contentItem) unmarshalValue(d *json.Decoder) (interface{}, error) {
    switch c.t {
    ...
    case pdp.TypeInteger:
        x, err := jparser.GetNumber(d, "value")
        if err != nil {
            return nil, err
        }

        if x < -9007199254740992 || x > 9007199254740992 {
            return nil, newIntegerOverflowError(x)
        }

        return int64(x), nil

    ...
    }

    return nil, newInvalidContentItemTypeError(c.t)
}

Here again we use GetNumber helper for json and check for integer overflow. Similar code is needed for JCON postprocessor (jcon/postprocessor.go - works with data when type information becomes available after content data):

func (c *contentItem) ppValue(v interface{}) (interface{}, error) {
    switch c.t {
    ...
    case pdp.TypeInteger:
        x, ok := v.(float64)
        if !ok {
            return nil, newNumberCastError(v, "value")
        }

        if x < -9007199254740992 || x > 9007199254740992 {
            return nil, newIntegerOverflowError(x)
        }

        return int64(x), nil

    ...
    }

    return nil, newInvalidContentItemTypeError(c.t)
}

PEP

Policy Enforcement Point also need to know how to work with new type. For that purpose we'll add integer marshaller and unmarshaller to pep package.

Let's start with marshaller in marshal.go file:

func intMarshaller(v reflect.Value) (string, string, error) {
    var s string
    switch v.Kind() {
    default:
        panic(fmt.Errorf("expected any integer value but got %q (%s)", v.Type().Name(), v.String()))

    case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
        s = strconv.FormatInt(v.Int(), 10)

    case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
        n := v.Uint()
        if n > math.MaxInt64 {
            return "", "", ErrorIntegerOverflow
        }

        s = strconv.FormatUint(n, 10)
    }

    return s, "integer", nil
}

Implementation requires ErrorIntegerOverflow to be defined. In the same file we can add:

var (
    ...
    // ErrorIntegerOverflow indicates that input structure contains integer
    // which doesn't fit to int64.
    ErrorIntegerOverflow = errors.New("integer overflow")
)

The marshaller should be registered in marshallersByKind for untagged structures and marshallersByTag for structures with tags. Later also required PDP type to internal type mapping.

var (
    ...
    marshallersByKind = map[reflect.Kind]fieldMarshaller{
        ...
        reflect.Int:    intMarshaller,
        reflect.Int8:   intMarshaller,
        reflect.Int16:  intMarshaller,
        reflect.Int32:  intMarshaller,
        reflect.Int64:  intMarshaller,
        reflect.Uint:   intMarshaller,
        reflect.Uint8:  intMarshaller,
        reflect.Uint16: intMarshaller,
        reflect.Uint32: intMarshaller,
        reflect.Uint64: intMarshaller,
        ...
    }

    marshallersByTag = map[string]fieldMarshaller{
        ...
        "integer": intMarshaller,
        ...
    }
    ...
    typeByTag = map[string]reflect.Type{
        ...
        "integer": reflect.TypeOf(0),
        ...
    }
)

Note that typeByTag maps integer type only to int. This is a limitation of current PEP version which supports only 1-to-1 mapping. Mapping typeByTag should be enhanced to be map[string][]reflect.Type and support translation of PDP type to several golang types.

Similarly we need unmarshaller in unmarshal.go:

func intUnmarshaller(attr *pb.Attribute, v reflect.Value) error {
    i, err := strconv.ParseInt(attr.Value, 0, 64)
    if err != nil {
        return fmt.Errorf("can't treat \"%s\" value (%s) as integer: %s", attr.Id, attr.Value, err)
    }

    switch v.Kind() {
    case reflect.Int:
        if i < math.MinInt32 || i > math.MaxInt32 {
            return fmt.Errorf("\"%s\" %d overflows int value", attr.Id, i)
        }

        v.SetInt(i)
        return nil

    case reflect.Int8:
        if i < math.MinInt8 || i > math.MaxInt8 {
            return fmt.Errorf("\"%s\" %d overflows int8 value", attr.Id, i)
        }

        v.SetInt(i)
        return nil

    ...
    case reflect.Int64:
        v.SetInt(i)
        return nil

    case reflect.Uint:
        if i < 0 || i > math.MaxUint32 {
            return fmt.Errorf("\"%s\" %d overflows uint value", attr.Id, i)
        }

        v.SetUint(uint64(i))
        return nil

    case reflect.Uint8:
        if i < 0 || i > math.MaxUint8 {
            return fmt.Errorf("\"%s\" %d overflows uint8 value", attr.Id, i)
        }

        v.SetUint(uint64(i))
        return nil

    ...
    case reflect.Uint64:
        if i < 0 {
            return fmt.Errorf("\"%s\" %d overflows uint64 value", attr.Id, i)
        }

        v.SetUint(uint64(i))
        return nil

    }

    return fmt.Errorf("can't set value %q of \"%s\" attribute to %s", attr.Value, attr.Id, v.Type().Name())
}

Here int and uint values are limited to MinInt32, MaxInt32 and MaxUint32 because golang specification aren't certain about its size - "The int, uint, and uintptr types are usually 32 bits wide on 32-bit systems and 64 bits wide on 64-bit systems".

It also needs registration in unmarshallersByType map:

var unmarshallersByType = map[string]fieldUnmarshaller{
    ...
    pdp.TypeKeys[pdp.TypeInteger]: intUnmarshaller,
    ...
}

Special Cases

Changes listed above allow to introduce new type to PDP. However there are several special cases which requires more specific changes:

• collection - type which can be built similarly to ordinary types on top of existing type (for example Set of Networks or List of Strings);
• types in algorithms - for example String or List of Strings with mapper;
• key types in content - for example String, Address or Domain.

Lets consider types in algorithms. Policy set, policy or rule identifier is a string. Any algorithm (like mapper) which uses string expression to locate particular policy requires very specific checks to be sure that result of embedded expression can be translated to the identifier. Later we can add for example enumration type which is based on some integer type however each value of the enumeration has name and can be used as identifier in mapper expression. In the case mapper requires specific to the enumeration implementation changes to be able to work with it.

Adding New Expression

New type and its value added above aren't useful until we have any way to use them. For now they can be used only as part of obligation. Other way to use a type is to pass values or attributes of the type to some expression. In this section we consider adding integer equal expression as an example. Any expression should satisfy several criteria:

• it should implement Expression interface;
• it should have validator of argument list and makeFunction* constructor;
• validator should be registered at FunctionArgumentValidators.

There is special class of expressions which are suitable for target clause of policy set, policy and rule. An expression should have following properties which make it target compatible:

• it should accept exactly two arguments;
• it should be possible to create index using the expression (later we are going to enhance PDP so it will be able to speed up search of particular rules or policies by converting set of their target expressions to some index).

If new expression satisfies the requirements (and our integer equal expression does) it should have special makeFunction* constructor and the constructor should be listed in TargetCompatibleExpressions map.

Expression Definition

An expression is represented by structure which usually holds list of its arguments. For integer equal example its:

type functionIntegerEqual struct {
    first  Expression
    second Expression
}

We can put the code to expr_integer_equal.go similar to other expressions.

Also we need implement GetResultType method and calculate method to make it compatible with Expression interface. Result of equal expression is always boolean so GetResultType is trivial. Method calculate uses context to calculate expression arguments then compares resulting integers and returns comparison result as boolean value:

func (f functionIntegerEqual) GetResultType() int {
    return TypeBoolean
}

func (f functionIntegerEqual) calculate(ctx *Context) (AttributeValue, error) {
    first, err := ctx.calculateIntegerExpression(f.first)
    if err != nil {
        return undefinedValue, bindError(bindError(err, "first argument"), "equal")
    }

    second, err := ctx.calculateIntegerExpression(f.second)
    if err != nil {
        return undefinedValue, bindError(bindError(err, "second argument"), "equal")
    }

    return MakeBooleanValue(first == second), nil
}

To calculate integer expression as an argument of equal function we need help from context (which code placed in context.go file):

func (c *Context) calculateIntegerExpression(e Expression) (int64, error) {
    v, err := e.calculate(c)
    if err != nil {
        return 0, err
    }

    return v.integer()
}

For now we have several integer expressions which can become argument of equal expression - immediate integer value, getting value of integer attribute, getting value from integer selector.

Registering Expression

The integer expression is fully defined but isn't used anywhere. We still need argument validator and constructor in expr_integer_equal.go:

func functionIntegerEqualValidator(args []Expression) functionMaker {
    if len(args) != 2 || args[0].GetResultType() != TypeInteger || args[1].GetResultType() != TypeInteger {
        return nil
    }

    return makeFunctionIntegerEqualAlt
}

func makeFunctionIntegerEqualAlt(args []Expression) Expression {
    if len(args) != 2 {
        panic(fmt.Errorf("function \"equal\" for Integer needs exactly two arguments but got %d", len(args)))
    }

    return functionIntegerEqual{
        first:  args[0],
        second: args[1],
    }
}

Put validator to expressions map in expression.go file:

// FunctionArgumentValidators maps function name to list of validators.
// For given set of arguments validator returns nil if the function
// doesn't accept the arguments or function which creates expression based
// on desired function and set of argument expressions.
var FunctionArgumentValidators = map[string][]functionArgumentValidator{
    "equal": {
        functionStringEqualValidator,
        functionIntegerEqualValidator,
    },
    ...
}

Registering Target Compatible Expression

Changes above make an expression available for rules and policies condition but not for target. For target we need to register in target.go file two arguments constructor:

// TargetCompatibleExpressions maps name of expression and types of its
// arguments to particular expression maker.
var TargetCompatibleExpressions = map[string]map[int]map[int]twoArgumentsFunctionType{
    "equal": {
        ...
        TypeInteger: {
            TypeInteger: makeFunctionIntegerEqual,
        },
        ...
    },
    ...
}

And constructor itself goes again to expr_integer_equal.go file:

func makeFunctionIntegerEqual(first, second Expression) Expression {
    return functionIntegerEqual{
        first:  first,
        second: second,
    }
}