Programmer's Guide

It is an introduction to the general architecture of Themis codebase and describes how to extend it with new features.

Extending Themis

The section describes how to extend Themis with custom features like types, functions, algorithms and selectors.

Adding New Type

Themis provides a number of attribute types like boolean, string and so on. A type gathers information about possible attribute processing. The information is embedded in various places of PDP sourcecode. This section shows all the places which need modifications to add new type. integer type is used here as an example of new type.

Type Constants

First of all one need to add Type* constant and name of new type to attribute.go file:

// Type* constants represent all data types PDP can work with.
const (
    ...
    // TypeInteger is integer data type.
    TypeInteger
    ...
)
...
// Type* collections bind type names and IDs.
var (
    // TypeNames is list of humanreadable type names. The order must be kept
    // in sync with Type* constants order.
    TypeNames = []string{
    ...
    "Integer",
    ...
    }
)

Position of name in TypeNames array should be the same as position of Type* constant. PDP automatically fills TypeKeys array with all lowercase versions of type names and TypeIDs map with mapping of the lowercase type identifiers to Type* constants.

Values of New Type

PDP stores attributes using AttributeValue structure. The structure can be created from native golang value, native can be obtained back from the structure and the structure supports bunch of other operations which bound to related native type. To make AttributeValue aware of new integer type we need to define related native type. Lets take int64 for example. Having this binding we need:

• to define MakeIntegerValue function which takes int64 value and creates instances of AttributeValue structure;
• to add related case to MakeValueFromString function;
• to add related case to Serialize method of AttributeValue structure;
• to add related case to describe method of AttributeValue structure;
• to add integer method to AttributeValue structure;

// MakeIntegerValue creates instance of integer attribute value.
func MakeIntegerValue(v int64) AttributeValue {
    return AttributeValue{
        t: TypeInteger,
        v: v}
}

// MakeValueFromString creates instance of attribute value by given type and
// string representation. The function performs necessary validation.
// No covertion defined for undefined type and collection types.
func MakeValueFromString(t int, s string) (AttributeValue, error) {
    switch t {
    ...
    case TypeInteger:
        n, err := strconv.ParseInt(s, 0, 64)
        if err != nil {
            return undefinedValue, newInvalidIntegerStringCastError(s, err)
        }

        return MakeIntegerValue(n), nil
    ...
    }

    return undefinedValue, newUnknownTypeStringCastError(t)
}

// Serialize converts attribute value to its string representation.
// No conversion defined for undefined value.
func (v AttributeValue) Serialize() (string, error) {
    switch v.t {
    ...
    case TypeInteger:
        return strconv.FormatInt(v.v.(int64), 10), nil
    ...
    }

    return "", newUnknownTypeSerializationError(v.t)
}

func (v AttributeValue) describe() string {
    switch v.t {
    ...
    case TypeInteger:
        return strconv.FormatInt(v.v.(int64), 10)
    ...
    }

    return "val(unknown type)"
}

func (v AttributeValue) integer() (int64, error) {
    err := v.typeCheck(TypeInteger)
    if err != nil {
        return 0, err
    }

    return v.v.(int64), nil
}

Functions new*Error are automatically generated from error descriptors in errors.yaml file. Error descriptor invalidIntegerStringCastError should be defined to get newInvalidIntegerStringCastError function in errors.go file (by go generate command).

Parsing New Type Values from YAST and JAST

Changes above introduce new type and values of the type to PDP. But we still need to make it accessible in policy files. YAST and JAST packages which are responsible for policy parsing contains special functions to parse immediate values. YAST contains the code at ast/yast/value.go file. As YAML has YAML native integer representation we also need to add specific method to get the value for our example with integer. This helper function should be added to ast/yast/context.go. YAML library can return numeric value as golang int, int64, uint64 or float64. To process all the cases following method can be added:

func (ctx context) validateInteger(v interface{}, desc string) (int64, boundError) {
    switch v := v.(type) {
    case int:
        return int64(v), nil

    case int64:
        return v, nil

    case uint64:
        if v > math.MaxInt64 {
            return 0, newUint64OverflowIntegerError(v, desc)
        }

        return int64(v), nil

    case float64:
        if v < -9007199254740992 || v > 9007199254740992 {
            return 0, newFloat64OverflowIntegerError(v, desc)
        }

        return int64(v), nil
    }

    return 0, newIntegerError(v, desc)
}

Here again uint64OverflowIntegerError, float64OverflowIntegerError and integerError descriptors should be defined in errors.yaml. Note that 2^53= 9007199254740992 is maximal integer which can be precisely represented as float64 (because float64 has exactly 53 bits for significand). Bigger integers can be represented with increasing precision loss from +/-1 for 2^53+1 to +/- 512 for MaxInt64 - 512. Numbers MaxInt64 - 511 and higher overflow int64 making it negative.

With the helper method unmarshaller for integer can be defined as following in ast/yast/value.go file:

func (ctx context) unmarshalIntegerValue(v interface{}) (pdp.AttributeValue, boundError) {
    n, err := ctx.validateInteger(v, "value of integer type")
    if err != nil {
        return pdp.AttributeValue{}, err
    }

    return pdp.MakeIntegerValue(n), nil
}

Now complete set of value unmarshallers:

func (ctx context) unmarshalValueByType(t int, v interface{}) (pdp.AttributeValue, boundError) {
    switch t {
    ...
    case pdp.TypeInteger:
        return ctx.unmarshalIntegerValue(v)

    ...
    }

    return pdp.AttributeValue{}, newNotImplementedValueTypeError(t)
}

For JAST parser we need as well to add helper method now to jparser/helpers.go file:

// GetNumber unmarshals number from JSON byte stream.
func GetNumber(d *json.Decoder, desc string) (float64, error) {
    t, err := d.Token()
    if err != nil {
        return 0, err
    }

    n, ok := t.(float64)
    if !ok {
        return 0, newNumberCastError(t, desc)
    }

    return n, nil
}

As golang encoding/json package represents any number as float64 we make only that type assertion here. With the helper we can put JAST unmarshaller to ast/jast/value.go:

func (ctx context) unmarshalIntegerValue(d *json.Decoder) (pdp.AttributeValue, error) {
    x, err := jparser.GetNumber(d, "value of integer type")
    if err != nil {
        return pdp.AttributeValue{}, err
    }

    if x < -9007199254740992 || x > 9007199254740992 {
        return pdp.AttributeValue{}, newIntegerOverflowError(x)
    }

    return pdp.MakeIntegerValue(int64(x)), nil
}

And call it:

func (ctx context) unmarshalValueByType(t int, d *json.Decoder) (pdp.AttributeValue, error) {
    switch t {
    ...
    case pdp.TypeInteger:
        return ctx.unmarshalIntegerValue(d)

    ...
    }

    return pdp.AttributeValue{}, newNotImplementedValueTypeError(t)
}

Using New Type Values in Content

The last step is to make content handlers and JCON parser aware of new type values. For content processing you need to update type checker and getter in content.go file:

func (c *ContentItem) typeCheck(path []AttributeValue, v interface{}) (ContentSubItem, error) {
    ...
    switch c.t {
    default:
        return nil, newUnknownContentItemResultTypeError(c.t)
    ...
    case TypeInteger:
        if _, ok := subItem.value.(int64); !ok {
            return nil, newInvalidContentValueTypeError(subItem.value, TypeInteger)
        }

    ...
    }

    return subItem, nil
}
...
func (v ContentValue) getValue(key AttributeValue, t int) (AttributeValue, error) {
    switch t {
    ...
    case TypeInteger:
        return MakeIntegerValue(v.value.(int64)), nil

    ...
    }

    panic(fmt.Errorf("can't convert to value of unknown type with index %d", t))
}

JCON parser requires unmarshaller for value of new type (jcon/item.go file):

func (c *contentItem) unmarshalValue(d *json.Decoder) (interface{}, error) {
    switch c.t {
    ...
    case pdp.TypeInteger:
        x, err := jparser.GetNumber(d, "value")
        if err != nil {
            return nil, err
        }

        if x < -9007199254740992 || x > 9007199254740992 {
            return nil, newIntegerOverflowError(x)
        }

        return int64(x), nil

    ...
    }

    return nil, newInvalidContentItemTypeError(c.t)
}

Here again we use GetNumber helper for json and check for integer overflow. Similar code is needed for JCON postprocessor (jcon/postprocessor.go - works with data when type information becomes available after content data):

func (c *contentItem) ppValue(v interface{}) (interface{}, error) {
    switch c.t {
    ...
    case pdp.TypeInteger:
        x, ok := v.(float64)
        if !ok {
            return nil, newNumberCastError(v, "value")
        }

        if x < -9007199254740992 || x > 9007199254740992 {
            return nil, newIntegerOverflowError(x)
        }

        return int64(x), nil

    ...
    }

    return nil, newInvalidContentItemTypeError(c.t)
}