JSON based thing descriptions

[draggett]

This is a proposal for a JSON based representation of thing data models for the type system proposed in Issue 119. It is a variant of JSON-LD and intended to provide compact descriptions that will be more appealing to web developers than other more verbose alternatives. The Web of Things is expected to be huge so why shouldn't we design a representation that best suites the needs and preference of Web developers?

Some people may ask why not use JSON schema? JSON Schema has the same role as XML Schema for XML, i.e. you can use it to verify that a JSON structure conforms to a given schema. Using JSON Schema to describe the data models for properties, actions and events would constrain them to the type system defined for JSON. Given that the Web of Things is intended to work with wide range of platforms, this assumption is likely to break.

If we were to adopt JSON Schema for applications where the assumption holds, then in principle it could be applied to the values for properties, actions and events. We could perhaps reference JSON schemas from JSON-LD, or we could define a synthesis of JSON-LD and JSON Schema. Both approaches would be considerably more complex that the approach outlined in this note.

The Web of Things is based upon W3C's Resource Description Framework (RDF), which is very general, and provides the extensibility that we need to meet evolving requirements. In particular, it allows us to define the type system independently of the schema language. We need agreed ways to express the data and interaction model exposed to applications, along with other metadata. Moreover, these ways should be acceptable to the developers, and should be usable on resource constrained devices.

Note that JSON Schema is is not yet a formal standard. It is defined on the json-schema.org website and a set of expired IETF Internet Drafts.

Memory considerations for Thing Descriptions

When an application on one device wants to interact with a remote thing, it can request the platform to create a proxy object for that thing based upon its thing description. Retaining the strings for the names of the properties, actions and events is expensive for resource constrained devices. It is therefore appealing to be able to map these names to numeric symbols which can be used in place of the names, both by the application and for the abstract messages exchanged with the platform hosting the thing. These symbols are scoped to the thing, and the algorithm for the mapping needs to be deterministic way so that all parties agree on the mapping.

For devices where RAM is in short supply, it is appealing to be able to store literals in FLASH memory. This implies the desirability of a way to distinguish what is changeable from what is not. This is analogous to the use of "const" in C++. It would be useful with atomic and compound data. Thing Descriptions should also make it possible for platforms to optimise the storage of data types, e.g. can a number be stored in a single byte or does it need more? Can sets be represented as bit vectors? In summary, Thing Descriptions should enable developers to provide information that platforms can use to optimise the storage and representation of data.

An alternative approach

The description starts with a JSON object with properties for the thing's "properties", "actions" and "events". These are in turn JSON objects with the names for the corresponding properties, actions and events. For example:

{
    "@context" : "http://example.com/foo123",
    "properties" : {
    },
    "actions" : {
    },
    "events" : {
    } 
}

Properties

A property is declared with its name and a JSON object with a set of annotations describing it, e.g.

"temp" : {
    "type" : "number",
    "min" : -20,
    "max" : 100,
    "units" : "celsius"
}

Where min and max act as constraints. Other common constraints include whether a given property is required or optional, whether null is allowed for the value, and for numbers a restriction to integer values. Enumerations could be expressed with an array of distinct values, e.g.

"fruit" : {
   "enum" : ["apple", "pear", "banana"]
}

If you don't need the annotations you can just give the type name as a short cut, e.g.

"switch" : "boolean"

For compound types, you can substitute the type name with a JSON object, e.g.

"location" : {
    "type" : {
       "lat": "number",
       "long" : "number"
    }
}

This can be used recursively for arbitrarily nested properties. Note that this can't be used together with the above short cut to avoid ambiguities between property names and annotations.

For situations where you need many properties with the same rich types, there is a means to declare this type once and refer to it, avoiding the need for redundant declarations, e.g.

"types" : {
    "temperature" : {
        "type" :  "number",
        "min" : -20,
        "max" : 100,
        "units" : "celsius"
    }
}

You can then refer to it using the name you just defined, e.g.

"temp1" : "temperature",
"temp2" : "temperature"

Of course you can use a JSON object when you need to use annotations, e.g. for the sensor location.

Note: an alternative would avoid the need for a separate "types" declaration and instead allow a reference to another property, e.g. "temp2" is declared to have the same type as "temp1".

"temp1" : {
        "type" :  "number",
        "min" : -20,
        "max" : 100,
        "units" : "celsius"
    },
"temp2" : "@temp1"

This could be generalised to allow paths for reuse of the types for sub-properties.

Things as Values

If you want to declare a thing as the value for a given property you use "thing" as the type name, and "uri" for its description, e.g.

"door" : {
    "type" : "thing",
    "uri" : "http://example.com/thing/door12"
}

which refers to a specific door. It may be convenient to use a shared thing description for a set of things, e.g. all of the guest room doors in a hotel. In such cases you can use "model" to refer to the shared description as in:

"door" : {
    "type" : "thing",
    "model" : "http://example.com/thing/door"
    "id" : "door12"
}

Note: I am not sure if "model" is the best name for this, perhaps "class"? Note also that "model" and "id" can also be used a the top level for a thing description.

Events

Events are expressed in the same way as properties.

Actions and Responses

Actions are a little more complicated since you need to specify the type for the actions input and output using "in" and "out" respectively, e.g.

"actions" : {
    "fade" : {
        "in" {
            "value" : "number",
            "time" : number
        }
        "out" : {
       }
    }
}

In this example "out" could have been left out since the action doesn't have any responses. A common annotation for "out" is whether it repeats, i.e. "repeats" : true. Values for events, actions and responses can be things as per the earlier example for properties.

Mapping to RDF

The URI for the thing description is taken as the subject for all of the top level names. These names are mapped to URIs via the context and taken as the predicate. The object for these triples is assigned as a new blank node if the value is a JSON object, or a URI if the value is a type name, in which case the type name is mapped to the URI via the context. This process is recursively applied for nested properties. There is a default context which is applies for names that are not given in the explicit context. As per JSON-LD, you can include "@context" in nested objects with the same operational semantics.

Integrity constraints as expressions

To express richer constraints we may want to explore the use of operator expressions analogous to conditional expressions in programming languages. These could cover constraints that apply across properties, actions and events. It could also include cardinality constraints involving, for example, the number if items in array.

Even richer constraints are best expressed as ontologies for semantic models of particular application domains. A simple example would state if a thing is a temperature sensor then it must define its physical units as one of {kelvin, celsius, fahrenheit}. Such ontologies are analogous to schemas for thing descriptions, whereas a thing description is a schema for the data and interaction model exposed to applications.

[danielpeintner]

I am not commenting on the general structure. I would like to purely focus on the type-system.
Your proposal describes how to use simple types and how to build composed types. This has been done already in the past (e.g, JSON schema) and I think there must be very good reasons to come up with something new. I wonder why should do it again just with a different flavor. I don't really think term differences such as "minimim" vs. "min" really matter.

Our goal should be to

• re-use what exists already (e.g., JSON Schema)
• use the tools build around for validataion
• profit from past experience
  (minimizes the change to get it wrong or to forget anything)

Could you describe which parts are missing in for example JSON schema? Thanks!

[draggett]

We should discuss the type system independently of the schema language so that we can work with platforms that use XML rather than JSON. Some platforms may even operate directly on triples.

You're also forgetting the importance of richer integrity constraints.

As far as I know, JSON Schema has yet to become a formal standard.

We should be exploring a range of approaches and evaluating their merits and weaknesses, both on technical grounds and on their acceptance by a broad range of developer communities. It is too early to say just what the "right" solution should be.

[danielpeintner]

Our idea is to provide a mapping to other formats (e.g., XML) but still use a known schema language as basis.
see https://github.com/danielpeintner/wot/blob/master/proposals/type-system/README.md#mappings-to-encodings (or PR #117)

You're also forgetting the importance of richer integrity constraints.

Do you have examples?

[draggett]

You could start with the wikipedia article on integrity constraints which describes some of the concepts. XML Schema includes some basic constraints, e.g. min/max, enumerations, regular expressions. Referential constraints include those where the accepted value for one property depends on the value of another. An example is a form where the data you need to provide depends upon the answers you gave for earlier questions. For the IoT, this corresponds to variant types where the constraints depend upon which variant applies to a particular instance. Another common constraint is whether something is required or optional. Another common constraint is whether null is allowed as a value. Cardinality constraints typically apply to relationships. See e.g. Adrienne Watt's chapter on integrity rules and constraints. Oracle provide some examples in respect to relational data.

p.s. I recommend that you research this yourself.

[danielpeintner]

This will be my last comment in this thread. We don't seem to understand each other and the discussion is going nowhere.

Referential constraints include those where the accepted value for one property
depends on the value of another.

XML Schema 1.0 does not provide referential constraints. XML Schema 1.1 does, but the spec is supported by very limited implementations.

Once again a warning: I don't think we should try to come up with a new type or grammar system supporting all constraints mentioned. Doing so we will fail !!

p.s. I recommend that you research this yourself.

I can try to research "your" use-cases and requirements and but I will not be successful.

Everyone needs to bring his requirements to the table so that we can list and try to handle them.

[draggett]

Why are you restricting this to what is in XML Schema?

The web of things is all about the data, and integrity constraints are important to robust operation. We can start with the most rudimentary constraints, but we need to allow for extension to richer ones.

The IG doesn't contain everybody, but the web of things is intended to cover a broad range of application domains and platforms. So we need to be diligent about looking at use cases and requirements. if we don't do that well, we will be producing yet another competing standard with a limited scope in terms of application domains and platforms. This is why I've worked so hard on reaching out to external organizations.

Let's have a phone call to talk this through.

[vcharpenay]

Dave, I integrated your proposal in our working document (please consider doing it yourself if you have further modifications). I have several remarks:

• the gain compared to JSON Schema is little in the examples you show. JSON Schema could serve as a basis for a larger model that the W3C could standardize and that includes constraints.
• to me, there is a confusion between values and quantities in your proposal. A physical quantity (e.g. temp) is the association of a value (e.g. 25) and a unit (e.g. °C). Until now, quantities had been defined as properties (and still will be in the future, I hope). Our work on complex types (or, as you put it, on a type system) only focuses on value types, not on quantities. There should not be any reference to units in your examples.
• in your example with "temp1" and "temp2", "temp2" has to be identical to "temp1", right? What is this second property for, then?

Besides that, there are relevant questions in your note. For instance, we had not really considered referencing until now. The last point you brought here was "things as types". Actually, there is nothing that prevents developers from doing that in the current state of the TD model. But this is something we should rather discuss in issue #119.

[draggett]

Physical units are clearly important for interoperability, so I am confused as to why you think we should ignore them.

Thing descriptions are metadata, so units are definitely in scope. Thus x is a property, x is named "foo", x is a "number", x has units "celsius", x is updated every 10 minutes, and so forth. We need to be able to make such declarations for sub-properties, e.g. y is a sub-property of x, y is named "bar", etc. The declarations could be expressed as a long flat list of triples, or it could be expressed hierarchically, reflecting the property/sub-property relationships. I see this is something for us to evaluate with developer communities outside of the IG.

Preparations for the next plugfest are clearly important, but so is the need for a longer term plan towards widely deployed web standards. For that we need to have a plan for engaging with the broader developer communities rather than just seeking a consensus within the IG. We're expecting to launch a Working Group, and its success will be much more likely if built upon the firm foundations that the IG can provide through work on exploring the design space and identifying candidates that have the broadest appeal.

Note that JSON Schema hasn't been standardised as yet, so we couldn't normatively reference it from W3C Recommendations. If we want it to be standardised, we could encourage the people behind it to progress it in the IETF where it is currently defined by expired Internet Drafts. Alternatively, we could encourage a W3C Member Submission and see if there is sufficient interest to drive it along the W3C Recommendation track. We couldn't just adopt it without copyright and IPR declarations by the people that created it.

You said:

in your example with "temp1" and "temp2", "temp2" has to be identical to "temp1", right? What is this second property for, then?

The wording in the issue says:

Note: an alternative would avoid the need for a separate "types" declaration and instead allow a reference to another property, e.g. "temp2" is declared to have the same type as "temp1".

So temp2 is being declared as a different sensor value but with the same type as for temp1. When comes to avoiding redundant declarations of types, you can either have a separate declaration analogous to C's typedef, or you can define the type for some property by reference to another property.

[vcharpenay]

Physical units are clearly important for interoperability, so I am confused as to why you think we should ignore them.

Of course they are important, I was just saying we should ignore them in our current proposal for types. Again, what we are discussing in this thread is not how to describe properties but property values, that is we try to allow more complex values for the JSON key valueType. We already proposed another way to define units for properties as annotation using an external vocabulary. Examples from the Current Practices document show that. Moreover, we plan on integrating JSON Schema with RDF so that we could annotate complex types as well.

So temp2 is being declared as a different sensor value but with the same type as for temp1.

I understand but "temp1" and "temp2" must have something different (maybe expressed through annotation, see e.g. reference in the examples?). The way you declare it prevents any further description of "temp2". Actually, as I illustrated in PR #139, what you mention is also possible with JSON Schema and schema.org, without having this issue.

[draggett]

The way you declare it prevents any further description of "temp2"

False. To annotate a property, you give a JSON object with its annotations as per the earlier examples.

[egekorkan]

Not relevant anymore, propose closing. Mentioned issues are addressed in the TD spec