Skip to content

Commit

Permalink
[WIP] TSP description, vision and guidelines
Browse files Browse the repository at this point in the history
My interpretation of the TSP. A tentative description of what it is,
what it aims to accomplish and the use cases it aims to solve.

Some initial guidelines to follow when extending the TSP to accomodate
new use cases.
  • Loading branch information
frallax committed Apr 10, 2023
1 parent fa3a7d9 commit 28d304f
Show file tree
Hide file tree
Showing 2 changed files with 215 additions and 5 deletions.
100 changes: 95 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,105 @@
# trace-server-protocol

Specification of the Trace Server Protocol

This protocol is built to decouple the backend and frontend of trace analysers, allowing traces to reside and be analysed on the backend, and visual models to be exchanged with a variety of clients.

The protocol is meant to be RESTful, over HTTP.
Specification of the Trace Server Protocol (TSP).

The specification is currently written in **OpenAPI 3.0** and can be pretty-visualized in the [github pages][tspGhPages].

**👋 Want to help?** Read our [contributor guide][contributing].

## About, why and goals of the TSP:

Similarly to the philosophy behind the Language Server Protocol (LSP),
the TSP is an open, HTTP-based protocol for use between *analytics or
interactive visualization web applications* and *servers that provide
trace analysis specific features* such as:

- generation of data that can interpeted as charts from trace analysis
- selection, filtering and correlation of elements in charts using data
contained in the traces
- statistics calculation from data contained in the traces

The goal of the protocol is to allow trace analysis support to be
implemented and distributed independently of any given analytics or
interactive visualization web application.

## What the TSP is:

The TSP is a RESTful API on top of the HTTP protocol that enables to:

- analyze time series data (e.g. traces and logs);
- return the result of the analysis in form of data that represents
*charts* or *charts components*;
- *navigate (or explore)* through the returned data;

Summarizing:

> the TSP is an API to explore data and navigate charts that has been
> generated from the interpretation of time series data.
With the term *chart* we mean different type of visualizations such as
XY charts, timegraph charts, tables, pie charts, etc.

With the term *chart components* we mean different elements that
can be used to build or enrich charts such as arrows, annotations,
bookmarks, etc.

With the terms *exploration and navigation* we mean interactions with
the charts, such as:

- **filter** specific components of the charts. For example:
- remove from the chart all components that do not match a condition;
- return components of the chart that do match a condition;
- **select** a subset of all data points of a chart. For example:
- given a Timegraph chart, return only the data between a specific
time interval
- **select** specific components of the charts. For example:
- when analyzing the trace of a SW execution, select only some
components of the chart that represent specific SW functions;
- when analyzing the trace of a SW execution, select only some
components of the chart executed on specific CPUs;
- given a Timegraph chart, automatically select areas of the chart
where the components match a specific condition (**zoom-on-crash-area**)
- **correlate** components between charts. For example:
- given 2 charts generated by the analysis of the same time series data,
it enables to correlate the chart components that represent the same
information. For example, given a table containing the max length
of an interval in a time chart, it enables to identify the corresponding
interval in the time chart (**go-to-min/max**)
- **correlate** chart components and their information. For example:
- given a interval in a time chart, extrapolate all information related
to that interval (e.g. **source location**)
- **order** specific parts of the charts. For example:
- return the **N longest** intervals of a timegraph chart;
- **customize** charts and charts components. For example:
- given a table containing statistics from analysis of all input data,
generate a new table containing statistics from analysis of a
subset of the input data;
- given a time chart where intervals represent thread execution over
time, and they are connected using arrows, generate a new time chart
showing the longest path of connected intervals (e.g. **critical path**)

## Guidelines for the TSP implementation

The main resources (endpoints) of the API should be:

- **data sources** (e.g. time series data such as traces and logs). These
are analyzed and used to create the data that can be retrieved;
- **charts** (e.g. tables, XY charts), used to retrieve data representing
a chart;
- **chart components** (e.g. annotations, styles), used to retrieve data
representing specific information for a chart;

The *exploration and navigation* should be achieved by:

- parameters in the HTTP message body data (usually when POST-ing), or
as query strings (usually when GET-ing). These parameters should be:
- **data source related**. For example, considering that data sources
have a strong relation to time, a classic parameter can be "time"
related (e.g. *requested_timerange*);
- **chart related**. For example, when "talking" to a resource representing
an XY chart, a parameter can be related to "XY dimensions" (e.g.
*requested_Y_elements*);

## Current version

The current version of the specification is currently implemented and supported in the [Trace Compass trace-server][tcServer] (reference implementation) and what is currently supported by the [tsp-typescript-client][tspClient].
Expand Down
120 changes: 120 additions & 0 deletions TSP-examples.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
# How to follow TSP guidelines

This file/document will be removed. It is meant to give some examples
of how to follow the TSP guidelines when extending the TSP.

NOTE:

- the examples below are just pseudocode to try to give a concrete
feeling on how to implement the exploration/navigation features and
follow the TSP guidelines. They are not meant to be seen as concrete
proposals to extend/modify the existing TSP;
- the GET / POST methods are chosen just to be more RESTful (i.e. try
to not use a POST when we do not create new resources)
- the parameters in the requests are always written as body params,
just to make it easier to write. But they do not have to be body params.

## Timegraph states (a.k.a. intervals)

```
GET tsp/api/experiments/{expUUID}/outputs/timeGraph/{outputId}/states
{"parameters":
{"requested_timerange":
{"start": 111111111,
"end": 222222222,
"nbTimes": 1920}, // to follow "TSP guidelines", would it be better "precision"? or "samples"?
"requested_items": [1,2] // to follow "TSP guidelines", would it be better "requested_row_ids"?
}
}
```

## Filter rows of a timegraph

```
GET tsp/api/experiments/{expUUID}/outputs/timeGraph/{outputId}/states
{"parameters":
{"requested_timerange":
{"start": 111111111,
"end": 222222222,
"nbTimes": 1920}, // to follow "TSP guidelines", would it be better "precision"? or "samples"?
"requested_row_labels":
[CPU0,CPU2] // this is the label of the rows. How does the cli know about name of rows?
// The cli "asks" to the server about the labels of rows in the chart (e.g.
// from the timegraph/tree endpoint)
}
}
```

## Filter states (a.k.a. intervals) of a timegraph chart

```
GET tsp/api/experiments/{expUUID}/outputs/timeGraph/{outputId}/states
{"parameters":
{"requested_timerange":
{"start": 111111111,
"end": 222222222,
"nbTimes": 1982},
"requested_intervals":
[ThreadA*,FunctionB*,BankTransactionC*]
}
}
```

## Filter states (a.k.a. intervals) of the chart with fullsearch

```
GET tsp/api/experiments/{expUUID}/outputs/timeGraph/{outputId}/states
{"parameters":
{"requested_timerange":
{"start": 111111111,
"end": 222222222,
"nbTimes": inf}, // or "max", however give the idea that we are trying to get all possible samples
"requested_intervals":
[ThreadA*,FunctionB*]
}
}
```

## Correlate components between charts (e.g. go-to-max)

An example of how to jump from a value in a table to the interval/state
that it represents in another chart.

First, ask some info to the table:

```
cli (ask): GET tsp/api/experiments/{expUUID}/outputs/<chart-type>/<chart-id>/tree
{"parameters":{"requested_times":[0,100000000]}} // Side question: why is "requested_times" needed?
srv (ret): {"headers":[{Min},{Max},...], "entries" :[{1 sec},{2 min},...]}
```

In order to implement the "go-to-max" functionality, the client asks to the server
to return the "time series data" (in this case an interval) that was used to calculate
the value "2 min" at the table index 1,1:

```
cli (ask): GET tsp/api/experiments/{expUUID}/outputs/<chart-type>/<chart-id>/tree
{"parameters":{"table_row": 1, "table_col":1]}}
srv (ret): {"data":[{"start": 1234, "end": 2345, "label": "ThreadA"}]}
```

The client uses than that info to "go to max" i.e. zoom in, filter or whaterver.
In theory it could also be possible to enable the server to "auto-align" other charts,
not sure if it is a good idea.

## Customize (e.g. select a subset of the input data to use to create a new chart)

Cli asks to the "time series input data" (e.g. the event table) to return some info on the input data:

```
cli (ask): GET tsp/api/experiments/{expUUID}/outputs/table/<chart-id>/columns
srv (ret): {"model":[{"name": "timestamp", "type":"number", ...}, {"name": "Device", "type":"string", ...}, ...]}
```

Cli asks to create a new statistics table using a subset of the input data.
The subset is selected using some of the previously returned params.

```
cli (ask): POST tsp/api/experiments/{expUUID}/outputs/<chart-type>
{"parameters":{"outputId/chart-id":"my.custom.chart", "include":[{"Device": "CPU0"},...]}}
```

0 comments on commit 28d304f

Please sign in to comment.