From 28d304fdf0c5981dafb1f38772e072232c53368d Mon Sep 17 00:00:00 2001 From: Francesco Robino Date: Fri, 24 Mar 2023 11:55:40 +0100 Subject: [PATCH] [WIP] TSP description, vision and guidelines My interpretation of the TSP. A tentative description of what it is, what it aims to accomplish and the use cases it aims to solve. Some initial guidelines to follow when extending the TSP to accomodate new use cases. --- README.md | 100 ++++++++++++++++++++++++++++++++++++++-- TSP-examples.md | 120 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 215 insertions(+), 5 deletions(-) create mode 100755 TSP-examples.md diff --git a/README.md b/README.md index 6a87e1e..73bf1db 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,105 @@ # trace-server-protocol -Specification of the Trace Server Protocol - -This protocol is built to decouple the backend and frontend of trace analysers, allowing traces to reside and be analysed on the backend, and visual models to be exchanged with a variety of clients. - -The protocol is meant to be RESTful, over HTTP. +Specification of the Trace Server Protocol (TSP). The specification is currently written in **OpenAPI 3.0** and can be pretty-visualized in the [github pages][tspGhPages]. **👋 Want to help?** Read our [contributor guide][contributing]. +## About, why and goals of the TSP: + +Similarly to the philosophy behind the Language Server Protocol (LSP), +the TSP is an open, HTTP-based protocol for use between *analytics or +interactive visualization web applications* and *servers that provide +trace analysis specific features* such as: + +- generation of data that can interpeted as charts from trace analysis +- selection, filtering and correlation of elements in charts using data + contained in the traces +- statistics calculation from data contained in the traces + +The goal of the protocol is to allow trace analysis support to be +implemented and distributed independently of any given analytics or +interactive visualization web application. + +## What the TSP is: + +The TSP is a RESTful API on top of the HTTP protocol that enables to: + +- analyze time series data (e.g. traces and logs); +- return the result of the analysis in form of data that represents + *charts* or *charts components*; +- *navigate (or explore)* through the returned data; + +Summarizing: + +> the TSP is an API to explore data and navigate charts that has been +> generated from the interpretation of time series data. + +With the term *chart* we mean different type of visualizations such as +XY charts, timegraph charts, tables, pie charts, etc. + +With the term *chart components* we mean different elements that +can be used to build or enrich charts such as arrows, annotations, +bookmarks, etc. + +With the terms *exploration and navigation* we mean interactions with +the charts, such as: + +- **filter** specific components of the charts. For example: + - remove from the chart all components that do not match a condition; + - return components of the chart that do match a condition; +- **select** a subset of all data points of a chart. For example: + - given a Timegraph chart, return only the data between a specific + time interval +- **select** specific components of the charts. For example: + - when analyzing the trace of a SW execution, select only some + components of the chart that represent specific SW functions; + - when analyzing the trace of a SW execution, select only some + components of the chart executed on specific CPUs; + - given a Timegraph chart, automatically select areas of the chart + where the components match a specific condition (**zoom-on-crash-area**) +- **correlate** components between charts. For example: + - given 2 charts generated by the analysis of the same time series data, + it enables to correlate the chart components that represent the same + information. For example, given a table containing the max length + of an interval in a time chart, it enables to identify the corresponding + interval in the time chart (**go-to-min/max**) +- **correlate** chart components and their information. For example: + - given a interval in a time chart, extrapolate all information related + to that interval (e.g. **source location**) +- **order** specific parts of the charts. For example: + - return the **N longest** intervals of a timegraph chart; +- **customize** charts and charts components. For example: + - given a table containing statistics from analysis of all input data, + generate a new table containing statistics from analysis of a + subset of the input data; + - given a time chart where intervals represent thread execution over + time, and they are connected using arrows, generate a new time chart + showing the longest path of connected intervals (e.g. **critical path**) + +## Guidelines for the TSP implementation + +The main resources (endpoints) of the API should be: + +- **data sources** (e.g. time series data such as traces and logs). These + are analyzed and used to create the data that can be retrieved; +- **charts** (e.g. tables, XY charts), used to retrieve data representing + a chart; +- **chart components** (e.g. annotations, styles), used to retrieve data + representing specific information for a chart; + +The *exploration and navigation* should be achieved by: + +- parameters in the HTTP message body data (usually when POST-ing), or + as query strings (usually when GET-ing). These parameters should be: + - **data source related**. For example, considering that data sources + have a strong relation to time, a classic parameter can be "time" + related (e.g. *requested_timerange*); + - **chart related**. For example, when "talking" to a resource representing + an XY chart, a parameter can be related to "XY dimensions" (e.g. + *requested_Y_elements*); + ## Current version The current version of the specification is currently implemented and supported in the [Trace Compass trace-server][tcServer] (reference implementation) and what is currently supported by the [tsp-typescript-client][tspClient]. diff --git a/TSP-examples.md b/TSP-examples.md new file mode 100755 index 0000000..1ff179f --- /dev/null +++ b/TSP-examples.md @@ -0,0 +1,120 @@ +# How to follow TSP guidelines + +This file/document will be removed. It is meant to give some examples +of how to follow the TSP guidelines when extending the TSP. + +NOTE: + +- the examples below are just pseudocode to try to give a concrete + feeling on how to implement the exploration/navigation features and + follow the TSP guidelines. They are not meant to be seen as concrete + proposals to extend/modify the existing TSP; +- the GET / POST methods are chosen just to be more RESTful (i.e. try + to not use a POST when we do not create new resources) +- the parameters in the requests are always written as body params, + just to make it easier to write. But they do not have to be body params. + +## Timegraph states (a.k.a. intervals) + +``` +GET tsp/api/experiments/{expUUID}/outputs/timeGraph/{outputId}/states +{"parameters": + {"requested_timerange": + {"start": 111111111, + "end": 222222222, + "nbTimes": 1920}, // to follow "TSP guidelines", would it be better "precision"? or "samples"? + "requested_items": [1,2] // to follow "TSP guidelines", would it be better "requested_row_ids"? + } +} +``` + +## Filter rows of a timegraph + +``` +GET tsp/api/experiments/{expUUID}/outputs/timeGraph/{outputId}/states +{"parameters": + {"requested_timerange": + {"start": 111111111, + "end": 222222222, + "nbTimes": 1920}, // to follow "TSP guidelines", would it be better "precision"? or "samples"? + "requested_row_labels": + [CPU0,CPU2] // this is the label of the rows. How does the cli know about name of rows? + // The cli "asks" to the server about the labels of rows in the chart (e.g. + // from the timegraph/tree endpoint) + } +} +``` + +## Filter states (a.k.a. intervals) of a timegraph chart + +``` +GET tsp/api/experiments/{expUUID}/outputs/timeGraph/{outputId}/states +{"parameters": + {"requested_timerange": + {"start": 111111111, + "end": 222222222, + "nbTimes": 1982}, + "requested_intervals": + [ThreadA*,FunctionB*,BankTransactionC*] + } +} +``` + +## Filter states (a.k.a. intervals) of the chart with fullsearch + +``` +GET tsp/api/experiments/{expUUID}/outputs/timeGraph/{outputId}/states +{"parameters": + {"requested_timerange": + {"start": 111111111, + "end": 222222222, + "nbTimes": inf}, // or "max", however give the idea that we are trying to get all possible samples + "requested_intervals": + [ThreadA*,FunctionB*] + } +} +``` + +## Correlate components between charts (e.g. go-to-max) + +An example of how to jump from a value in a table to the interval/state +that it represents in another chart. + +First, ask some info to the table: + +``` +cli (ask): GET tsp/api/experiments/{expUUID}/outputs///tree + {"parameters":{"requested_times":[0,100000000]}} // Side question: why is "requested_times" needed? +srv (ret): {"headers":[{Min},{Max},...], "entries" :[{1 sec},{2 min},...]} +``` + +In order to implement the "go-to-max" functionality, the client asks to the server +to return the "time series data" (in this case an interval) that was used to calculate +the value "2 min" at the table index 1,1: + +``` +cli (ask): GET tsp/api/experiments/{expUUID}/outputs///tree + {"parameters":{"table_row": 1, "table_col":1]}} +srv (ret): {"data":[{"start": 1234, "end": 2345, "label": "ThreadA"}]} +``` + +The client uses than that info to "go to max" i.e. zoom in, filter or whaterver. +In theory it could also be possible to enable the server to "auto-align" other charts, +not sure if it is a good idea. + +## Customize (e.g. select a subset of the input data to use to create a new chart) + +Cli asks to the "time series input data" (e.g. the event table) to return some info on the input data: + +``` +cli (ask): GET tsp/api/experiments/{expUUID}/outputs/table//columns +srv (ret): {"model":[{"name": "timestamp", "type":"number", ...}, {"name": "Device", "type":"string", ...}, ...]} +``` + +Cli asks to create a new statistics table using a subset of the input data. +The subset is selected using some of the previously returned params. + +``` +cli (ask): POST tsp/api/experiments/{expUUID}/outputs/ + {"parameters":{"outputId/chart-id":"my.custom.chart", "include":[{"Device": "CPU0"},...]}} +```