Skip to content

Latest commit

 

History

History
195 lines (163 loc) · 10.6 KB

gathering_ideas.md

File metadata and controls

195 lines (163 loc) · 10.6 KB

gathering requirements

  • gantt chart timeline with markers to visualize processes and bottlenecks
  • multi level / stacked gantt chart suggested by peter bindels to visualize chronological data that is spaced unevenly (decades vs. seconds)
  • time/frequency analysis (group events per period at plot counts/timebox over time)
  • pluggable input sources
  • pluggable visualizers
  • portability for MacOs, Linux and Windows
  • a 1 microsecond accurate storage will cover most usecases. On very time critical systems where time is measured in nanoseconds (1e-9), picoseconds (1e-12) or femtoseconds (1e-15) optimizations are usually measured/aggragated in loops. This seems to be generally available:
  • the description of the Trace Event Format contains both very good ideas for the storage format and visualization

ideas

limitations of existing tools to consider:

  • online tools (such as perfetto) can (and do) change input format over time and become incompatible with older data (bugs or not, it is nolonger ABI compatible), so consider a 'frozen' offline viewer to unsure that very data can be read anytime. Perfetto can be installed locally, so that can be a way to deal with that.
  • some trace formats (like VCD) require that you know up front what 'signals' are going to be present, this is probablematic, because when 'streaming' logs all possible occurances of names of function/events are not known up front. So this kind of format requires post-processing.
  • some trace format (like chrome's trace.json) require special begin and end markers, [ {"first": "v1"} , {"second": "v2"} ] this may pose a problem when a program crashes and it does not write the end marker.

more...

  • online filters/highlighing storage to sync settings over difference workstations (login with google account?)
  • what do we do with internationalization? Chinese, Japanese characters, etc...

https://simplicable.com/new/information-analysis

  • gather
  • aggregation / correlation
  • map (different sources may provide the same information in slightly different ways)
  • categorization
  • summaries (frequency, totals occurances)
  • analysis
  • visualization

Concepts:

  • multi-in/multi-out
    • in: for example, take multiple logfiles, sockets, or other input
    • out: stream to disk, stream to memory-ring-buffer
  • client/server
    • for example, multiple server instances on linux and a client/viewer on windows (or vice versa)
    • server side buffer (late connecting client can get messages from the past x time)
    • Debugview++ could be one of the 'servers' for windows?
    • consider .dotnet core or java for a front-end visualization, to make the UI multi-platform

Visualization:

communication:

Filter:

  • separation for input and view filter, meaning filtered messages from input filters is not kept at all, and filtered messages from view filters and only not shown. (debugview++ only has the latter, which can make it difficult to setup very long lasting tests.)

Search:

  • search
  • filter !
  • (trigger action)

Possible inputs:

  • compact binary format to collect high-resolution data (telemetry) for games and financial trading applications.
  • VCD (Value Change Dump) an ASCII-based format described by IEEE standard 1364-2001
  • logs from a text formatted file
  • logs from OutputDebugString API (Win32 specfic)
  • logs from http(s)
  • logs from stdin
  • logs from structured CSV (TICS results)
  • logs from binary formatted file (use customizable plugin?)
  • remote log sources, such as ssh/sftp
  • remote instance of the same tool, attach/monitor it remotely
  • [windows specific] Event log !
  • [windows specific] Event Tracing for Windows (ETW) is designed for C and C++ developers who write user-mode applications but can also be used in kernel drivers

Formatting:

  • allow to edit columns, on tabs that read form files, maybe apply restrictions on OutputDebugString/Event log tabs.
  • unformating/parsing by allowing specification of colomn separator or fixed width (per colomn)
  • allow/be agnostic to (log)files without timestamps

Visualize:

  • open event source in new tab, allow to merge tabs
  • listview, coloring / highlighting
  • timelines (gantt charts) with annotations
  • summaries / reports ?
  • scaling from scattered datapoints milliseconds <=> years ?

Outputs:

  • choose filtered / unfiltered
  • to file
  • to socket? (for transport / remote monitoring)

commercial logo?

google keywords

Telemetry is an automated communications process by which measurements and other data are collected at remote or inaccessible points and transmitted to receiving equipment for monitoring.

process mining

We almost certainly have overlap between our goals and the goals from process mining

references

print debugging

site references

TimeScale+Grafana -> grouping by seconds an lowest timeunit may be a problem?

  • Flamegraph?

free similar applications:

non-free similar applications: