FrameTime
=========

Measure the time your computer/application/monitor takes from receiving
a keypress to displaying something different.

Why?
----

I believe response time is hugely important for the _feel_ of software. To that
end I'm interested in measuring and comparing the response time of different
applications. In the past I've used typometer[1], Which is easy to use and
requires almost no setup.

Typometer has a pretty severe limitation however. It only measures the time
until another application can read the pixel, a value I ultimately don't care
about. Instead I want to know the entire roundtrip time. From me pressing the
key, to my eye being able to see the change on the monitor. No software only
solution can provide that.

Nvidia has the LDAT (Latency Display Analysis Tool) as part of their _reviewer
toolkit_[2], but to my knowledge only people with platforms large enough to
provide PR for Nvidia can get a hold of that. I'd have a hard time trusting
a measurement device I can't audit anyway.

FrameTime is an alternative to these options. It's cheap and available: You can
make it yourself for around $20. It's open: So you can audit and fix it. It's
external: So it captures end-to-end. It's composable: So you can automate it
and use it has part of your build pipeline.

[1]: https://pavelfatin.com/typometer/
[2]: https://www.nvidia.com/en-us/geforce/news/nvidia-reviewer-toolkit/

How it works
------------

The hardware emulates a USB HID keyboard and a serial device. The serial device
is used for control signal to configure the measurement and start it. When the
measurement begins the device starts a timer and sends a keystroke. Using
a photodiode the device continuously monitors the light level of the monitor
for some time while streaming the measurements and their timings over the
serial interface. When measurement is completed (after a set amount of time)
the device then sends a reset keystroke and emits a terminator on the serial
bus. The measurement can then be analyzed after the fact.

FrameTime consists of a firmware, which is flashed to the teensy, and a client,
which talks to the firmware and works with the results. The firmware is should
be compilable using the Makefile and the avr-gcc toolchain. to flash the
resulting hex file I use teensy-loader-cli as such:

    teensy-loader-cli --mcu atmega32u4 main.hex

The teensy should then reboot and be ready. You will see that it will register
itself as an USB HIDevice.

The client is written in Python. To use it, place the device on the screen and
run the command

    client.py -s <sample_count> -d <delay> -o data.measure

The sample_count is the number of tests you want to run, and the delay is the
time between tests. Measurements will be written to data.measure.

You can then analyze the results with

    analyze.py data.measure --header

Which will output some interesting statistics. An example can be see in this
analysis data collected with xterm

                   title     signal    lag_min  lag_delta  rise_mean rise_stddev
                   xterm     19.106  43973.000 142593.000  62733.758   15776.126

analyse.py can take multiple measure files at once to output an analysis of all
of them.

Example
-------

We can compare the performance of cat on xterm and sublime text 3. We do this
by opening each application with a large font size and a black background.
For xterm this is accomplished with arguments

    $ xterm -fa monaco -fs 100 -bg black

For sublime text we have to modify the settings. This is left as an extersize
to the reader.

We then position the measurement device such that the light sensor will recieve
more light after a press of the 'a' button. For xterm this means we place the
light sensor on top of the new cursor position. For sublime it means we place
it on top of the a letter. We then run the client

    $ sleep 10; client.py -s 10000 -d 0.04 -o xterm.measure
    $ sleep 10; client.py -s 10000 -d 0.04 -o subl.measure

The initial sleep is used to allow us to move focus back to the window we want
to test.

After both tests have run we analyze the results

    $ analyze.py xterm:data3.measure sublime:subl.measure --header -t .01
                   title     signal    lag_min  lag_delta  rise_mean rise_stddev
                   xterm     19.106  48423.000 126472.000  62733.758   15776.126
                 sublime     16.358  97788.000 133648.000  52678.657   18754.689

The results show us that sublime text 3 seems to take 50 000 cycles more to
display our letter. Since the teensy runs at 16MHz we can convert it to seconds

    $ units 50000/16Mhz
        Definition: 0.003125 s

So sublime text 3 is about 3 milliseconds slower than xterm

Hardware
-------

The intended (and tested) hardware is a teensy2 with a SFH206K photodiode
connected between pin D4 and ground. Theoretically any atmega32u4 based
microprocessor should work.

Teensy 2.0 - $16     https://www.pjrc.com/store/teensy.html
SFH206K    - $ 1.37  https://www2.mouser.com/productdetail/osram-opto-semiconductors/sfh-206-k?qs=K5ta8V%252bWhtZrKhv9kwxagw==
Enclosure  - $ 1.00  Assumes you have a 3D printer handy
===================
Total      - $18.37

Attribution
-----------

The USB portion of this code is largely cobbled together from the examples
provided by pjrc for the teensy, their copyright notice is included in the
LICENSE.pjrc file.

Original work is licensed under the GPLv3. A complete copy of the license can
be found in LICENSE.