Comparison with hyperfine

[sharkdp]

Author of hyperfine here, hi!

However, poop does report peak memory usage as well as 5 other hardware counters, which I personally find useful when doing performance testing. Hey, maybe it will inspire the Hyperfine maintainers to add the extra data points!

I only learned about this project today. I didn't know about the call to action — you should have told us 😄. You might have seen this, but for reference, we have an open issue (sharkdp/hyperfine#86) about this and even a early draft PR which I hopefully finish at some point. Having some competition is definitely a nice motivation, so thanks for creating this project. Adding even more performance counters is a very interesting idea (even if it might be a Linux-only feature).

I quickly wanted to comment on the rest of the "Comparison" paragraph:

Poop does not run the commands in a shell. This has the upside of not including shell spawning noise in the data points collected, and the downside of not supporting strings inside the commands.

Hyperfine supports both use cases via the --shell/-N options:

  -S, --shell <SHELL>
          Set the shell to use for executing benchmarked commands. This can be
          the name or the path to the shell executable, or a full command line
          like "bash --norc". It can also be set to "default" to explicitly
          select the default shell on this platform. Finally, this can also be
          set to "none" to disable the shell. In this case, commands will be
          executed directly. They can still have arguments, but more complex
          things like "sleep 0.1; sleep 0.2" are not possible without a shell.
  -N
          An alias for '--shell=none'.

Even when using --shell=none/-N, we support things like hyperfine -N 'echo "hello world"', i.e. we properly handle quoted arguments etc.

Poop treats the first command as a reference and the subsequent ones relative to it, giving the user the choice of the meaning of the coloring of the deltas. Hyperfine always prints the wall-clock-fastest command first.

Hyperfine also supports this using the --reference and --sort options.

      --reference <CMD>
          The reference command for the relative comparison of results. If this
          is unset, results are compared with the fastest command as reference.

      --sort <METHOD>
          Specify the sort order of the speed comparison summary and the
          exported tables for markup formats (Markdown, AsciiDoc, org-mode):
            * 'auto' (default): the speed comparison will be ordered by time and
              the markup tables will be ordered by command (input order).
            * 'command': order benchmarks in the way they were specified
            * 'mean-time': order benchmarks by mean runtime

In general, let us know if you think we should change our defaults, and I'm open to discussing this. For example, maybe --shell=none should be the default.

Also, feel free to close this ticket. I mainly wanted to say hi and make sure that we very much appreciate new ideas in the command-line-benchmarking world 👍

[andrewrk]

Hello! Thanks for the friendly message :)

I've done my best to make the comparison in the readme fully accurate. Please feel free to suggest a rewording via pull request if you feel it is still missing something. I'll close the issue since there is no action item, but we can still continue the discussion here.

Happy hacking!