diff --git a/_posts/2023-11-24-tebako-benchmarking.adoc b/_posts/2023-11-24-tebako-benchmarking.adoc new file mode 100644 index 0000000..c8f2047 --- /dev/null +++ b/_posts/2023-11-24-tebako-benchmarking.adoc @@ -0,0 +1,171 @@ +--- +layout: post +title: "Benchmarking of tebako package against original Ruby applications" +date: 2023-11-24 00:00:00 +0800 +categories: + - tebako + - packaging +author: + name: Maxim Samsonov + email: m.samsonov@computer.org + use_picture: assets + social_links: + - https://github.com/maxirmx +excerpt: >- + Tebako package created from a Ruby application introduces four features that can negatevaly affect performance. + In this post we discuss performance comparison of Teback package and original application and show that negative + impact is minimal. +--- + +:imagesdir: ./assets/benchmarking + += Benchmarking of tebako package against original Ruby applications + +== Rationale + +Tebako is an executable packager. It packages a set of files into a single +executable binary that allows a user to run a selected file from the packaged +software as if it is a mounted filesystem. + +As we discussed in the earlier https://www.tebako.org/blog/2023-02-24-tebako-technology-data-flow/[post] +tebako introduces additional layer for most of functions tha access filesystem. This layer routes calls either to +original implementation or to dwarfs in-memory packaged filesystem. If a call is served by dwarfs tebako consumes +additional resources for decompression. + +If Ruby application includes native extension, its shared library is extracted +to host temporarily folder and loaded from there that may create addional delay. + +Finally Tebako package includes the whole Ruby standard library and all files that were used +to install and build additional Ruby gems and theier native extension. The bigger file may taje additional time for startup. + +These four features may have performance impact that we have estimated and discuss in the present post. + +== Executive summary + +In our very moderate test environment (specified below) we observed the following behavior: + +* Tebako package parses Ruby code faster then native interpreter. Routing from additional file system access layer + reading dwarfs including decompression +is faster then reading from disk, even from SSD. +* External file processing is not noticebly slower for Tebako package despite of additional file system access layer introduced by Tebako +* Tebako package initialization adds a 'penalty' time needed to read a fairly big file that conatains Ruby standard lib. The application not necessarily +uses the whole library but we still need to map it to application memory address space on startup. +* Any native extension or other native code like Java jars in tebako add initialization time because we are decompressing them to temp and loading again + +Tebako package introduces additional time that is consumed during initialization. This time depends on the application size and the size of its native extensions, +it dows not depend on complexity of the data processed by the application or any other volatile factors +In our tests additional time mentioned above varies from 0.03 seconds for 'Hello, world" script to 3 seconds for metanorma application + +== Tests and results + +In order to address different aspects of possible performance issues we ran 4 different tests: + +* Simple "Hello, world!" script that allowes to measure performance of Ruby core +* https://rubygems.org/gems/coradoc[Coradoc gem] that uses Ruby code to process external files +* https://rubygems.org/gems/vectory[Vectory gem] that uses native extension to process external filesystem +* Сomplex https://metanorma.org[metanorma application] that combines uses numerous Ruby gems, native extensions and Jave code + +Simple sript and coradoc, vectory gem tests included several runs with varying repetition of calls. +Metanorma test included execution of various commands that generate various load profiles. + +=== 'Hello, world!' scripts + +``` +if (argv = ARGV).empty? + puts "No arguments given" + exit(1) +end + +if argv[0].to_i < 1 + puts "Argument must be a positive integer" + exit(1) +end + +argv[0].to_i.times do |i| + puts "Hello, world number #{i}!" + puts "Gem path: #{Gem.path}" +end +``` + +image::/assets/blog/hello-world-benchmarking.jpg["Hello, world!" benchmarking results] + +=== Coradoc gem + +``` +if (argv = ARGV).empty? + puts "No arguments given" + exit(1) +end + +if argv[0].to_i < 1 + puts "Argument must be a positive integer" + exit(1) +end + +argv[0].to_i.times do + require "coradoc" + sample_file = File.join(__dir__, "fixtures", "sample.adoc") + require "coradoc/legacy_parser" + Coradoc::LegacyParser.parse(sample_file)[:document] + + require "coradoc/oscal" + sample_file = File.join(__dir__, "fixtures", "sample-oscal.adoc") + document = Coradoc::Document.from_adoc(sample_file) + Coradoc::Oscal.to_oscal(document) + + syntax_tree = Coradoc::Parser.parse(sample_file) + Coradoc::Transformer.transform(syntax_tree) +end +``` +image::/assets/blog/coradoc-benchmarking.jpg[Coradoc gem benchmarking results] + +=== Vectory gem benchmarks + +``` +require "tempfile" + +if (argv = ARGV).empty? + puts "No arguments given" + exit(1) +end + +if argv[0].to_i < 1 + puts "Argument must be a positive integer" + exit(1) +end + +argv[0].to_i.times do + require "emf2svg" + + svg = Emf2svg.from_file(File.join(__dir__, "fixtures", "img.emf")) + + Tempfile.create(["output", ".svg"]) do |tempfile| + tempfile.write(svg) + puts "SVG written to #{tempfile.path}" + end +end +``` + +image::/assets/blog/vectory-benchmarking.jpg[Vectory gem benchmarking results] + +=== Metanorma package benchmarking + +Metanorma application benchmarking included execution of utility commands: ```metanorma help```, ```metanorma version``` +and generation of sample sites (ietf, ieee, iec, iso, iho) using + +``` +metanorma site generate samples -c samples/metanorma.yml -o site- --agree-to-terms +``` + +image::/assets/blog/metanorma-benchmarking.jpg[Metanorma benchmarking results] + +== Benchmarking environment + +``` +Model Name: Mac mini +Model Identifier: Macmini9,1 +Chip: Apple M1 +Total Number of Cores: 8 (4 performance and 4 efficiency) +Memory: 16 GB +Ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [arm64-darwin21] +tebako executable packager 0.5.5 +``` diff --git a/assets/blog/coradoc-benchmarking.jpg b/assets/blog/coradoc-benchmarking.jpg new file mode 100644 index 0000000..5f95458 Binary files /dev/null and b/assets/blog/coradoc-benchmarking.jpg differ diff --git a/assets/blog/hello-world-benchmarking.jpg b/assets/blog/hello-world-benchmarking.jpg new file mode 100644 index 0000000..41b169f Binary files /dev/null and b/assets/blog/hello-world-benchmarking.jpg differ diff --git a/assets/blog/metanorma-benchmarking.jpg b/assets/blog/metanorma-benchmarking.jpg new file mode 100644 index 0000000..25c56ee Binary files /dev/null and b/assets/blog/metanorma-benchmarking.jpg differ diff --git a/assets/blog/vectory-benchmarking.jpg b/assets/blog/vectory-benchmarking.jpg new file mode 100644 index 0000000..820afb3 Binary files /dev/null and b/assets/blog/vectory-benchmarking.jpg differ