How to run derailed_benchmark's build locally (for contributing) ?

[rsanheim]

I have a tiny fix for an issue we see in our app that I'd like to contribute back, but I can't get tests to run locally. Are there docs anywhere for how to run the build locally? From the looks of the build failures, I think I'm just missing something with how the rails specific gemfiles interact with running the build.

I grabbed a snippet from #220 to try to run things in what I think is the correct way, but the full suite then errors out with environment issues since it can't find rake. I'm using ruby 3.1.2 installed via asdf on a M1 Macbook Pro, and this is running off latest HEAD:

[~/src/oss/derailed_benchmarks (main)] [1] > git rev-parse HEAD
d9f40e4b26fc87caea4c5c9a91314ce2fba909c0
[~/src/oss/derailed_benchmarks (main)]> which ruby
/Users/rsanheim/.asdf/shims/ruby
[~/src/oss/derailed_benchmarks (main)]> ruby -v
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [arm64-darwin21]
[~/src/oss/derailed_benchmarks (main)]> git rev-parse HEAD
d9f40e4b26fc87caea4c5c9a91314ce2fba909c0

Full output is included below, though I have to link to the build failure in a gist as its too long.

BUNDLE_GEMFILE=gemfiles/rails_6_1.gemfile bundle install

[~/src/oss/derailed_benchmarks (main)]> BUNDLE_GEMFILE=gemfiles/rails_6_1.gemfile bundle install
Using rake 13.0.6
Using concurrent-ruby 1.1.10
Using i18n 1.12.0
Using minitest 5.16.3
Using tzinfo 2.0.5
Using zeitwerk 2.6.4
Using activesupport 6.1.7
Using builder 3.2.4
Using erubi 1.11.0
Using racc 1.6.0
Using nokogiri 1.13.9 (arm64-darwin)
Using rails-dom-testing 2.0.3
Using crass 1.0.6
Using loofah 2.19.0
Using rails-html-sanitizer 1.4.3
Using actionview 6.1.7
Using rack 2.2.4
Using rack-test 2.0.2
Using actionpack 6.1.7
Using nio4r 2.5.8
Using websocket-extensions 0.1.5
Using websocket-driver 0.7.5
Using actioncable 6.1.7
Using globalid 1.0.0
Using activejob 6.1.7
Using activemodel 6.1.7
Using activerecord 6.1.7
Using marcel 1.0.2
Using mini_mime 1.1.2
Using activestorage 6.1.7
Using mail 2.7.1
Using actionmailbox 6.1.7
Using actionmailer 6.1.7
Using actiontext 6.1.7
Using public_suffix 5.0.0
Using addressable 2.8.1
Using bcrypt 3.1.18
Using benchmark-ips 2.10.0
Using bundler 2.3.11
Using xpath 3.2.0
Using capybara 2.18.0
Using dead_end 4.0.0
Using ffi 1.15.5
Using get_process_mem 0.2.7
Using thor 1.2.1
Using heapy 0.2.0
Using memory_profiler 1.0.1
Using mini_histogram 0.3.1
Using ruby-statistics 3.0.1
Using derailed_benchmarks 2.1.2 from source at `..` and installing its executables
Using orm_adapter 0.5.0
Using method_source 1.0.0
Using railties 6.1.7
Using responders 3.0.1
Using warden 1.2.9
Using devise 4.8.1
Using m 1.6.0
Using timeout 0.3.0
Using net-protocol 0.1.3
Using net-imap 0.3.1
Using net-pop 0.1.2
Using net-smtp 0.3.3
Using power_assert 2.0.2
Using sprockets 4.1.1
Using sprockets-rails 3.4.2
Using rails 6.1.7
Using sqlite3 1.5.3 (arm64-darwin)
Using test-unit 3.5.5
Using webrick 1.7.0
Bundle complete! 12 Gemfile dependencies, 69 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
[~/src/oss/derailed_benchmarks (main)]>

Then trying to run the full build:

BUNDLE_GEMFILE=gemfiles/rails_6_1.gemfile bundle exec rake

gives a bunch of output, full log is here: https://gist.github.com/rsanheim/4be23c3049a017c0acfa75f4eec17a44

Here is a snippet of the errors that get repeated:

Running: env TEST_COUNT=5 bundle exec rake -f perf.rake perf:heap_diff --trace
bundler: failed to load command: rake (/Users/rsanheim/.asdf/installs/ruby/3.1.2/bin/rake)
/Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/definition.rb:36:in `build': /Users/rsanheim/src/oss/derailed_benchmarks/test/rails_app/gemfiles/rails_6_1.gemfile not found (Bundler::GemfileNotFound)
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler.rb:197:in `definition'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler.rb:145:in `setup'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/setup.rb:20:in `block in <top (required)>'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/ui/shell.rb:136:in `with_level'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/ui/shell.rb:88:in `silence'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/setup.rb:20:in `<top (required)>'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/cli/exec.rb:56:in `require_relative'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/cli/exec.rb:56:in `kernel_load'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/cli/exec.rb:23:in `run'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/cli.rb:483:in `exec'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/cli.rb:31:in `dispatch'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/cli.rb:25:in `start'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/exe/bundle:48:in `block in <top (required)>'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.11/exe/bundle:36:in `<top (required)>'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/bin/bundle:25:in `load'
	from /Users/rsanheim/.asdf/installs/ruby/3.1.2/bin/bundle:25:in `<main>'
E

Error:
TasksTest#test_heap_diff:
RuntimeError: Expected 'env TEST_COUNT=5 bundle exec rake -f perf.rake perf:heap_diff --trace' to return a success status.
Output: bundler: failed to load command: rake (/Users/rsanheim/.asdf/installs/ruby/3.1.2/bin/rake)

Any help is appreciated, as this library is great and I'd be interested in helping contribute and keep things up to date with latest rails and such. Thanks @schneems!

[schneems]

How to run derailed_benchmark's build locally (for contributing) ? #226

Clone the project:

$ git clone ...

Then in your rails app point to that path:

gem "derailed_benchmarks", path: "../../<wherever>/<you>/<installed_it>"

Then use it in your project as you would otherwise i.e bundle exec derailed ...

That's how I do it. It might be helpful once you get that working to add some docs in the CONTRIBUTING guide.

[rsanheim]

Sorry, I may be misunderstanding...or not stating my question clearly :).

My intent was to run the actual derailed tests inside my local fork of derailed to verify changes I made, independent of projects we are using derailed with. I saw that there is a sample rails project inside derailed, but I wasn't able to figure out how to run the derailed tests against that sample rails project.

For example, I made a small change to a part of stats_from_dir / stats_for_file, and I wanted to verify it works against this rails app inside derailed before submitting a PR.

Is there something I'm missing to run the derailed tests from a fresh clone, beyond bundle exec rake ?

[pcai]

@rsanheim hello from the future. there seems to be a bug here but you can work around it: an absolute path must be specified for BUNDLE_GEMFILE

this works for me:
BUNDLE_GEMFILE=/workspaces/derailed_benchmarks/gemfiles/rails_7_2.gemfile bundle exec rake test

I get a similar error as shown in your logs if i try:
BUNDLE_GEMFILE=gemfiles/rails_7_2.gemfile bundle exec rake test

Even though my pwd is correct. It looks like this code in the repo is throwing it off:

result = Bundler.with_original_env { `cd '#{rails_app_path}' && #{cmd} 2>&1` }

but i havent dug into things enough to know how to address