Backport 17203 and 17267 9.0

[yaauie]

Release notes

From #17203:

• FIX: The plugin manager now correctly cleans up plugins and gem dependencies that are no longer needed after invoking install, remove, or update.
• FIX: The plugin manager's update command no longer fails to upgrade plugins that were initially installed with --version flag

What does this PR do?

Combined clean backports of #17203 and #17267 to 9.0

#17203 includes a regression to the behavior of bin/logstash-plugin install's --preserve flag, which was addressed in #17267.

Why is it important/What is the impact to the user?

• For users who update, install, or remove plugins, cleaning out deactivated dependencies can reduce the disk footprint of the logstash installation
• For building Logstash artifacts, this allows us to build a true subset of the dependency graph after removing plugins that are not needed in a minimalized artifact

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

From #17203:

1. Installing a plugin should remove overwritten plugins and orphaned dependencies

   find vendor/bundle/jruby/*/* -depth 1 | sort > vendor_snapshot.txt
   bin/logstash-plugin install --version 4.1.2 logstash-output-email
   find vendor/bundle/jruby/*/* -depth 1 | sort > vendor_compare.txt
   diff -u vendor_snapshot.txt vendor_compare.txt
   

2. Updating a plugin should remove the previous version (and its shared dependencies)

   bin/logstash-plugin install --version 4.1.2 logstash-output-email
   find vendor/bundle/jruby/*/* -depth 1 | sort > vendor_snapshot.txt
   bin/logstash-plugin update logstash-output-email
   find vendor/bundle/jruby/*/* -depth 1 | sort > vendor_compare.txt
   diff -u vendor_snapshot.txt vendor_compare.txt
   

3. Removing a plugin should remove the plugin gem and its unshared dependencies:

   find vendor/bundle/jruby/*/* -depth 1 | sort > vendor_snapshot.txt
   bin/logstash-plugin remove logstash-integration-aws
   find vendor/bundle/jruby/*/* -depth 1 | sort > vendor_compare.txt
   diff -u vendor_snapshot.txt vendor_compare.txt
   

From #17267 (edited to be relevant to this PR):

1. check out the branch
2. revert the fixing [behavior] and observe test failures:
   back-out the portion of the breaking commit

   git diff 6b1160199ce8b7f9167dcb1f645d5e863ff3f150^! -- lib/pluginmanager/install.rb | patch --strip 1 --reverse
   ci/integration_tests.sh specs/cli/install_spec.rb
   

3. revert the portion of Pluginmanager clean after mutate #17203 that caused the regression, and observe that tests pass:

   git diff f8a6b71d20c880e5f73b598f84120a34905dc6b9^! -- lib/pluginmanager/gemfile.rb | patch --strip 1 --reverse
   ci/integration_tests.sh specs/cli/install_spec.rb
   

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 6b11601