Trainer fails when Scan fails

[bdrelling]

Lately, I'm only able to have trainer run successfully when scan succeeds and passes all tests. If even one test fails, I get the following error:

    Unit tests failed

Followed by the sad explosion icon next to trainer and the fastlane finished with errors message.

I wanted to look into this so I saw there was a commit 7 days ago, the only instance of the message I could find the following, from the trainer_action.rb file, lines 11-13:

resulting_paths.each do |path, test_successful|
    UI.test_failure!("Unit tests failed") unless test_successful
end

I can provide any working examples you might need to reproduce this. I'm very stumped after an hour or so of searching for the solution. If my tests pass, trainer runs. If any tests fail, trainer kills the lane.

[KrauseFx]

Are you sure this wasn't always the case? The only change that was recently applied was 002f5d6

[bdrelling]

Actually no, I'm not confident that this is recent. Looking through my Fastfile history I had broken trainer out into another lane previously that was actually being left before reaching it, so it's entirely possible the issue has been around for longer.

This is a bug in trianer though, right?

[bdrelling]

@KrauseFx I just wanted to bring this back up since it seems like a bug. I have the same problem with it failing and have to catch the exception in order for everything to work. I run the scan followed directly by trainer, then it fails and shows the little 💥 mark in the fastlane summary next to trainer having run for 0ms. Bottom of script says:

[22:03:24]: fastlane finished with errors

[!] Unit tests failed
script returned exit code 1

My workaround in the meantime is to just do:

begin
      trainer
rescue => exception
end

And everything actually works fine.

[vytis]

We are also experiencing the same issue. It seems that even when fail_build: true is set (as seen from output), the whole build is flagged as failure and fastlane test exits with status 1:

[15:15:38]: ------------------
[15:15:38]: --- Step: scan ---
[15:15:38]: ------------------

< Xcode output omitted>

** TEST EXECUTE FAILED **
[15:16:20]: Exit status: 65
+--------------------+----+
|      Test Results       |
+--------------------+----+
| Number of tests    | 11 |
| Number of failures | 1  |
+--------------------+----+

[15:16:20]: ---------------------
[15:16:20]: --- Step: trainer ---
[15:16:20]: ---------------------

+------------------+---------------------------------------------------------------------------+
|                                  Summary for trainer 0.8.0                                   |
+------------------+---------------------------------------------------------------------------+
| output_directory | test                                                                      |
| fail_build       | false                                                                     |
| path             | DerivedData/Logs/Test/Test-Wire-iOS-2018.10.04_15-15-56-+0200.xcresult/T  |
|                  | estSummaries.plist                                                        |
| extension        | .xml                                                                      |
+------------------+---------------------------------------------------------------------------+

[15:16:20]: `Helper.gem_path` is deprecated. Use the `ROOT` constant from the appropriate tool module instead.
Successfully generated 'test/TestSummaries.xml'
+----------------------------+---------------------------------------------------------------------------+
|                                              Lane Context                                              |
+----------------------------+---------------------------------------------------------------------------+
| DEFAULT_PLATFORM           | ios                                                                       |
| PLATFORM_NAME              | ios                                                                       |
| LANE_NAME                  | ios test                                                                  |
| SCAN_DERIVED_DATA_PATH     | DerivedData                                                               |
| SCAN_GENERATED_PLIST_FILES | ["DerivedData/Logs/Test/Test-Wire-iOS-2018.10.04_15-15-56-+0200.xcresult  |
|                            | /TestSummaries.plist"]                                                    |
| SCAN_GENERATED_PLIST_FILE  | DerivedData/Logs/Test/Test-Wire-iOS-2018.10.04_15-15-56-+0200.xcresult/T  |
|                            | estSummaries.plist                                                        |
+----------------------------+---------------------------------------------------------------------------+
[15:16:20]: Unit tests failed

+------+------------------+-------------+
|           fastlane summary            |
+------+------------------+-------------+
| Step | Action           | Time (in s) |
+------+------------------+-------------+
| 1    | opt_out_usage    | 0           |
| 2    | default_platform | 0           |
| 3    | scan             | 41          |
| 💥   | trainer          | 0           |
+------+------------------+-------------+

[15:16:20]: fastlane finished with errors

[!] Unit tests failed

$> echo $status
1

We are running fastlane 2.105.2 and trainer 0.8.0 with Xcode 10.

[rist]

We have the same issue with 2.108.0 and fail_build set to false

+-------------------------+---------+---------+
|                Used plugins                 |
+-------------------------+---------+---------+
| Plugin                  | Version | Action  |
+-------------------------+---------+---------+
| fastlane-plugin-trainer | 0.4.1   | trainer |
+-------------------------+---------+---------+

.....

+----------------------------+-----------------------------------------------------------------------------------------+
|                                               Summary for scan 2.108.0                                               |
+----------------------------+-----------------------------------------------------------------------------------------+
| devices                    | ["iPhone XS"]                                                                           |
| skip_testing               | ["GeorgeGoAPI-Unit-Tests/AustrianLoginAPITests"]                                        |
| scheme                     | George Go Debug                                                                         |
| fail_build                 | false                                                                                   |
| workspace                  | ./George Go.xcworkspace                                                                 |
| derived_data_path          | /Users/ristd/Library/Developer/Xcode/DerivedData/George_Go-cejywaflmbkysxasspomufxpouav |
| skip_detect_devices        | false                                                                                   |
| clean                      | false                                                                                   |
| skip_build                 | false                                                                                   |
| output_directory           | ./fastlane/test_output                                                                  |
| buildlog_path              | ~/Library/Logs/scan                                                                     |
| include_simulator_logs     | false                                                                                   |
| disable_concurrent_testing | false                                                                                   |
| should_zip_build_products  | false                                                                                   |
| result_bundle              | false                                                                                   |
| open_report                | false                                                                                   |
| skip_slack                 | false                                                                                   |
| slack_only_on_failure      | false                                                                                   |
| use_clang_report_name      | false                                                                                   |
| reinstall_app              | false                                                                                   |
| xcode_path                 | /Applications/Xcode.app                                                                 |
+----------------------------+-----------------------------------------------------------------------------------------+

[15:29:43]: Unit tests failed

+------+--------------------+-------------+
|            fastlane summary             |
+------+--------------------+-------------+
| Step | Action             | Time (in s) |
+------+--------------------+-------------+
| 1    | xcversion          | 0           |
| 2    | clear_derived_data | 15          |
| 3    | scan               | 483         |
| 💥   | trainer            | 0           |
+------+--------------------+-------------+

[15:29:43]: fastlane finished with errors

[!] Unit tests failed