Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add problem report test #5867

Merged
merged 1 commit into from
Feb 28, 2024
Merged

Add problem report test #5867

merged 1 commit into from
Feb 28, 2024

Conversation

niklasberglund
Copy link
Contributor

@niklasberglund niklasberglund commented Feb 26, 2024

Created test for submitting problem reports. On stagemole all problem reports are ignored and not sent to support, therefor the current implementation fails if not running against stagemole in order to avoid spamming support. Please let me know if you have feedback on this approach. Without any safeguard it would be easy to accidentally run the test against production.

This PR includes some fixes to the MullvadVPNUITests build configuration to make the Staging configuration run - before it was only set up for Debug configuration . continueAfterFailure = false was also moved from AccountTests to BaseUITestCase so that it is set for all tests, not only tests under AccountTests.

To test this PR run testSendProblemReport under SettingsTests. Note that you need to change to run Staging configuration(Debug is the default configuration) and NO_TIME_ACCOUNT_NUMBER and HAS_TIME_ACCOUNT_NUMBER must be set for Staging configuration:

NO_TIME_ACCOUNT_NUMBER[config=Debug] = <production-account-number-here>
NO_TIME_ACCOUNT_NUMBER[config=Staging] = <staging-account-number-here>
HAS_TIME_ACCOUNT_NUMBER[config=Debug] = <production-account-number-here>
HAS_TIME_ACCOUNT_NUMBER[config=Staging] = <staging-account-number-here>

This change is Reviewable

@niklasberglund niklasberglund added the iOS Issues related to iOS label Feb 26, 2024
Copy link

linear bot commented Feb 26, 2024

Copy link
Contributor

@acb-mv acb-mv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 10 of 10 files at r1, all commit messages.
Reviewable status: 0 of 10 files reviewed, all discussions resolved (waiting on @mojganii and @rablador)

Copy link
Contributor

@buggmagnet buggmagnet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 8 of 10 files at r1, all commit messages.
Reviewable status: 0 of 10 files reviewed, 4 unresolved discussions (waiting on @acb-mv, @mojganii, @niklasberglund, and @rablador)


ios/MullvadVPN.xcodeproj/project.pbxproj line 605 at r1 (raw file):

		8529693A2B4F0238007EAD4C /* TermsOfServicePage.swift in Sources */ = {isa = PBXBuildFile; fileRef = 852969392B4F0238007EAD4C /* TermsOfServicePage.swift */; };
		8529693C2B4F0257007EAD4C /* Alert.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8529693B2B4F0257007EAD4C /* Alert.swift */; };
		8532E6872B8CCED600ACECD1 /* ProblemReportSubmittedPage.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8532E6862B8CCED600ACECD1 /* ProblemReportSubmittedPage.swift */; };

Don't forget to order files by name when you add a new file


ios/MullvadVPN.xcodeproj/project.pbxproj line 7485 at r1 (raw file):

		A93A1D062B59145C00F7796C /* Staging */ = {
			isa = XCBuildConfiguration;
			baseConfigurationReference = 852969382B4ED818007EAD4C /* UITests.xcconfig */;

Curious about these changes, was there something that was not configured properly in the xcode project file ?


ios/MullvadVPNUITests/AccountTests.swift line 12 at r1 (raw file):

class AccountTests: LoggedOutUITestCase {
    override func setUpWithError() throws {

⚠️ Unneeded Overridden Functions Violation: Remove overridden functions that don't do anything except call their super (unneeded_override)


ios/MullvadVPNUITests/SettingsTests.swift line 14 at r1 (raw file):

class SettingsTests: LoggedOutUITestCase {
    func testSendProblemReport() throws {
        #if !MULLVAD_ENVIRONMENT_STAGING

Instead of doing this (which also leaves a permanent warning when not running the staging environmnent which is not great)

We can do the following

#if !MULLVAD_ENVIRONMENT_STAGING
let shouldSkipTest = true
#else
let shouldSkipTest = false
#endif

try XCTSkipIf(shouldSkipTest, "This test should only run in the Staging environment")

@niklasberglund niklasberglund force-pushed the test-problem-reports-ios-439 branch from 2a10c87 to 9f652fa Compare February 27, 2024 10:05
Copy link
Contributor Author

@niklasberglund niklasberglund left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 10 files reviewed, 4 unresolved discussions (waiting on @acb-mv, @buggmagnet, @mojganii, and @rablador)


ios/MullvadVPN.xcodeproj/project.pbxproj line 605 at r1 (raw file):

Previously, buggmagnet wrote…

Don't forget to order files by name when you add a new file

Oops, will remember next time. Ordered by name now.


ios/MullvadVPN.xcodeproj/project.pbxproj line 7485 at r1 (raw file):

Previously, buggmagnet wrote…

Curious about these changes, was there something that was not configured properly in the xcode project file ?

The Debug configuration for MullvadVPNUITests target was correctly set up before but not Staging and MockRelease. Previously I've only been running tests against production so was unaffected, but since this test runs on staging I've been using Staging configuration and encountered issues.


ios/MullvadVPNUITests/AccountTests.swift line 12 at r1 (raw file):

Previously, buggmagnet wrote…

⚠️ Unneeded Overridden Functions Violation: Remove overridden functions that don't do anything except call their super (unneeded_override)

Removed 👍


ios/MullvadVPNUITests/SettingsTests.swift line 14 at r1 (raw file):

Previously, buggmagnet wrote…

Instead of doing this (which also leaves a permanent warning when not running the staging environmnent which is not great)

We can do the following

#if !MULLVAD_ENVIRONMENT_STAGING
let shouldSkipTest = true
#else
let shouldSkipTest = false
#endif

try XCTSkipIf(shouldSkipTest, "This test should only run in the Staging environment")

That's much better! Have changed it.

Copy link
Contributor

@rablador rablador left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 7 of 10 files at r1, 3 of 3 files at r2, all commit messages.
Reviewable status: 0 of 10 files reviewed, 4 unresolved discussions (waiting on @acb-mv, @buggmagnet, @mojganii, and @rablador)

Copy link
Contributor

@buggmagnet buggmagnet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 3 of 3 files at r2, all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved

@buggmagnet buggmagnet force-pushed the test-problem-reports-ios-439 branch from 9f652fa to 2abc5ed Compare February 28, 2024 07:57
@buggmagnet buggmagnet merged commit b74b165 into main Feb 28, 2024
4 of 5 checks passed
@buggmagnet buggmagnet deleted the test-problem-reports-ios-439 branch February 28, 2024 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
iOS Issues related to iOS
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants