Magento Functional Testing Framework Changelog

3.7.0

GitHub Pull Requests:

#842 -- Eliminated AspectMock from FileStorageTest
#843 -- Eliminated AspectMock from ObjectExtensionUtilTest
#844 -- Eliminated AspectMock from TestObjectHandlerTest
#845 -- Eliminated AspectMock from SuiteObjectHandlerTest
#846 -- Eliminated AspectMock from ActionGroupObjectTest
#847 -- Removed not used mocked object
#848 -- Eliminated AspectMock usage from ActionObjectTest
#850 -- Eliminated AspectMock from TestGeneratorTest
#852 -- Eliminated AspectMock from ModuleResolverTest
#853 -- Eliminated AspectMock from PersistedObjectHandlerTest
#855 -- Eliminated AspectMock from OperationDataArrayResolverTest
#856 -- Eliminated AspectMock from DataExtensionUtilTest
#857 -- Eliminated AspectMock from ParallelGroupSorterTest
#859 -- Eliminated AspectMock usage from SuiteGeneratorTest
#861 -- Eliminated aspect mock from mock module resolver builder
#862 -- Eliminated AspectMock where it was imported but never used
#863 -- Eliminated AspectMock from MagentoTestCase
#864 -- Eliminated AspectMock usage from TestLoggingUtil
#865 -- Eliminated aspect mock from object handler uti
#866 -- Added access/secret key config parameters
#867 -- Added empty query and fragment testing to the UrlFormatterTest
#868 -- PHP 8 support - fix code related to changes in CURL
#869 -- The squizlabs/php_codesniffer composer dependency has been updated to version 3.6.0
#870 -- Removing the csharpru/vault-php-guzzle6-transport not needed dependency
#871 -- Changed loose comparisons into strict
#872 -- Fix broken MFTF tests

3.6.1

Enhancements

Maintainability
- Updated allure dependencies to pull package from new repo allure-framework/allure-php-api.

3.6.0

Enhancements

Maintainability
- Updated composer dependencies to be PHP 8 compatible with the except of codeception/aspect-mock.

GitHub Pull Requests:

#830 -- Add ability to configure multiple OTPs
#832 -- Updated monolog/monolog to ^2.2
#833 -- Removed usage of AspectMock in FilesystemTest
#834 -- Removed usage of AspectMock in AnnotationsCheckTest
#838 -- Removed usage of AspectMock in DeprecatedEntityUsageCheckTest
#841 -- Removed usage of AspectMock in GenerationErrorHandlerTest
#854 -- Updated "monolog" to the latest version 2.3.1

3.5.1

GitHub Pull Requests:

#825 -- Update allure-codeception in order to support php8

3.5.0

Enhancements

Customizability
- Added new config:parallel --groups option in generate:tests command to generate and split tests/suites into required number of execution time balanced groups.

Fixes

Added --no-sandbox chrome option in functional suite configuration.

GitHub Pull Requests:

#824 -- Fix typo in introduction.md
#816 -- Update mftf.md
#812 -- Added examples and modified url links in assertions.md

3.4.0

Enhancements

Maintainability
- Added support for composer 2.

3.3.0

Enhancements

Usability
- #817 -- Add support for admin WebAPI token refresh.
Maintainability
- #814 -- Update dependencies in order to make mftf php8 compatible, fix running phpcpd
- #815 -- Upgrade csharpru/vault-php to 4.1

Fixes

Fixed test generation error in a split suite group (--config=parallel) to allow generation of subsequent groups.
Fixed an issue where test extends from a skipped parent is not properly skipped.

3.2.1

Fixes

Fixed issue that causes Magento bin/magento to fail when xdebug 3 is used. GitHub Issue #808

GitHub Pull Requests:

#806 -- Enable an extending entity to overwrite a requiredEntity binding
#809 -- Add MFTF documentation for AWS S3

3.2.0

Enhancements

Usability
- Introduced error tolerance during test and suite generation. See the command page for details. Addressed github issue #276.
Maintainability
- Updated annotation static-check to check all required annotations.

Fixes

Fixed issue where CUSTOM_MODULE_PATHS env variable does not use all paths.
Fixed issue where run:test only records the last failed test in failed file.

3.1.1

Traceability
- Removed travis.yml and replaced with .github/workflows/main.yml

Fixes

Fixed issue with XPath locators for waits in MagentoPwaWebDriver.

3.1.0

Enhancements

Customizability
- Introduced the new return action that allows action groups to return a value. See the actions page for details.
- Introduced new MFTF command that invokes vendor/bin/codecept run. See the mftf page for details.
Usability
- Introduced new action pause, to invoke codeception interactive pause for debugging during test execution. See the Interactive Pause page for details.
- Introduced a new .env configuration option ENABLE_PAUSE, to enable the new pause feature.
Maintainability
- Added a new static check that checks for the usage of the pause action. See the command page for details.
Modularity
- MFTF now supports the use of actions from multiple modules within suites.
Traceability
- A deprecation notice is now added at test execution time for deprecated metadata usage.

Fixes

Fixed issue with suite precondition failure for createData with required entity.

GitHub Issues/Pull Requests:

#547 -- Fix invalid behavior of MAGENTO_BACKEND_BASE_URL
#742 -- Fix Waits In MagentoPwaWebDriver
#750 -- Docs: Outlining the difference between Allure severity levels
#683 -- Docs: Renamed sample test name with the correct one
#691 -- Docs: Branch name updates
#678 -- Docs: Command added to modify the web server rewrites configuration
#745 -- Docs: Remove invalid sample test name

3.0.0

Enhancements

Customizability
- Introduced MFTF helpers <helper> to create custom actions outside of MFTF.See custom-helpers page for details
- Removed deprecated actions <executeSelenium> and <performOn>.
- <group value="skip"/> no longer skips a test. Instead, the test is added to the skip group.
Maintainability
- Added support for PHP 7.4.
- Added support for PHPUnit 9.
- Dropped support for PHP 7.0, 7.1, 7.2.
- Schema updates for test entities to only allow single entity per file except Data and Metadata.
- Support for sub-folders in test modules.
- Removed support to read test entities from <magento>dev/tests/acceptance/tests/functional/Magento/FunctionalTest.
- Removed file attribute for <module> in suiteSchema.
- Removed action pauseExecution and added pause. See actions page for details
- Removed action formatMoney and added formatCurrency. See actions page for details
- Improved assertion actions to support PHPUnit 9 changes. See assertions page for details
  - Added new actions: assertEqualsWithDelta, assertNotEqualsWithDelta, assertEqualsCanonicalizing, assertNotEqualsCanonicalizing, assertEqualsIgnoringCase, assertNotEqualsIgnoringCase.
  - Added new actions: assertStringContainsString, assertStringNotContainsString, assertStringContainsStringIgnoringCase, assertStringNotContainsStringIgnoringCase for string haystacks.
  - Removed actions: assertInternalType, assertNotInternalType, assertArraySubset.
  - Removed delta option from assertEquals and assertNotEquals.
- Added static check deprecatedEntityUsage that checks and reports references to deprecated test entities.
- Added static check annotations that checks and reports missing annotations in tests.
- Updated bin/mftf static-checks command to allow executing static-checks defined in staticRuleSet.json by default. See command page for details
- Added support for Two-Factor Authentication (2FA). See configure-2fa page for details
- Added new upgrade script to remove unused arguments from action groups.
- mftf.log no longer includes notices and warnings at test execution time.
- Added unhandledPromptBehavior driver capability for Chrome 75+ support.
- Added the Chrome option --ignore-certificate-errors to functional.suite.dist.yml.
Traceability
- Removed --debug option NONE to disallow ability to turn off schema validation.
- Notices added for test entity naming convention violations.
- Metadata file names changed to *Meta.xml.
- Introduced new .env configuration VERBOSE_ARTIFACTS to toggle saving attachments in Allure. See configuration page for details
- Changed the bin/mftf static-checks error file directory from the current working directory to TESTS_BP/tests/_output/static-results/.
Readability
- Support only nested assertion syntax See assertions page for details.
- Documented 3.0.0 Backward Incompatible Changes.
- Removed blacklist/whitelist terminology in MFTF.
Upgrade scripts added to upgrade tests to MFTF major version requirements. See upgrade instructions below.
Bumped dependencies to support PHP/PHPUnit upgrade.

Upgrade Instructions

Run bin/mftf reset --hard to remove old generated configurations.
Run bin/mftf build:project to generate new configurations.
Run bin/mftf upgrade:tests. See command page for details.
After running the above command, some tests may need manually updates:
- Remove all occurrences of <executeInSelenium> and <performOn>.
- Remove all occurrences of <module file=""/> from any <suite>s.
- Ensure all <assert*> actions in your tests have a valid schema.
Lastly, try to generate all tests. Tests should all be generated as a result of the upgrades.
- If not, the most likely issue will be a changed XML schema. Check error messaging and search your codebase for the attributes listed.

Fixes

Throw exception during generation when leaving out .url for amOnPage.
request_timeout and connection_timeout added to functional.suite.yml.dist.
Fixed ModuleResolver to resolve test modules moved out of deprecated path.
Fixed issue of resolving arguments of type entity in action groups within a custom helper.
Fixed reporting issue in output file for testDependencies static check.
Fixed a bug in actionGroupArguments static check when action group filename is missing ActionGroup.
Fixed issue of running suites under root _suite directory in Standalone MFTF.
Fixed issue with custom helper usage in suites.
Fixed issue with decryption of secrets during data entity creation.
Fixed issue with merging of array items in data entity.
Fixed issue where an extended data entity would not merge array items. Array items should merge properly now.
Fixed issue where Chrome remains running after MFTF suite finishes.
Fixed javascript error seen on Chrome 83 for dragAndDrop action.
Fixed allure issue when WebDriverCurlException is encountered in afterStep.

GitHub Issues/Pull Requests

#567 -- log attachments for failed requests.

Demo Video links

MFTF 3.0.0 RC1
MFTF 3.0.0 RC2
MFTF 3.0.0 RC3

2.6.4

Fixes

added dependency to packages MFTF used but never specified in composer.json

2.6.3

New Feature

--filter option was added to bin/mftf generate:tests command. For more details please go to https://devdocs.magento.com/mftf/docs/commands/mftf.html#generatetests

2.6.2

Fixes

Fixed float conversion error in test generation

2.6.1

Usability
- Introduced new .env configuration ELASTICSEARCH_VERSION to support multiple elasticsearch versions
Maintainability
- Added deprecation notices for upcoming MFTF 3.0.0
Replaced facebook webdriver with php-webdriver to support PHP version updates

2.6.0

Usability
- magentoCron action added by community maintainer @lbajsarowicz
Traceability
- MFTF generated cest files are fully compatible for Codeception dry-run.
Modularity
- mftf generate:tests and mftf run:test commands now accept suite scoped test names in format [suitename:testname]....
Maintainability
- Support deprecated syntax for the following test entities:
  - Test
  - Action Group
  - Data
  - Metadata
  - Page
  - Section
  - Section Element
    - See DevDocs for details
- Improved mftf static-checks command to allow executing all or specific static checks.
- Added a new static check that checks and reports unused arguments in action groups.
Customizability
- AWS Secrets Manager has been added as an additional credential storage.
  - See DevDocs for details
Bumped dependencies to latest possible versions

Fixes

Fixed missing before, after, failed steps in cest file when generating tests with --allow-skipped option.
Fixed suites and tests display issue in Allure Suites page after mftf run:group command.
createData action now shows a meaningful error message at runtime when the entity does not exist.

GitHub Issues/Pull requests:

#537 -- Refactor of TestGenerator class
#538 -- FEATURE: command to execute Cron Jobs

2.5.4

Demo Video

Traceability
- Introduced new mftf doctor command
  - Command verifies and troubleshoots some configuration steps required for running tests
  - Please see DevDocs for more details
- <*Data> actions now contain API Endpoint and Request Header artifacts.
- Introduced new .env configurations ENABLE_BROWSER_LOG and BROWSER_LOG_BLOCKLIST
  - Configuration enables allure artifacts for browser log entries if they are present after the step.
  - Blocklist filters out logs from specific sources.
Customizability
- Introduced timeout="" to magentoCLI actions.

GitHub Issues/Pull requests:

#317 -- RetrieveEntityField generation does not consider ActionGroup as part of namespace
#433 -- Add possibility to include multiple non primitive types in an array

Fixes

A test now contains attachments for every exception encountered in the test (fix for a test <after> exception overriding all test exceptions).
Fixed hard requirement for MAGENTO_BASE_URL to contain a leading /.
magentoCLI actions for config:sensitive:set no longer obscure CLI output.
WAIT_TIMEOUT in the .env now correctly sets pageload_timeout configuration.
Fixed an issue where run:group could not consolidate a group that had tests in and out of <suite>s.

2.5.3

Fixes

Fixed an issue where createData actions would cause an exception when used in <suite> hooks.

2.5.2

Traceability
- Allure report enhanced to display file path of tests.
Maintainability
- Added support to read MFTF test entities from <magento>dev/tests/acceptance/tests/functional/<vendor_name>/<module_name>/*
- Removed path deprecation warning from ModuleResolver.
- Refactored problem methods to reduce cyclomatic complexity.

Fixes

Fixed issue with builds due to absence of AcceptanceTester class.

GitHub Issues/Pull requests:

#348 -- executeInSelenium command fixed to prevent escaping double quotes.

2.5.1

Fixes

Fixed missing use statement in the generate:suite command

GitHub Issues

#471 -- PHP Fatal error: MftfApplicationConfig not found in GenerateSuiteCommand

2.5.0

Traceability
- Allure output has been enhanced to contain new test artifacts created and used per MFTF step:
  - makeScreenshot will contain an attachment under its Allure step.
  - seeInCurrentUrl and all other Url asserts now contain an attachment with the expects vs actual comparison.
  - createData and all other Data actions now contain attachments with Request Body and Response Body.
Modularity
- Added a new mftf run:manifest command to run testManifest files generated by generate:tests.
  - See DevDocs for details
- mftf generate/run:test commands now implicitly generates the suite the test exists in.
  - If a test exists in multiple suites, it will generate it in all suite contexts.
- mftf run:test <testName> will now only run the exact test provided, regardless of what is generated.
Maintainability
- Added an --allow-skipped flag that allows MFTF to ignore the <skip> annotation. This was added to the following commands:
  - generate:test
  - run:test
  - run:group
  - run:failed
Customizability
- <entity> defined in data.xml can now reference other <entity> directly.
  - See DevDocs for details
- Added vault as an alternative credential storage.
  - See DevDocs for details

Fixes

Fixed an issue where grab action variables were not substituting correctly when used as an element parameter.
Framework will not throw a descriptive error when referencing a $persisted.field$ that does not exist.
MFTF test materials that extends="" itself will no longer cause infinite recursion.
Fixed an issue where a test could not reference a $data.field$ whose casing was modified by the API that it used.
Fixed an issue with the default functional.suite.yml where it was incompatible with symfony/yaml 4.0.0.
Improved test generation performance via class refactors (~10% faster).

GitHub Issues/Pull requests:

#377 -- Non-API operations fixes

2.4.5

Fixes

Fixed an issue where .credentials was required when using <createData> actions with field overrides.

2.4.4

Fixes

Fixed an issue where _CREDS could not be resolved when used in a suite.

2.4.3

Customizability
- Use of _CREDS has been extended to <magentoCLI> and <createData> actions
Traceability
- A Test step is now generated and injected in the allure report when a test is reported as BROKEN.

Fixes

static-checks command now properly returns 1 if any static check failed.
MFTF Console Printer class correctly utilizes --steps and other flags passed directly to codecept commands.
*source actions correctly print when using userInput or html attributes.
XML Comments should no longer throw an error in parsing when used outside test/actionGroup

GitHub Issues/Pull requests:

#703 -- SMALL_CHANGE: Minor change suggested

2.4.2

Traceability
- Test action stepKeys are now included in both console output and Allure report.
- XML Comments are now automatically converted into a <comment> action.

Fixes

Moved epfremme/swagger-php dependency to suggests block due to a conflict with Magento extensions.

2.4.1

Traceability
- XSD Schema validation is now enabled by default in generate:tests, run:test, run:failed, run:group
- --debug option for the above commands has been updated to include different debug levels
  - See DevDocs for details

Fixes

Fixed an issue where skipReadiness attribute would cause false XSD Schema validation errors.

2.4.0

Enhancements

Maintainability
- Added new mftf static-checks command to run new static checks against the attached test codebase
  - See DevDocs for details
- Added new mftf generate:docs command that generates documentation about attached test codebase
  - See DevDocs for details
Traceability
- Allure reports for tests now contain collapsible sections for actionGroups used in execution.

Fixes

Fixed an issue where magentoCli would treat argument="0" as a null value.
Fixed an issue where amOnPage and waitForPwaElementVisible would not utilize the timeout attribute correctly when MagentoPwaWebDriver is enabled.
Fixed an issue where invalid XML characters would cause Allure to throw an exception without a resulting report.
Fixed codeception.dist.yml configuration for keeping previous test run results.
PHP Notices are no longer thrown when XML is missing non-necessary attributes.
Removed unusable fillSecretField action from schema.

GitHub Issues/Pull requests:

#338 -- Return exit codes of process started by 'run:test', 'run:group' or 'run:failed' command
#333 -- Added Nginx specific settings to getting started doc
#332 -- executeInSelenium action does not generate proper code
#318 -- Reduce cyclomatic complexity in Problem Methods
#287 -- Update requirements to include php7.3 support

2.3.14

Enhancements

Maintainability
- command.php is now configured with an idleTimeout of 60 seconds, which will allow tests to continue execution if a CLI command is hanging indefinitely.

2.3.13

Enhancements

Traceability
- Failed test steps are now marked with a red x in the generated Allure report.
- A failed suite <before> now correctly causes subsequent tests to marked as failed instead of skipped.
Customizability
- Added waitForPwaElementVisible and waitForPwaElementNotVisible actions.
Modularity
- Added support for parsing of symlinked modules under vendor.

Fixes

Fixed a PHP Fatal error that occurred if the given MAGENTO_BASE_URL responded with anything but a 200.
Fixed an issue where a test's <after> would run twice with Codeception 2.4.x
Fixed an issue where tests using extends would not correctly override parent test steps
Test actions can now send an empty string to parameterized selectors.

GitHub Issues/Pull requests:

#297 -- Allow = to be part of the secret value
#267 -- Add PHPUnit missing in dependencies
#266 -- General refactor: ext-curl dependency + review of singletones (refactor constructs)
#264 -- Use custom Backend domain, refactoring to Executors responsible for calling HTTP endpoints
#258 -- Removed unused variables in FunctionCommentSniff.php
#256 -- Removed unused variables

2.3.12

Enhancements

Fetched latest allure-codeception package

2.3.11

Fixes

mftf run:failed now correctly regenerates tests that are in suites that were parallelized (suite => suite_0, suite_1)

2.3.10

Enhancements

Maintainability
- Added new mftf run:failed commands, which reruns all failed tests from last run configuration.

Fixes

Fixed an issue where mftf would fail to parse test materials for extensions installed under vendor.
Fixed a Windows compatibility issue around the use of Magento's ComponentRegistrar to aggregate paths.
Fixed an issue where an element with no type would cause PHP warnings during test runs.

2.3.9

Fixes

Logic for parallel execution were updated to split default tests and suites from running in one group.

2.3.8

Fixes

ModuleResolver will now only scan under MAGENTO_BP/app/code/... and MAGENTO_BP/vendor/... for /Test/Mftf directories.
Fixed an issue where Test.xml files that did not end with *Test.xml would not be scanned for duplicates and other XML validation.

2.3.7

Enhancements

Traceability
- Test generation errors output xml filename where they were encountered, as well as xml parent nodes where applicable.
- Duplicate element detection now outputs parent element where duplicate was found.
Maintainability
- Standalone MFTF can now be pointed at a Magento installation folder to generate and execute tests.
  - See DevDocs for more information.
- MFTF now checks for test and actionGroup elements that have the same name in the same file.
Customizability
- Updated prefered syntax for actionGroup arguments that use xml.data (old syntax is still supported)
  - Old: xml.data
  - New: {{xml.data}}
Modularity
- ModuleResolver now utilizes each Magento module's registration.php to map MFTF test material directories.

Fixes

The waitForPageLoad action now correctly uses the given timeout attribute for all of its checks.
Firefox compatibility issues in javascript error logging were fixed.
Fixed an issue where arguments containing - would not properly resolve parameterized selectors.
Fixed an issue where actions using parameterArray would not resolve $persisted.data$ references.
Fixed an issue where composer installations of Magento would fail to parse MFTF materials under a path vendor/magento/module-<module>/

2.3.6

Enhancements

Maintainability
- A -r or --remove flag has been introduced to bin/mftf commands to clear out the contents of the _generated folder before generation. This flag has been added to the following commands:
  - generate:tests
  - generate:suite
  - run:test
  - run:group
Customizability
- Persisted data handling mechanisms have been reworked.
  - All persisted data is now referenced with the single $ syntax (old syntax is still supported):
    - $persistedData.field$
  - Persisted data resolution now starts in its own scope and broadens if no matching stepKey was found in the current scope.
  - Added support for referencing suite persisted data in tests.
  - Added support for removing data created in between test scopes (test, before/after, suite).
- An attribute skipReadiness has been added to all test actions, allowing the individual test action to completely bypass the ReadinessExtension if it is enabled.

Fixes

To prevent Allure reporting from collating tests with identical title, the testCaseId annotation is now automatically prepended to the title annotation when tests are generated.
The magentoCLI command now correctly removes index.php if it is present in the MAGENTO_BASE_URL.
Invalid XML errors now indicate which XML file caused the error.
Attempting to extend a test that does not exist now skips the generation of the test.
Fixed an issue where a suite would generate invalid PHP if the before or after contained only createData actions.
Fixed an issue where a selector inside an actionGroup would incorrectly append the actionGroup's stepKey to the selector.

2.3.5

Fixes

Removed PageReadinessExtension from default enabled extensions due to Jenkins instability.

2.3.4

Fixes

MagentoWebDriver overrides parent::_after() function and remaps to runAfter(), necessary to solve compatibility issues in Codeception 2.3.x.

2.3.3

Fixes

Defaults in etc/config/functional.suite.dist.yml changed: window-size to 1280x1024, and removed --ingonito flag.

2.3.2

Fixes

The executeJs function no longer escapes persisted variables referenced via $$persisted.key$$.
Extending a test no longer fails to generate the parent test's before/after blocks if the parent was skipped.

2.3.1

Enhancements

Maintainability
- mftf build:project now copies over the command.php file into the parent Magento installation, if detected.

2.3.0

Enhancements

Traceability
- MFTF now outputs generation run-time information, warnings, and errors to an mftf.log file.
- Overall error messages for various generation errors have been improved. Usage of the --debug flag provides file-specific errors for all XML-related errors.
- Allure Reports now require a unique story and title combination, to prevent collisions in Allure Report generation.
- The features annotation now ignores user input and defaults to the module the test lives under (for clear Allure organization).
- The <group value="skip"/> annotation has been replaced with a <skip> annotation, allowing for nested IssueId elements.
- Tests now require the following annotations: stories, title, description, severity.
  - This will be enforced in a future major release.
Modularity
- MFTF has been decoupled from MagentoCE:
  - MFTF can now generate and run tests by itself via bin/mftf commands.
  - It is now a top level MagentoCE dependency, and no longer relies on supporting files in MagentoCE.
  - It can be used as an isolated dependency for Magento projects such as extensions.
- generate:tests now warns the user if any declared <page> has an inconsistent module (Backend vs Magento_Backend)
- The --force flag now completely ignores checking of the Magento Installation, allowing generation of tests without a Magento Instance to be running.
Customizability
- Various test materials can now be extended via an extends="ExistingMaterial" attribute. This allows for creation of simple copies of any entity, actionGroup, or test, with small modifications.
- test and actionGroup deltas can now be provided in bulk via a before/after attribute on the test or actionGroup element. Deltas provided this way do not need individual before/after attributes, and are inserted sequentially.
- Secure and sensitive test data can now be stored and used via a new .credentials file, with declaration and usage syntax similar to .env file references.
- A new <generateDate> action has been added to allow users to create and use dates according to the given date and format.
  - See DevDocs for more information on all above Customizability features.
Maintainability
- New bin/mftf commands have been introduced with parity to existing robo commands.
  - robo commands are still supported, but will be deprecated in a future major release.
- The mftf upgrade:tests command has been introduced, which runs all test upgrade scripts against the provided path.
  - A new upgrade script was created to replace all test material schema paths to instead use a URN path.
- The mftf generate:urn-catalog command has been introduced to create a URN catalog in PHPStorm to support the above upgrade.
- A warning is now shown on generation if a page's url is referenced without specifying the url ({{page}} vs {{page.url}}).
- An error is now thrown if any test materials contain any overriding element (eg different <element>s in a <section> with the same name)
  - This previously would cause the last read element to override the previous, causing a silent but potentially incorrect test addition.
- Test distribution algorithm for --config parallel has been enhanced to take average step length into account.

Fixes

_after hook of tests now executes if a non test-related failure causes the test to error.
Fixed periods in Allure Report showing up as •.
Fixed Windows incompatibility of relative paths in various files.
Suites will no longer generate if they do not contain any tests.
Fixed an issue in generation where users could not use javascript variables in executeJS actions.
Fixed an issue in generation where entity replacement in action-groups replaced all entities with the first reference found.
Fixed an issue in generation where createData actions inside actionGroups could not properly reference the given createDataKey.
Fixed an issue where suites could not generate if they included an actionGroup with two arguments.
Fixed an issue in generation where calling the same entity twice (with different parameters) would replace both calls with the first resolved value.
The magentoCLI action now correctly executes the given command if the MAGENTO_BASE_URL contains index.php after the domain (ex https://magento.instance/index.php)
The stepKey attribute can no longer be an empty.
Variable substitution has been enabled for regex and command attributes in test actions.

GitHub Issues/Pull requests:

#161 -- MAGETWO-46837: Implementing extension to wait for readiness metrics.
#72 -- declare(strict_types=1) causes static code check failure (fixed in #154)

2.2.0

Enhancements

Traceability
- Javascript errors are now logged and reported in test output.
- Test failures are no longer overwritten by failures in an <after> hook.
- Tests will no longer execute an <after> hook twice if a failure triggered in the <after> hook.
- Tests marked with <group value="skip"> will now appear in generated Allure reports.
  - Along with the above, the robo group command no longer omits the skip group (skipped tests are picked up but not fully executed).
Modularity
- MFTF no longer relies on relative pathing to determine its path to tests or Magento (favoring composer information if available).
- Tests and test materials are now read in from Magento modules as well as extensions in addition to dev/tests/acceptance.
  - See DevDocs Getting Started for details on expected paths and merge order.
Customizability
- Creation of Suites is now supported
  - <suite> can include tests via name, module, or <group> tags.
  - Consolidation of preconditions can be achieved via use of <before/after> tags in a <suite>
    - All normal test actions are supported
    - Data returned from actions is not available for reference in subsequent tests (createData or grab actions).
  - robo generate:tests generates all suites and tests, and can be given a JSON configuration to generate specific test/suites.
  - See MFTF Devdocs "Suite" page for more details.
- <deleteData> may now be called against a url instead of a stepKey reference.
- <dragAndDrop> may now be given an additional x/y offset.
- <executeJS> now returns a variable based on what the executed script returns.
- Added <element> type="block".
- <page> elements may now be blank (contain no child sections).
Maintainability
- robo generate:tests --config parallel now accepts a --lines argument, for grouping and sorting based on test length.
- robo generate:tests now checks for:
  - Duplicate step keys within an actionGroup.
  - Ambiguous or invalid stepKey references (in merge files).
- robo generate:tests now suppresses warnings by default. The command now accepts a --verbose flag to show full output including warnings.

Fixes

Exception message for the <conditionalClick> action now correctly references the selector given.
Usage of multiple parameterized elements in a selector now correctly resolves all element references.
Usage of multiple uniqueness references on the same entity now generate correctly.
Persisted entity references are correctly interpolated with <page> url of type="admin".
Metadata that contains 2 or more params in its url now correctly resolve parameters.
Arguments can now be passed to x and y attributes in actionGroup.
Arguments can now be passed to nested <assert*> action elements.
The <seeInField> action can now be used to assert against empty strings.
Empty <data> elements within an <entity> now generate correctly.
Mapping of the <magentoCLI> to the custom command has been fixed.

GitHub Issues/Pull requests:

#89 -- Add ability to use array entities as arguments.
#68 -- Excessive double quotes are being generated in WaitForElementChange method arguments (fixed in #103)
#31 -- Can't run tests without a store having "default" store code (fixed in #86)

2.1.2

Enhancements

Added support for PHP version 7.2

2.1.1

Enhancements

Modularity
- MFTF now supports the existence of tests as composer package dependencies or as living alongside core modules. Supported paths:
  - magento2ce/vendor/[vendor]/[module]/Test/Acceptance
  - magento2ce/app/code/[vendor]/[module]/Test/Acceptance
Maintainability
- Robo command generate:tests now accepts a --nodes argument that specifies the number of test manifest files to generate for parallel configuration.

Fixes

Data returned by grab and createData actions can now be used in <actionGroup> test steps by their stepKey reference.
Fixed an issue where <requiredEntity> elements inside <entity> data xml would overwrite one another when merged.
Fixed an issue where <object> elements inside <operation> metadata xml would overwrite one another when merged.
Nested assertion syntax now correctly allows for values passed in to resolve {{entity.data}} references.
Test action <selectMultiOption> now correctly resolves entity references passed in to filterSelector and optionSelector.
The robo command generate:tests --force no longer requires a MAGENTO_BASE_URL to be defined in the .env file.
All feature and story annotations now live under in the method and not class level in output test php.
- This is to work around a bug with the Allure-Codeception adapter version 1.2.6, which was a dependency update in MFTF 2.1.0.

2.1.0

Enhancements

Traceability
- Severity in <annotation> tags now properly reflect Magento severity values.
Modularity
- Added ability to pass in simple values to actionGroups via addition of type attribute in actionGroup <argument> declaration.
  - For examples, see devdocs article on actionGroups.
- Merging resolution now depends on Magento instance's installed modules. This also means merging order now follows the expected module merging order.
Customizability
- Added <assertArrayIsSorted> action. This action takes in an array of data and asserts that the array is properly sorted, according to the provided sortOrder
- Added <selectMultipleOptions> action. This is a variation of <searchAndSelectOptions> that is given a filterSelector, optionSelector, and an <array> of options to select.
  - For a working sample, see SearchAndMultiselectActionGroup.xml under Catalog in magento2ce.
- Test actions that deal with <url...> now utilize and grab the page's full url, not just the /path?query#fragment portion.
- All <assert...> actions support a clearer, more readable nested syntax.
  - Both old and new syntax are supported. See devdocs Assertions article for examples.
- Added support for overriding a data-entity's field values during test runtime, prior to persistence via <createData> actions.
- Added removeBackend="true" attribute to <operation>. Only applicable to operation definitions of type="adminFormKey", attribute prevents pre-pending of MAGENTO_BACKEND_NAME to the url specified.
  - Specific to use case where adminFormKey operations don't follow MAGENTO_BASE_URL/MAGENTO_BACKEND_NAME/MAGENTO_BACKEND_NAME/API_URL format.
Readability
- Data Entities used in tests (<test>, <page>, <section>, <element>, <data>) now require alphanumeric naming.
Maintainability
- Documentation for all test actions have been added to XML schema. Turning on documentation hinting will display relevant information while writing test XML.
- All references to .env file contents are now resolved at test runtime, as opposed to generation.

Fixes

Fixed an issue with using the character - in parameterized selector references.
- Users should now be able to use any characters except for ' when providing a 'stringLiteral' to a parameterized selector/url.
Fixed an issue where entity substitution was not enabled in all <assert...> test actions.

GitHub Issues/Pull requests:

#37 -- Unable to make API requests using self signed certificate to HTTPS domain (fixed in #39)

2.0.3

Enhancements

Readability
- Added the ability to refer to custom_attribute data in persisted entities via key instead of index.
  - ex. url_key in category entity: $category.custom_attributes[3][value]$ and $category.custom_attributes[url_key]$ are both valid.
Maintainability
- Added check for duplicate stepKey attributes at test generation. This check is scoped to <testAction> tags within a single <test> tag in a single file.

Fixes

Fixed inability to use <actionGroup> with <arguments> in test hooks.
Fixed inability to use 0 as data in an entity.
Fixed an issue where using <annotation> tag of <useCaseId> would cause test generation failures.
Fixed an issue where the <closeAdminNotification> action could not be used twice in in a <test>.
Fixed an issue where specifying duplicate test actions in test delta's would cause generation errors.
Fixed an issue where test failure screenshots were being taken at the end of the test hook, as opposed to at the point of failure.
Operation metadata with an auth of type adminFormKey will now automatically append specified MAGENTO_BACKEND_NAME if necessary.

2.0.2

Enhancements

Customizability
- Added the <magentoCLI> test action. Action takes the given command="" and passes it for execution in Magento Environment.
  - Note: Installation step to enable above action has been added. See Step 5 in the MFTF Getting Started article.
Maintainability
- Tests now run actions declared in <after> hook in both successful and failed test runs.

Fixes

Fixed inability to use [] characters within selector/url parameters.
Fixed a bug where the <formatMoney> action did not return a variable for test use.
Fixed a bug where the <waitForLoadingMaskToDisappear> action could not be used twice in an <actionGroup>.

2.0.1

Fixes

Fixed an issue with group annotation.

2.0.0

Enhancements

Modularity
- Replaced the <loginAsAdmin> test action with the action group LoginAsAdmin.
- Added the .env file variable CUSTOM_MODULE_PATHS which can be used to point to any custom extensions that you may want to write tests against.
- Added the <page area=".."> property to distinguish between admin and storefront.
- Added support for SectionName.elementName references in any function attributes.
Customizability
- Changed page objects where area="admin" to prepend the MAGENTO_BACKEND_NAME value from the .env file.
- Added support for HTTP requests that do not require authentication.
Readability
- Renamed <config> XML root nodes to match the content they contain, e.g. <tests> or <pages>.
- Renamed all instances of the word Cest with Test. The Cest name will no longer be used in the MFTF project.
Maintainability
- Removed the returnVariable property from any test actions that return values. Instead, the stepKey property will be used as the name of the variable and be referenced as before.

Fixes

Fixed the unselectOption.parameterArray property to work as expected.
Fixed a crash if you had a system environment variable set with the same name as any variable in the .env file.
Fixed any actions that refer to CurrentUrl, such as <seeInCurrentUrl>, to now look at the full webdriver address.
Fixed the <waitForPageLoad> test action to not assume that you always want to dismiss UI notifications.

1.0.0

Core features

Traceability for clear logging and reporting capabilities
Modularity to run tests based on modules/extensions installed
Customizability to have an ability to customize existed tests
Readability using clear declarative XML test steps
Maintainability based on simple test creation and overall structure

Supported systems

Operation systems

Windows 10
macOS Sierra

Browser

Chrome (Latest) with ChromeDriver Latest

Known issues

Support for Firefox is currently incomplete. This will be resolved to support Firefox 57 (Quantum) and latest Gecko driver in next minor release.
MAGENTO_BASE_URL in .env file must have / at the end. Example: http://magento.com/

Files

CHANGELOG.md

Latest commit

History

CHANGELOG.md

File metadata and controls

Magento Functional Testing Framework Changelog

3.7.0

GitHub Pull Requests:

Enhancements

3.6.0

Enhancements

GitHub Pull Requests:

3.5.1

GitHub Pull Requests:

3.5.0

Enhancements

Fixes

GitHub Pull Requests:

3.4.0

Enhancements

3.3.0

Enhancements

Fixes

3.2.1

Fixes

GitHub Pull Requests:

3.2.0

Enhancements

Fixes

3.1.1

Fixes

3.1.0

Enhancements

Fixes

GitHub Issues/Pull Requests:

3.0.0

Enhancements

Upgrade Instructions

Fixes

GitHub Issues/Pull Requests

Demo Video links

2.6.4

Fixes

2.6.3

New Feature

2.6.2

Fixes

2.6.1

2.6.0

Fixes

GitHub Issues/Pull requests:

2.5.4

GitHub Issues/Pull requests:

Fixes

2.5.3

Fixes

2.5.2

Fixes

GitHub Issues/Pull requests:

2.5.1

Fixes

GitHub Issues

2.5.0

Fixes

GitHub Issues/Pull requests:

2.4.5

Fixes

2.4.4

Fixes

2.4.3

Fixes

GitHub Issues/Pull requests:

2.4.2

Fixes

2.4.1

Fixes

2.4.0

Enhancements

Fixes