Magento Functional Testing Framework (MFTF)
Summary of the Article
Magento Functional Testing Framework (MFTF) was introduced by Tom Erskine at Mage Test Fest. MFTF has already seen 2 major releases and the latest version available till today is 2.3.12.
Before, getting to know about the MFTF (Magento functional Testing framework), we should be aware of the Functional Testing framework.
What is Functional Testing Framework(FTF)?
FTF is an open source cross-platform solution where we can develop functional tests for a Magento application. These tests can be performed at any time with the options of –
- Running a single test independently
- Running multiple tests together as a test suite
- Running all available tests in one-go
Tests usually cover the functionality of a business entity. The goal is to find discrepancies between expected and real behavior of the entity.
What is the need for functional testing?
Magento 2 offers a very complex architecture and it becomes very difficult to manually test everything. Also, manual testing can result in human mistakes due to complex flows.
Although unit and integration testing can cover the whole code but still don’t guarantee that everything will execute in the same way on different devices and browsers.
What is Magento Functional Testing Framework (MFTF)?
Key Features of MFTF –
- Flexible, because of modularity support.
- Customizability, for existing tests.
- Informative, because of the reporting tool.
- Suitable, because of test suites.
- Compatible, because of web driver selection
Benefits of MFTF over Unit Tests
MFTF tests are written in XML, so, you no longer need to learn PHP to write tests.
Merging is the most beneficial feature of MFTF. For example, we add a new field to the existing registration form of Magento website.
In the case of unit testing, we need to write test cases again for the complete page but in MFTF, we just need to provide extra attribute details and rest we can use the same test case code from the core.
Environment setup for MFTF
- PHP version
Upgrade your PHP version according to Magento instance which you are using. In our case, we will be using Magento 2.3, and our PHP version is 7.2.13
Install the composer on your environment. Please follow the below link to install composer.
You need to install Java on your system to run the Selenium server.
- Selenium server
Now install selenium jar file on your instance, which will help you to run MFTF test cases.
Download the jar file and place it in the Magento root directory or some server (depends upon your requirement). Please follow the below link for download –
- Web Browser Driver
You need browser driver (Mozilla, Chrome, Safari) to run your test cases. In our case, we are using the Google Chrome Driver. Follow the below link for download –
Step by step guide to setup Magento Functional Testing Framework
Step 1 – Fresh Install Magento 2.3
I hope you are already aware of Magento 2 installation. But still, if you are new please refer to the following link for Magento 2 installation –
You can also download the Magento 2.3 from here and just extract it to your localhost directory. After that just hit the Magento directory on your browser and complete the Magento setup.
Step 2 – Install dependencies
We need to install the MFTF. Please follow below command –
composer install -d dev/tests/acceptance/
Step 3 – Build the project
In the Magento project root, run –
Step 4 – Edit environmental settings
In the [Magento_Root]/dev/tests/acceptance/ directory, edit the .env file to match your system.
Specify the following parameters, which are required for tests –
- MAGENTO_BASE_URL – Must contain a domain name of the Magento instance that will be tested. Ex: http://127.0.0.1/magento2mftf/
- MAGENTO_BACKEND_NAME: must contain the relative path for the Admin area. Ex: admin
- MAGENTO_ADMIN_USERNAME: must contain the username required for authorization in the Admin area. Ex: admin
- MAGENTO_ADMIN_PASSWORD: must contain the user password required for authorization in the Admin area. Ex: admin12345
Step 5 – Enable the Magento CLI commands
In the [Magento_Root]/dev/tests/acceptance directory, run the following command to enable the MFTF to send Magento CLI commands to your Magento instance.
cp dev/tests/acceptance/.htaccess.sample dev/tests/acceptance/.htaccess
Step 6 – Generate and run tests
To run tests, you need a running Selenium server and mftf commands.
Run the Selenium server in another terminal
Run the Selenium server in the terminal. Both selenium jar file and browser driver should be on the same place. In our case, both reside in the Magento root directory.
For example, the following commands run the Selenium server for Google Chrome –
cd <path_to_directory_with_selenium_server_and_webdriver>/Java -Dwebdriver.chrome.driver=chromedriver -jar selenium-server-standalone-3.14.0.jar
Generate the tests
If you want to run complete test cases you need to run below command
vendor/bin/codecept run functional
Using vendor/bin/mftf you can run specified group (vendor/bin/mftf run:group product),
To run particular tests
vendor/bin/mftf run:test LoginAsAdminTest
To run previously failed test
Run all generated Magento tests
In the dev/test/acceptance run below command to run all test cases
vendor/bin/mftf run functional
Step 7 – Generate Reports
During testing, the MFTF generates test reports in CLI. You can generate visual representations of the report data using Allure Framework.
We will install allure using npm. First, install npm if it is not available on your system using the following command –
sudo apt install npm
npm install -g allure-commandline –save-dev
Run Allure to check Report –
We need below command to check test result report in allure –
allure serve dev/tests/_output/allure-results/
Example of MFTF test case for the sample module
We have created one module with a form on the frontend and we named it Seller Create Form. We will write MFTF test cases for this form –
MFTF Test Case Structure
<magento_root>/app/code/<vendor_name>/<module_name>/Test/Mftf/ is the directory to create new tests.
Custom Module MFTF have 6 folder structure –
We will put our test cases in this folder. The test cases are nothing but simple XML files each file can contain multiple tests or a single test.
Convention says that you should put one test in each file and file name and test name must be the same and the name should follow camel casing.
MFTF <tests> is considered a sequence of actions with associated parameters.
Inside page folder, we can define pages that will be visited during test case execution. The naming convention of file name and page name must be the same, and the name will be following camel casing and all the names must end with “Page” suffix.
A <section> is a reusable part of a <page> and is the standard file for defining UI elements on a page used in a test.
An action group is the group of actions (like click on the button, page load etc.). It is very useful to create a group of actions for better reusability, like LoginToStorefrontActionGroup,
SignUpNewUserFromStorefrontActionGroup in the customer test module. In many test cases, it must require the customer to login or register.
Any test case will need some dummy data for completing the test case, like dummy data for a product to test product create flow, dummy data for the customer to test registration and login feature. All the data XML files must end with data suffix.
While executing test cases sometimes tester might need some data entities to be created at runtime and used in the test or delete all the created data after the test is complete. This can be done using metadata. You can create operations like create, delete, update, get.
After creating, our own MFTF test cases for a custom module, we need to run generate command so that Magento will consider our test cases as well with their own default cases.
In the Magento root directory, run the below command –
After this, we will run our test cases with the help of our group id (mentioned in Action Group File)
vendor/bin/mftf run:group sellercreate
Now, if we want to check allure report for the same, run below command –
allure serve dev/tests/_output/allure-results/
Magento MFTF enables us to test user interactions with web applications. It helps developers/testers in automating regression testing.
Hope the above example, could have added some idea about the new functional testing framework and how it manages everything with XML files and the relation between them.
If you have any doubts, please feel free to comment.