Author: Ceyda ÖZAYDIN KASA / SW Test Architect – Test Services

As the software testers, many of us have some familiarity with test automation tools. Some of us may have used them for a long period, some of us may have played with them a little only, or have heard about their names, without even knowing their functionality.

The era of test automation tools started at the beginning of 2000’s, and during these many years, they have not stopped with their progress and development. Because of the high demand for test tools in very different areas of software testing, they have evolved in so many directions, which at the final leads to the emergence of a huge market, that introduces a wide range of test automation tools. Any of these tools, whether paid or free, can be put in use upon project needs and tester preference.

But why are they so popular? That’s because, they offer some benefits, which can be listed as below:

• They enable decrease in repeating manual tasks, resulting in time and effort saving.
• Tests are executed in a more consistent and repeatable manner (More consistently generation of test data and test scenarios, based on given requirements. More consistently execution of test cases by the same test tool, in a predefined sequence and frequency).
• Test results are evaluated more objectively (Tools provide metrics, test coverage is under control).
• It is easier to reach the test related information (Test execution progress, defect rates, statistical data and graphics about performance).

However, some risks may also arise. They are:

• Expectations on the benefits of test tools are unrealistic (Their functionality and usefulness).
• There can be much more dependency on the test tool than it ought to be (Test team may imagine that all manual tests are suitable to be automated, or the tool may be thought as the substitute for test design, test implementation etc.).
• Integration problems may arise when already existing tools (requirement management tool, configuration management tool, defect management tool etc.) are attempted to be integrated with the new test automation tool.
• The organization can face problems based on the difference between expected and actual cost and effort.
• The version control mechanism of the testware (test documentation, test scenarios, test data, software under test) may be neglected.
• The newly employed test tool may not support a new platform or technology, which is expected to be.
• There can be supplier related problems, especially for open source, free tools.

Remark that the answer of the question: “How to select which tool” is very critical, because it has direct impact on the success of a test automation project. For this reason, in the “Test Automation Strategy” diagram given below, evaluation and selection of the most appropriate test tool is given as the following step immediately after the feasibility analysis phase of the project.

While keeping this strategy in mind, criteria for the selection of test tools can be summarized as below:

• Tool has to be studied against project requirements and restrictions.
• Tools integration capability with the other, already existing tools has to be assessed.
• Tools supplier has to be evaluated regarding i.e. training offers, supporting abilities and commercial aspects. Also, open source, noncommercial tools have to examined meticulously.
• In-house requirements and sources for coaching / mentoring and assistance and training needs related with the new tool have to be determined.
• A benefit / cost evaluation has to conducted based on a tangible business scenario, i.e., a PoC (Proof of concept) study or a pilot project.

Now, let’s look at the test (automation) tools which have gained the most popularity in recent years, mainly due to their price-effort efficiency and usability. Because they vary in a wide range, I categorized them under certain titles, based on their scope and functionality.

UI (Web) Test Automation Tools: Testing the UI (Graphical User Interface) of an application to verify its functionality (i.e., UI workflows, navigation, state transitions, calculations, validating logical functionality of each button, validating input fields, etc.) is known as UI testing. Automated UI testing mentions the usage of specific tools for automating manual UI test tasks. The list of tools for automated “Website” UI testing includes following popular brands…

API Test Automation Tools: API testing enables software teams to validate their API’s quality by confirming that it is working as expected. API test automation refers to using a testing tool to programmatically execute API tests in a scheduled manner, or in CI/CD pipelines. Let’s see some of the tools, which are highly demanded in the market…

Mobil Test Automation Tools: Mobile app testing is the checking that the mobile application fulfils the specifications, is compatible with the devices it is supposed to be used on and is good at some characteristics like functionality, performance, security, usability, deployability etc. When it comes to automation, the developers or testers automate the testing process on the mobile devices. Here are some mostly used automation tools…

Acceptance Test Automation Tools: Acceptance Test has many types, but its main goal is to determine that the software does what it was designed to do in real-world. This is usually an end-to-end test to check for fulfilling all requirements and is performed based on customer view. Automated UAT utilizes scripted tests that run automatically by means of an automated testing tool. Some most common examples seem to be following…

Performance Test Tools: They are used to measure the load, stability, response time of the application, which help the test team to validate the performance of a software.  They usually support automation by means of their computerized script generation and execution facilities. Top performance test tools are given below…

Last but not least, it should be mentioned now about test automation pyramid, including test automation tools. The pyramid proposes that the lower-level tests (unit, component) should be the majority of an application’s overall automated test count. As tests become more high-level, thus covering end-to-end functionality, and possibly rely on systems outside of the application, then it is a good practice to have fewer of automated tests. This principle can be displayed with the help of the diagram given below:

According to this approach, some of the tools described in earlier sections can be deployed in the test automation pyramid as below:

As conclusion, deciding for a test automation tool and employing it in the organization’s landscape is a critical problem to solve. There are many aspects which have to be taken in consideration during the evaluation and selection phases of the tool. Fortunately, there are many information sources and well documented best practices to support the software team while incorporating any test automation tool.