Manual Testing vs. Automated Testing
A key step in the software development process is testing. It is important to ensure that the software product behaves as designed before releasing it to the end users. While no software is defect free, reducing the number of bugs in a piece of software is critical to a company’s profit and reputation. In the battle to decrease the number of bugs found in their software and systems, many company struggle with the decision of whether to utilize manual software testing or automated software testing. When contemplating manual testing vs. automated testing, one form of testing may be more appropriate than another and a company and its development team must understand what each form of testing entails and the strengths and weaknesses of each.
Manual testing is the process of manually checking software for defects. In this method, a tester performs checks as if they are the end-user. All features of the software under test are exercised to ensure that it behaves according to design and customer expectation. A test plan, which may involve the use of test cases, is used in the implementation of this type of testing. Manual testing is often considered to be a labor-intensive activity that could simply be automated.
Manual Testing of Small and Large Projects
The extent of testing required for manual testing may vary based on the size of the project. For small-scale projects, exploratory testing may be adequate. This is an informal approach where the tester does not follow a rigorous testing procedure, but explores the application’s user interface. The point of the exercise is to use as many features as possible and to use information gained in prior tests to instinctively derive additional tests. The success of this type of manual testing relies greatly on the knowledge of the tester because a lack of expertise will lead to the test being incomplete. In addition to being simple, another advantage of this informal approach is the acquisition of intuitive insight to how it feels to use the product.
Large-scale projects that use manual software testing have to follow a more rigorous methodology in order to find more defects. It has to be a more systematic approach that typically focuses on predetermined test cases. Though each company creates its own process, the process often involves the following steps:
1. Select a high-level test plan. In this test plan, methodology is selected, and resources (i.e. people, equipment, space) are identified and acquired.
2. Write test cases. These test cases are very detailed and identify clear and concise steps to be taken by the tester including expected outcomes.
3. Assign test cases. The test cases are assigned to testers, who are expected to follow the given steps and record results.
4. Report the finding. A test report is created that provides details of the findings of the testers. It is used to determine whether the software can be released or if corrective action is required.
Phases of Manual Testing
Most teams that utilize manual testing go through four phases. These phases include the following:
Unit Testing – In the first phase of testing, the developer writing the code usually tests the units of the code that they are writing. In some cases, a peer may also test the code.
Integration Testing - The second phase is where several pieces of code are integrated to form a larger block of code. In this phase, black box testing technique is used most of the time. Sometimes white box testing is used as well.
System Testing - In the third phase, the software is tested as a system. It is examined from all possible dimensions as well as for all intended platforms and purposes. This is done in order to observe any abnormalities in the system. Black box testing technique is typically used in this phase.
User Acceptance Testing - The final phase is performed in order to get customer approval of the finished product. There are two types of acceptance testing: alpha testing and beta testing.
Automated testing involves a process of using a computer program to perform testing. It is common for an automated test to involve a manual process that is already in place and can be more effective in identifying certain types of defects. In this type of testing, software is used to control the execution of tests, compare and predict outcomes, setup testing preconditions, and any other necessary control and reporting functions. Automated testing can usually be run quickly and is often a very cost effective method of testing software products. This is especially true for products that have a long maintenance life, where software patches alter the function of previous features.
Automated testing tools can usually be used to automatically perform tasks such as product installation, generation of test data, GUI interaction, problem detection, defect logging, and more. It is unique because these things can be done without having to be performed end-to-end.
There are several requirements that must continuously be satisfied when automating tests. They are the following:
• Operating system and platform independence
• Data driven capability (i.e. Input, Output, and Meta Data)
• Report customization (i.e. database access and crystal reports)
• Simplified debugging and logging
• Ease of version control (minimize use of binary files)
• Customizability & Extensibility (i.e. ability to integrate with other tools)
• Common Driver (For example, in the Java development ecosystem, that
• Ability to support unattended test runs for integration (using build processes and batch runs)
• Automated notification upon failure or reaching of threshold levels
• Support if distributed applications
There are basically two approaches to automated testing: code-driven testing and graphical user interface (GUI) testing.
In code-driven testing, the interfaces to classes, modules, and/or libraries are tested using variety of input arguments in order to validate that the outputs that are returned are correct. In some cases, unit tests are written to define the functionality before the actual code is written. The code considered complete when all tests pass. It is believed that this method of testing produces more reliable and less costly software because code coverage is better, and it is run constantly throughout the development phase rather than a few times at the end. This helps the developer to instantly discover defects caused by changes. The earlier a defect is detected, the less expensive it is to fix. Another benefit is that code complexity decreases because code duplication is minimized.
The GUI testing approach can be applied to any application that has a graphical user interface. The advantage of this approach is that it does not require a lot of software development. In order to confirm that the visible behavior of a program is correct, GUI testing is a framework that generates user interface events such as keystrokes and mouse clicks, so that the changes that result in the user interface can be observed.
Several of the tools used for automated testing have record and playback features that allow users to record user actions and replay them repeatedly for the purposes of comparing expected results with actual outcomes. However, these features can introduce some reliability and maintainability issues because modifications to the GUI such as button name changes and relocations require that the test be recorded again.
There are several variations of GUI testing. One is used to test web sites. In these cases, the interface is the web page and it “observes” HTML rather than window events. Another variation is called script-less test automation. It does not use record/playback, but builds a model of the application being tested and enables the tester to create test cases by editing in test parameters and conditions. Despite the name, this does not require scripting skills, but has the flexibility and power of scripting.
Comparison: Manual Testing vs. Automated Testing
A common message when weighing manual testing vs. automated testing, automated software testing is often thought to be the best way to increase the effectiveness, efficiency and coverage of your software testing because they usually save time and money, improve accuracy, and assist developers and tester in doing things that manual tests simply cannot. A computer can follow a sequence of steps faster than a person, and can run continuously overnight if necessary.
However, there are some that believe that while manual tests are indeed labor- intensive, it is incorrect to underplay the importance of human intelligence and assume that all manual testing can be replaced by automated tools. Also, one must consider that sometimes the time spent creating the automated test can sometimes match the time saved by the testing. As a matter of fact, depending on the type of system being tested and the tools that are chosen for test automation, some automated testing may require more labor than a manual testing approach. In addition to this, sometimes interpreting the results of the large amount of data produced by automated testing tools can present a challenge of its own. This means that automated testing becomes more cost effective in cases of regression testing and test-driven development, when the same tests can be reused many times over the results can be interpreted quickly. So if test software is not likely to be used in the future, then a manual approach is more favorable.
Items such as device drivers and software libraries should be tested using the automated testing approach. In addition to this, for performance testing and load testing (testing large numbers of users) it is better to simulate with automated testing rather than manual testing.
On the other hand, GUIs where the layout changes frequently are extremely difficult to check using automated testing. There are test frameworks that may be useful for regression testing of GUIs but they rely on recording sequences of keystrokes and mouse movements, followed by playback and observation of user interface response. If these buttons are moved or changed in any other way in subsequent releases, these recordings may not work properly. In these cases, manual testing may be more effective then automated testing. When making decisions regarding manual testing vs. automated testing, one should take into account the items being tested. In some instances, one form of testing may be more useful than the other, but neither is ideal for all cases.