Smoke Testing

In the previous tutorials, we have studied Manual Testing, Automation Testing, Regression Testing, and other types of testing. Here, we will look into Smoke Testing, the need for Smoke Testing, characteristics of Smoke Testing, methods of performing Smoke Testing, why it is performed, advantages and disadvantages of Smoke Testing, and more.

Before moving forward to smoke testing, we'll first understand the two most important terms related to smoke testing, i.e., Software Build and Release.

What is a Software Build?

In the software development life cycle, a software build is a stand-alone form of software which consists of a few modules and features of the software. While developing software, developers divide the software into more petite builds or versions based on the priority of modules as per the requirements, which makes the development process easy and quick. For example, the first build of the software is released with only two basic features, i.e., user registration and login. The next build will be the improvised version of the existing build in which more parts and functionalities will be added. Similarly, each build will be an enhanced version of the current build with more functionalities.

A software build goes through different environments in its lifespan, which are mentioned below:

• Development (DEV) – It is the environment used for writing the code and developing the build. It is only accessible to developers and not to other persons, and this environment is owned by developers.
• Testing (QA) – The environment is used for testing the software build. Once a build is developed, it is moved to the testing environment for testing activities. This environment is owned by the testers, and developers don't have access to it.
• User Acceptance Testing (UAT) – This environment is used by business users for user acceptance testing. In this environment, a software build is tested from the business perspective, and it is only accessible to business users and sometimes accessible to testers on special permissions and requirements.
• Production (PROD) – This is the production environment used by the end-users. Once the software build is tested, it is moved to the production environment, which is a real-time environment.
• Staging – This environment exactly resembles the production environment. Before moving the software build to the production environment, the build comes into the staging environment for last-minute checks. The software installed into the staging environment should exactly match the production environment.

What is Release?

Like build, software release is also an important term in SDLC (software development life cycle). When the final or last version/ build of the software gets deployed into production for the end-users is known as a software release. In other words, when the finalized product is handed over to the client or customers is termed a release. Before releasing a product, several activities are performed on the software, such as requirement gathering, designing, development, and testing, until the product gets deployed into production.

The release can be either public or private based on the organizations and requirements of the software upgrades. In software development, there are mainly three types of releases available, which are mentioned below:

1. Major Software Release – This type of software release includes a large number of changes and modifications done to the software, along with adding new features.
2. Minor Software Release – This software release includes smaller updates and changes to the software builds.
3. Emergency Release – This type of software release includes urgent or end-moment changes and defect fixes that reduce the system's performance. Most of the end-users challenges and difficulties are resolved in this release.

What is Smoke Testing?

Smoke Testing is performed to validate whether the recent software build is testable or not. In other words, it is the initial or first test performed after every build is deployed and before it is released into production. When the software or application is deployed in QA (Quality Assurance) or testing environment to validate the quality and stability of the software, smoke tests are the first tests performed on the deployed build. A smoke test is referred to as the minimal set or the least number of tests executed on a single software build.

It mainly focuses on the core and essential modules and features of the application. The primary purpose of smoke testing is to ensure that the software build is stable for the other testing process, i.e., flow blown or rigorous testing. It acts as a confirmation for the testing team or QA to proceed with the subsequent testing activities. It is also known as Build Verification Testing, Confidence Testing, or Build Acceptance Testing.

Smoke testing is a time-saving process which also acts as a small and rapid regression testing of the critical and significant functionalities of the product. It is done to identify issues and faults in the system early so defects can be fixed as early as possible. When a defect is encountered by the testing team, it will be reported to the development team for a fix. In the meantime, testers proceed with the testing of other software builds or independent testable modules.

For smoke testing, we don't need to prepare test cases separately. Test cases can be selected from already designed cases for rigorous testing. Usually, in smoke testing, we concentrate on the test suit that covers the core and significant functionalities of the build. Therefore, the number of test cases is minimal in smoke testing, and test execution is also a time-saving process.

Need for Smoke Testing

Like other testing types, Smoke testing is also a crucial element of the software testing life cycle and the software development life cycle as it validates the stability, quality, and reliability of the system or product. It ensures that each software build is correct and works per the software's expectations. With the help of smoke testing, we can enhance the quality of the software by detecting defects in the early stage, and it saves a lot of testing efforts for full-blown testing.

Below are some reasons behind the need for smoke testing:

• To ensure that the system is stable and testable.
• To identify all possible defects in each software build as early as possible so that defects can be fixed in the early stages of the testing process.
• To validate the functionality, reliability, and correctness of the system for further rigorous testing.
• With the help of smoke testing, we can easily detect all the show stoppers in each deployed build.
• It makes the identification and correction of defects faster and easier.

Characteristics of Smoke Testing

As discussed above, smoke testing is the initial test performed on any software, application, and system after a build is deployed in the testing environment. It helps in analyzing the system’s correctness and stability for flow-blown testing. There are various characteristics and benefits of smoke testing, and some of them are listed below:

• One of the main characteristics of smoke testing is that it can be scripted and documented so that we can use the documents as a reference in future.
• It helps in detecting the issues and errors in every new build so that they can be fixed as early as possible.
• Smoke testing can be referred to as the type of regression testing, as it also tests the system after every new build is deployed.
• Smoke testing can be performed or executed quickly. (The term “Quickly” depends on a specific situation and may vary depending on different situations.)
• It provides broad coverage throughout the system and tests the entire deployed software build.
• It is mainly carried out by testers and QA engineers.
• Smoke testing can be stable or unstable as it validates the stability of the software build. Therefore, the result can be durable and vice versa.
• Smoke testing should also have a self-scoring ability as automation testing.

Methods of performing Smoke Testing

Smoke testing can be performed in different ways. There are mainly three methods for performing Smoke testing, as mentioned below:

1. Manual Testing: In this testing type, the testing team is responsible for preparing, writing, modifying, and updating the test cases for each software build. Testers usually analyze and select test cases from the existing test suite developed for rigorous testing. Smoke testing is generally done manually in several organizations. Manual testing techniques may differ from organization to organization, but the intention behind the test is to test the accuracy and stability of each system’s build.
2. Automated Testing: In this testing type, testing is done with the help of an automation tool responsible for handling the overall testing activities. It is beneficial to use in the case of a project deadline given by the client. Whenever a new build is delivered to QA, the team can perform test execution with the help of automation and record the smoke test cases for future builds. Recording all the manual steps and activities is the main benefit of automating the smoke testing process, and it also helps in reducing the time and effort done manually.  
3. Hybrid Testing: As the name suggests, Hybrid testing is a blend of manual and automation testing. In hybrid testing, the testing team can either prepare and document test cases manually or automate them with the help of an automation tool. It helps in enhancing the performance of the testing process as it is a combination of both manual and automation testing.

Who will do Smoke Testing?

As discussed earlier, when the software build is deployed in the testing environment, the initial test performed on the build is Smoke testing which is mainly done by the QA leads or Test Engineers. The testing team is the one who conducts smoke testing on every build of the developed product. The person responsible for smoke testing has the benefit of knowing the system thoroughly and should be able to test each module/ feature of the system. Once the build is deployed in QA, they will deeply examine the software and its requirements to proceed with smoke testing. The team will begin with the execution to expose and discover all possible bugs present in the system, which helps in delivering a high-quality standard product.

Although, in some organizations, smoke testing can be performed by both developers and testers. However, it is usually done by a tester.

Testers should be able to accept or reject the deployed software build after executing smoke test cases. If a build is denied by the tester, they should be able to provide valid reasons and correct information to the development team.

After the completion of smoke testing, testers will start performing Sanity testing. It is a subset of regression testing with narrow and deep testing, and it is mainly a double check for the system's accuracy and stability.

When to conduct Smoke Testing?

Smoke testing is mainly performed whenever the latest software build is delivered to the testing environment or staging setup. The smoke testing process is more focused on ensuring the software build received from the development team should be testable and stable. However, it is not exhaustive testing like regression and unit testing, and it should be done at the beginning of the software testing stage and performed at the build level. Smoke testing is usually used with different testing levels such as Integration testing, System testing, and Acceptance testing levels.

Smoke testing starts with the deployment of the software build in the QA environment. Testers will start selecting test cases from the already designed cases for full-blown testing. Once the selection is completed, testers will start executing the test cases. If the testers encounter any bugs or issues during the test execution, the smoke test is said to be failed. Otherwise, if the tests are successfully passed, testers will move forward to functional testing. In case of failure, the issues are reported back to the developers to fix the problems and make the system stable. When a new build is developed, or the existing one is modified in any environment (testing environment and production environment), smoke testing should be done to test the build's stability.

Example: Suppose banking software is being developed by an organization. In the latest build, they have added a new feature to the existing software, i.e., the option to send money. Once the build gets deployed in the QA environment, the first test going to be performed on it is a smoke test to check whether the build is stable for further rigorous testing or not. It validates the system against the requirements specified by the client and also detects early issues and bugs present in the system.

Why is Smoke Testing done?

In software testing, Smoke testing plays a crucial role as it is the first test performed on any software’s latest build to verify the stability and accuracy of the software. It helps us in distinguishing good-quality, and poor-quality software builds and also disqualifies all poor-quality builds which are unable to satisfy the requirements defined by the clients. It acts as a pre-check for the entire system before performing full-blown or rigorous testing. It also helps in identifying bugs, broken functionalities and interactions between the existing and new features or modules in the early stages of the software testing cycle.

Smoke testing helps in reducing the time, effort, and money used in the testing process. As an outcome, smoke testing ensures that the system is being built and working as per the requirements. If the system deviates from its objective, smoke testing makes the system meet the criteria. Testers can move forward to other testing types only after the successful completion of smoke testing. Below mentioned are the reasons for the importance of smoke testing:

• It helps in validating the accuracy, correctness, and stability of every latest build.
• It helps in rectifying all the show-stoppers of the build.
• It helps in the early identification and correction of all severe flaws.

If we don’t perform smoke testing on every new software build, it can lead to severe product failure while doing precise testing in later stages. It will make the testing process difficult and slow as more defects will be encountered, and developers will take more time to fix them. Identifying bugs and flaws in later stages can also affect product timelines and deliverables.

Smoke Testing Process/ Cycle

As mentioned above, smoke testing is performed whenever a new build is deployed to verify the build's stability. The main motive of doing smoke testing is to ensure the software build is appropriately developed, and it acts as an acceptance criterion for every new software build and version. Smoke testing is crucial as it verifies the system builds and prevents an unstable or broken build from being moved into the production unit.

The smoke testing cycle starts when a software build is installed in the QA environment and ends with the successful test completion of the last software build or version. It keeps on repeating until we receive all the stable builds of the software being developed. If any of the smoke tests fail, the build is rejected and moved back to the development team for corrections. Once the build is modified and fixed by developers, it will be tested again to ensure its accuracy and stability.

We need to follow a step-by-step procedure to perform smoke testing efficiently. It includes multiple steps, which are mentioned below:

1. Select the method for performing smoke tests – The first step is to select the method testers will use for smoke testing. We have already discussed the various techniques of performing smoke testing, i.e., Manually, Automated, and Hybrid methods. The method of testing will be selected by the testing team based on the business requirements of the software.
2. Identify and prepare a list of test cases for testing – After deciding on the testing method, the team will analyze the software build and select the test scenarios and cases based on the requirements. Test cases will be chosen from the already designed test cases for full-blown testing. However, the smoke test suite should contain some basic and standard test cases related to the feature, purpose and functioning of the software, and specifically, it should include build verification, business-critical features, integration, and navigation test cases.
3. Execute test cases and record the result – Once the test scenarios and cases are decided, the actual test execution will start. In this step, the tester will execute each test case to verify the system’s stability. If the test cases are passed successfully, the tester will move to the next test case. Otherwise, the failed test case will be reported to the developer with a valid reason for failing. Once all the test cases are passed, the test execution is recorded and documented for future reference. Also, it acts as evidence that the build is ready for comprehensive testing. Recording and writing smoke tests also makes the next steps of the testing process more accessible.
4. Analyze smoke testing results – After completing smoke tests, testers should study and analyze the results to find out if the system needs further modifications or is stable for functional testing. The analysis is also done to verify whether the overall smoke testing is successful or failed.

Below is the flowchart of the smoke testing cycle:

Smoke Testing Tools

Like other testing types, smoke testing can also be performed through several tools. Two of the best and most popular smoke testing tools are mentioned below:

• Selenium - Selenium is an open-source automation testing tool mainly used to test web applications. It is one of the most popular and commonly used testing tools, which is widely used for Functional and Regression testing. However, it also supports non-functional testing and is the best tool for performing Smoke testing. The main feature of Selenium is that it is compatible with multiple programming languages such as Java, Python, C#, Ruby, PHP, etc., and automation testing frameworks. It can also be used with different operating systems such as Windows, Linux, and Mac and browsers like Google Chrome, Internet Explorer, Firefox, Opera, etc.
• PhantomJS – It was one of the best-automated Smoke testing tools used for testing web-based applications. It was a headless browser used for automating web pages. It is used to support multiple web standards and works fluently with both development and testing workflow. The services of PhantomJS were discontinued in March 2018 due to a lack of contributors.

Difference between Smoke Testing and Regression Testing

Below mentioned is the comparison between Smoke testing and Regression Testing:

Smoke Testing	Regression Testing
Smoke testing is usually performed on every latest or new software build.	Regression testing is usually performed in the later stages of software testing.
It is also called surface-level or Build Verification testing, performed on the initial software builds.	Whereas regression testing is referred to as deep-level testing that is performed on stable software builds.
Smoke testing can be referred to as a part of basic software testing and a subset of Acceptance Testing.	On the other hand, regression testing acts as the superset of smoke and sanity testing.
It is done to verify the stability, accuracy, and correctness of the product’s build.	It is done to ensure that the system is working as expected and not impacted by any code and functional modification.
It can be performed by both developers and testers, but mostly it is done by testers.	It is usually performed by the testers.
Smoke tests can be documented as documenting smoke tests is one of the essential steps of the smoke testing process.	On the other hand, we don’t document regression test cases.
Smoke testing is a cost-effective testing process.	The cost of regression testing is higher compared to smoke testing.
It is a time-saving testing process and requires fewer efforts and human resources in testing.	As compared to smoke testing, regression testing is a time-consuming process and requires more effort and human resources to test the system.

Advantages of Smoke Testing

Below are the benefits of Smoke testing:

• It is straightforward to perform smoke testing.
• Smoke testing helps in the early identification and correction of the issues and defects of the software build.
• It helps in improving the quality of the system.
• It also helps in reducing the risk associated with the system.
• The progress of smoke testing is easier to track and access.
• It saves testing efforts and time.
• It makes it easier to detect critical and severe flaws in every software build.

Disadvantages of Smoke Testing

Below are the disadvantages of Smoke testing:

• Smoke Testing does not cover all the functionality in the application, and only a specific part of the testing is done.
• We can still encounter errors and bugs even after implementing all smoke tests.
• Executing smoke tests manually is a time-consuming process and should not be done manually in the case of extensive projects.
• It can’t be implemented against negative tests or with invalid input.
• One of the significant drawbacks of smoke testing is that the smoke test suite contains a limited number of test cases. Therefore, we can't identify other possible issues.