Friday, August 28, 2015

A Proposal to write the Manual for How to Perform Software Testing








A Proposal to write the Manual for How to Perform Software Testing






Colorado State University
Department of Electrical Engineering
JTC300 Professional and Technical Communication



Minh Anh Nguyen
Phone: 970-223-7603









Today, most high tech activities involve software or programs. There are many software products available in today’s fast-growing and competitive markets. Each software product serves various purposes in numerous fields such as medicine, science, building, accounting, agriculture, etc.  These software products must satisfy users’ needs in the best way.  Customers want more from software releases with new features to be implemented in a short time period, but they do not like to work with defected software.  Releasing software with any problems or bugs in it may affect the users’ experience, which makes a bad impact on quality reputation of the company name or brand.  Customers will remember a bad quality product. This is one of the reasons why testing of software before releasing is very important. [1, 2, 4, 5]
As an example to show why software must be tested before releasing to the user, say a group of software engineers from company A, have designed a new feature to display a login dialog box when the user logs in to a system or computer. This new feature is an additional request from a customer from company B, to enhance the current software.  Due to the short time period and deadline, the software engineers did not test the software after they developed it and assumed that, since this new feature represented just a small change to the code, it would not affect the existing software.  They also believed that no mistakes could be made in the software since most of software engineers in the team are senior engineers with more than 10 years’ experience.  They released this software to you as a user from company B.  You also have deadline and been excited to use this new release for a presentation.  You followed exactly the instructions which were provided to you by the software engineers in order to install this newly released software onto your system. Alas, when you complete the installation, a blue screen is displayed on your system due to a bug in the software.  You are not happy with this issue because you have to explain the situation with your manager, spend time to troubleshoot the problem, and then also describe the problem to the software engineers.  As a result, your tasks fall behind the schedule and you do not trust any products from the company A.  This is one situation where testing the software is very important before deploying it. Testing cannot be ignored because it impacts all the end users of a piece of software. If the software would have gotten tested thoroughly for all the possible user operations this problem would have been found well in advance and gotten fixed before the software was released.
Releasing the software without testing will mean there is a high risk of issues being found, and failure of the software. It would waste a lot of user time and effort when using it. It may actually result in more serious issues, such as security breaches, data loss and corruption. Software is written by people such as software engineers or developers. People are imperfect and make mistakes.  Some of those mistakes are unimportant, but some of them are expensive or dangerous.   Software testing is necessary because it helps in identifying defects. The testing process may not fix the entire defect present in the software application, and it cannot guarantee that the software is 100% error free, but it provides user friendliness.
I propose to write a quick technical manual on how to perform software testing.  A technical manual provides readers with the information they need to test software in general. Software is a set of instructions to perform some tasks. A technical manual also can serve either very specific or very general reader groups. The biggest challenges in designing technical manuals are often connected to determining and meeting the needs of my intended audiences. 
Audiences of this manual are software testing students, engineers, and professionals who would like to understand why software testing is important and the basics of how to do software testing. This would include types, methods, techniques and levels of testing. This manual provides enough ingredients to start with the software testing process, and allow readers can take themselves from low to higher levels of expertise. For engineering students who are probably not familiar with software testing theories, I will have to provide selected background information from my sources. My intent is to share with readers why software testing is important and the some basic ideas of software testing.
In this manual I will achieve the following goals: 1. the first goal is to explain the principles of software testing and some of the basic requirements of software.  2. The second primary goal is to provide some basic instructions of how to test software and types of software testing.  3. The third goal is to explain the benefits of software testing.  To achieve these goals, I will rely on four books as sources that give an overview and definitions.  I also will provide some instructions on how to perform software testing and an example of a simple test report template that helps readers understand and prepare for some of the software testing tasks.
Software testing is considered as one phase of the software development life cycle.  It is the process of finding bugs in the software. It is an activity to check whether the actual results match the expected results and to ensure that the software system is defect free. 
This section presents my schedule, costs and qualifications for performing the proposal manual project.  The proposed manual research project ends in a formal report that will be completed by July 29, 2015.  To reach this goal, I will follow the schedule present in table 1.  Since I already have several books and articles on software testing, most of my time will be spent selecting and reading through the information, finding the key results and sharing them with my audiences on testing.  There is no cost associated with performing this research manual because I obtain all my sources for from the library.   My manual research will be submitted online, therefore, there will be no cost associated with printing and spiral binding of the manual. 
Assignment
Due dates
pages
Preliminary Technical Manual Proposal
July 23
4
Technical Manual (task analysis)
July 24
3
Progress Report and Notes
July 25
3
Technical Manual (first draft)
July 26
10
Technical Manual (second draft)
July 27
14
Formal report
July 29
16
Table1. Assignment Sequence and Timetable
Qualifications: I graduated from Colorado State University in 2001 with a bachelor degree in Computer Engineering.  Since 2003, I have worked as a design-for-test (DFT) engineer and a test engineer for Intel and Agilent Technologies companies.  Presently, I am a quality assurance (QA) engineer at KeySight Technologies in Loveland, Colorado.  My main tasks in this position are: to write test scripts or programs, test documentations, test planning, and test reports, and to perform software testing on various software packages, systems and frameworks.  With over 12 years of software and system testing experiences, I believe that I will be able to share some ideas and issues that I learned from my hands-on experiences.  Besides my software testing experiences, I am interested in software testing education, i.e., developing materials and educational tools for teaching software testing or software engineering in general. Therefore, I think that I qualified for writing a manual on how to perform software testing.
In my Software Testing career, I have found that some people think that testing is all about executing scripts against an application with the view of trying to find as many errors in the code as possible.   But there are many other activities that are involved in testing. Some people also think that, when they are a senior programmer, they no longer need to test their code.   This only works in the ideal world, where the developers create software that works at the first go, and does not have any issues. In the real life, this is not often the case, because bugs appear in the software testing.  If  a bug was found after the software release, it means that time will be need to be spent finding a fix and performing more testing on it;  these tasks must take place while the end users are using the software. The time taken to fix defects after the software is released is significantly more than during the testing phase.  This is because the fixes need further testing, and need to align to any maintenance releases or other schedules that the organization has set up.  For this reason, I want to write this manual to show what is the importance of software testing? How to perform software testing? And, what are the various test types?









References

1.      Limaye, M. (2009). Software testing: Principles, techniques and tools. New Delhi: Tata McGraw-Hill Education Private.
2.      Patton, R. (2006). Software testing (2nd ed.). Indianapolis, IN: Sams Pub.
3.      PUMA- Publication MAnagement. (n.d.). Retrieved July 8, 2015, from http://puma.isti.cnr.it/rmydownload.php?filename=cnr.isti/cnr.isti/2004-TR-36/2004-TR-36.pdf.
4.      Javed, A. (2012). How To Improve Software Quality Assurance In Developing Countries. ACIJ Advanced Computing: An International Journal, 17-28.
5.      Myers, G., & Badgett, T. (2004). The art of software testing (2nd ed.). Hoboken, N.J.: John Wiley & Sons.




How to Perform Software Testing



 







 

How to Perform Software Testing









Colorado State University
Department of Electrical Engineering
JTC300 Professional and Technical Communication


Minh Anh Nguyen








Table of Contents

List of figures/illustrations                                                                                  3

Chapter One: Introduction of Software Testing                                      4

Chapter Two: Software or application requirements                                           5                     
2.1  Prerequisite to installation                                                                            5
2.2  Net framework                                                                                           5
2.3  Internet connection                                                                         5
2.4  Operating system                                                                                        5
2.5   Third party development tools                                                                    6
2.6  Hardware                                                                                                   6
           
Chapter Three: Testing                                                                          6
3.1  User requirements                                                                                       6
3.2  Functional testing requirements                                                                    6
3.3  Non-Functional testing requirements                                                            7
3.4  Test case or suite                                                                                        8
3.5  Test (execute test)                                                                                      8
3.6  Verified test case                                                                                        9
3.7  Ease of use                                                                                     9
3.8  Simple test report                                                                                       9
3.9  Scheduler                                                                                                   11

Chapter Four: Benefits of software testing                                                         11

Troubleshooting                                                                                                13

References                                                                                                       18

Glossary of Basic Terms                                                                                   19










List of figures

Figure1. Example of software test report template                                             10
Figure2. Example of software defect report template                                         16























Chapter One: Introduction of Software Testing

Software testing is the process of evaluating the developed system to assess the quality of the final product or computer software.  The purposes of software testing are to discover errors, mistakes, and flaws in the developed software and to ensure that software satisfies the specified requirements of customers. Software testing is a crucial part of the software development cycle [2, 4, 5,7]. It is closely linked to what has come to be known as structured analysis and design.  The software development cycle is a structure enforced on the development of a software product. During the development cycle, software engineers and test engineers are working together to find bugs and ensuring that the product must include all necessary functions and be compatible with the targeted environment [1,3,4,5]. Therefore, the software testing process consumes a lot of time during the software development cycle. The software development cycle begins when a software or product is released and made available for the public because there are suggestions for improvement from customers that can be taken into consideration in making the product or software better.
Software testing includes the development of procedural scripts that not only save time and resources when applications are updated, but also speed up the process of testing when regression testing is necessary. These scripts  also include the development of  test cases or suits, test reports, test results, analysis of test results, and instructions for setting up for each test case [1,4].
There are plenty of software testing books which will tell test engineers or users how to test well.  This user’s guide provides some tips, techniques, and guidance for developing software testing programs based on extensive practical experience.

Chapter Two: Software or application requirements

2.1 Prerequisite to installation
Before installing or updating any new software, users need to know some of the requirements for installation to avoid any unexpected difficulties during execution. This would be included uninstalling any previous or existing software, and identifying the Net framework, internet connection, operating system, etc.
2.2 .Net Framework

Net framework is a subject which is commonly bundled with different versions of Windows Operating System. Some frameworks such as Themis require .NET 4.0 runtime or higher to be installed in advance before launching the application. The .NET 4.0 program can be downloaded for free from Microsoft Download.
2.3 Internet Connection

If the software or application can be accessible through LAN or GPIB, then it will not require a faster internet line with a minimum of 1Mbps, since LAN connections typically come with at least 10Mbps.  However, if the application is accessible through the internet, then it will likely require a faster internet line with minimum of 1Mbps. This requirement is needed to avoid a high rate of failure.
2.4 Operating System
Some software or applications can only be installed and run on a 64-bit operating system. Users must check to see whether or not the software is compatible with their operating system such as Windows, Linux, Mac, etc, before they install the software.
2.5 Third Party Development Tool
Some applications or software require development tools which have been developed by other software companies such as Dell, Intel, etc.  The purpose of buying and using third party development tools is to facilitate getting the element values or results for specific tests in the HTML format which customers commonly request.
2.6 Hardware
Users also need to check the application/software compatibility with different hardware configurations. Hardware is mostly a computer, but sometime it is a test instrument. The computer hardware has a central processing unit (CPU); this CPU is the heart of the computer system and is responsible for executing, or running the software.  The CPU can only understand a series of binary numbers such as 0s and 1s, therefore the software must translate human language into series of codes which are made up of binary numbers [2, 3]. Each code has its own meanings and operations.  Hardware and Software are two main types of components required for a system to work.

Chapter Three: Testing

3.1  User Requirement
Defining user requirement is an important phase which helps software developers and testers in specifying system functions to make sure the program does what the user needs it to do.
3.2 Functional Testing Requirement
Functional testing addresses several requirements which are needed for successful software testing. It deals with requirement documentation and shows that any specific features will work. Therefore, function testing is done by using the design specifications provided by the design team or the functional specifications provided by customers. The goal of this type of testing is to verify that the software or application meets customer’s requirements, that it does not have any major bugs, and to ensure that the software is ready for release to the public. Functional testing is a black-box type of testing and is used to evaluate the software’s compliance with its specified requirements [2, 4, 5].  In actual or live mode, test engineers need to verify specific functions of the code.  This type of test would be executed before the non-function testing.  Four requirements of functional testing are: 1. Preparation of test data based on the specifications of functions. 2. Accurate representation and of customers’ requirements. 3. The execution of test cases. 4. Observation of the actual and expected test outputs. Some types or methods of functional tests are: unit testing, smoke testing, interface testing, regression testing and user-acceptance testing [2, 4, 3].
Positive and negative functional testing are two basic categories of functional testing.  Positive functional testing involves inputting valid inputs to see how the software application responds to these and also testing to determine if outputs are correct. Negative functional testing involves using different invalid inputs, unanticipated operating conditions and other invalid operations.
3.3 Non-Functional Testing Requirement
Non-functional testing is a type of testing of software for its non-functional requirements. It is designed specially to evaluate the readiness of a software or product to handle subjects or problems which are not covered by functional testing.  It is also used to measure and compare the results of testing attributes of software [3, 4, 5]. The goal of this test is to see whether or not the software stands up to customer expectations or behaves correctly and meets performance requirements.  Non-functional testing is used to determine whether or not the software will provide a good user experience.  This test can be started after the completion of functional testing. Non-functional testing has a great influence on customers.  This test demonstrates how much time the software will take to complete a task or responds to a request. Some types or methods of non-functional tests are: installation of software, loading of software, and configuration of software [2, 3, 5]. 
3.4 Test Case or suite

A test case is a sequence of steps that software test engineers or quality assurance engineers create to perform a certain test on the software or application. A good test suite is one that detects real faults because the set of faults in a program is unknowable. Test cases involve required inputs and are sometime referred to as test scenarios.  A test suite contains more than one test case and the successful completion of one test case must occur before the next test case can begin.  For example, test case 1: login hardware, test case 2: load software, test case 3: open test file, test case 4: run test file, test case 5: logout.  Test case 2 cannot start until test case 1 finishes.  A test suite is a collection of test cases, which is used to test a software or application to show how the software behaves.  A test suite also contains detailed instructions, descriptions or goals for each test case and information on the software configuration or prerequisite steps
3.5 Test (Execute test)
For any software, users can execute tests in two basic forms: actual or live mode and simulation mode. There are some differences between test runs in actual or simulation mode. Simulation mode only allows an application to perform a mock run on the defined test collection. The test results or values are dummy values that the users have created. The purpose of simulation is to avoid errors discovered during actual test runs. Any error messages which are received during the simulation must be fixed before a run in the live mode.   In actual test or live mode, the software allows users to define values but test results are real and generated by the system or instrument itself. Any mismatches or wrong values which are received during live mode testing are considered bugs.  Software test engineers must contact the software development engineers and explain the issues, and then both engineering teams must work together to resolve the issues before releasing the software to customers.
3.6 Verified Test Case
A verification process is needed to making sure that test case is setup correctly. The verification processes can be performed by eye-ball checking or simulation.
3.7 Ease of Use
Some software or applications are designed in such a way that navigation through them is user-friendly and lacks complexity.  Test engineers need to test all the new features which are embedded in the software to ensure that these features work correctly.
3.8 Simple Test Report
A test summary report is prepared at the end of testing and shared with all involved team members such as software developers, software testers, etc. The goal of a test report is to explain details and activities about the testing performed on the software to the project manager and team members.  Therefore, the test report must be clearly written, concise and easy to understand. The test report is extremely important because it is evidence for the testing performed and the testing conclusion.  During the test cycle, the test report is a communication sends out to the quality assurance team describing activities of the day or software behavior [2, 3].  The test report should include defect information, test description, and test case or requirement number.
Some companies have a test report template or special tool to generate test report, which makes work much easier for test engineers, because they only have to fill-in the test results.  If there is no test report template available, then test engineers or quality assurance engineers must create one. There are many types of test reports are available online.  A sample of a test report template that I created and presently used to submit test reports to my customers is shown in the figure below.
                        Figure1. Example of software test report template
Individual testers or test engineers will be sending their status on execution of the test cases or a test report to  the  lead  tester, project manager, and software engineers,  on  a  timely  basis  as  described  in  the  test  plan  document.
3.9 Scheduler
Careful logical scheduling is useful for implementing new ideas. Imagine that in large organizations, interaction occurs in a complex environment and each different host system has its own availability issues. Sometimes, test equipment and software require a special set up before engineers can perform any testing.  Therefore, in order to avoid wasting time to setup the equipment, software test engineers are often required to work in weird hours in order to test the software or application.

Chapter Four: Benefits of software testing


There are many different approaches and techniques to test software, with different benefits for software quality and the development process. Software Testing is a process of finding errors while executing a program. It is also the process of validation and verification of the software product, which in turn should deliver zero defects, and result in reliable and quality software products for users with more accurate and reliable results. Software testing provides a means to cut errors, reduce maintenance time or costs, and to reduce the cost of software development.  It provides a way of analyzing a software item to detect the differences between the software’s existing and required functionality.  Each piece of software or application needs to pass all of a testing program to ensure that consumers have a good experience with the product. Failure to release a working product that meets customer requirements can damage company’s reputation and reduce overall product sales.  During my experience in software testing, I have seen diverse business benefits such as improved quality, a gain in confidence for programmers, and earning trust from customers.
Testing techniques refer to different methods or ways of testing particular features of a computer program, system or product. Only the testing phase can ensure the quality of the software. Currently, there are many different software testing techniques and methods that software test engineers and software development engineers can use to improve software quality and reliability.









Troubleshooting

The goal of software troubleshooting is to filter out and resolve problems, and to restore the software to normal operation.  Software troubleshooting is the process of scanning, identifying, diagnosing and resolving technical or source-code-related problems, errors and bugs in software.  This can be done by using both functional and non-functional testing requirements.  Software troubleshooting can also be performed when software needs to be configured correctly, such as resolving issues due to incorrect installation or restoring software after corruption or file deletion caused by a virus.
There are many software troubleshooting techniques or steps that test engineers can use to find reasons for failure in software applications. Test engineers can use some of the troubleshooting tools to check basic connectivity issues, or when they set up a test environment. These tools are normally are normally called Third Party Development Tools such as  Agilent IO and NIMAX, and they can be brought from KeySight Technologies and National Instruments (NI) companies. Each test engineers would have their own methods and check list of troubleshooting procedures.  A sample check list of software troubleshooting techniques that I created and presently use to find the cause of failing tests and to confirm error messages is provided below. Following this check list will show whether a problem is not software related or not a real bug: 
1.      Check if software versions are compatible. If things don’t work in a particular environment the problem may be related to browser compatibility, the operating system version, product version, etc. This step will help avoid some invalid bugs being logged later by showing up front that the software is not supported in any way.
2.      If test engineers are using any configuration file for testing their application, then they must make sure this file is up to date as per the application requirements. Many times some global configuration file is used to pick or set some application flags. Failure to maintain this file as software requirements change will lead to malfunctioning of the application under test. This is not a bug.
3.      Check to see if a missing table is the main reason that the application will not work properly. Check to see if a database is a proper set.  This is not a bug.
4.      Check to see if there is any other hardware issue that is not related to the software application.
5.       Make sure that the application hardware and software prerequisites are correct.
6.      Check to see if all software components are installed properly on the computer or test instrument.
7.      Check whether registry entries are valid.
8.       For any failure, look into the ‘system event viewer’ for details. Test engineers can trace out many failure reasons from the system event log file.
The troubleshooting steps test engineers follow will be of great help during their test execution. Software troubleshooting will provide test engineers with answers to the following questions:
1.      What is not working?
2.      Why is it not working?
3.      How can you make it work?
4.      What are the possible reasons for the failure?

Test engineers need to perform software troubleshooting before they submit any bug reports.  This is to confirm that an error message is really a bug and to make sure that they did not miss any important flag or they did not configure their test setup properly.  The first step of the software troubleshooting process is identifying the problem.  In this step, test engineers will be checking on possible issues that can cause such problems and work on measures and then alternatives to find a solution.  Troubleshooting will help test engineers to decide whether the ambiguity they found in their software application under test is really a bug or any test setup mistake. 
If there is a bug or defect in the software application or product, test engineers must submit a bug or defect report so that software developers can fix the problem. Test engineers should try to get 100% of the bugs resolved. This requires bug reporting skill.
Most companies have a bug report template or special tool to generate a bug report. This template makes work much easier for test engineers because they only have to fill-in the test issue and error messages that they saw during the execution of the test and after troubleshooting steps.  If there is no bug report template available, then test engineers or quality assurance engineers must create one. There are many types of test bugs report template available online.  A sample of a test bug report template that I created and presently use to submit test bug reports to my software developers or engineers is shown in the figure below.
Figure2. Example of software defect report template
After any bug fix, test engineers must verify that the defect is fixed and they must retest all test cases to ensure that the fix will not cause any more issues for the software application. For example, say a test engineer is verifying a defect fix, and finds that there are 3 test cases failed due to this defect. Once the test engineer verifies the defect fix as resolved, the test engineer will retest or test the same functionality again by executing the same test cases that failed earlier. Test engineers should submit a closed bug report to close the issue, if they confirm that the defect is resolved.











References


1.      Javed, A. (2012). How To Improve Software Quality Assurance In Developing Countries. ACIJ Advanced Computing: An International Journal, 17-28.
2.      Limaye, M. (2009). Software testing: Principles, techniques and tools. New Delhi: Tata McGraw-Hill Education Private.
3.      Myers, G., & Badgett, T. (2004). The art of software testing (2nd ed.). Hoboken, N.J.: John Wiley & Sons.
4.      Patton, R. (2001). Software testing (1st ed.). Indianapolis, IN: Sams Pub.
5.      Patton, R. (2006). Software testing (2nd ed.). Indianapolis, IN: Sams Pub.
6.      PUMA- Publication MAnagement. (n.d.). Retrieved July 8, 2015, from http://puma.isti.cnr.it/rmydownload.php?filename=cnr.isti/cnr.isti/2004-TR-36/2004-TR-36.pdf.
7.      Tzmann, M., & Young, C. (2003). Software testing and internationalization (English ed.). Salt Lake City: Lemoine International.











Glossary of Basic Terms

·         Assigned To: The name of the person that is assigned to analyze or fix the defect.
·         Attachments: Attach any additional information like screenshots and logs.
·         Black box testing: is a functional testing.
·         Close Date: Date which the test engineer has checked the fixed issues and closed the issues.
·         Defect or Enhancement: Defect means the software is not working the way it says it will. Enhancement means the software is not working the way customers want.
·         Defect ID/Number: A unique ID number used to identify the issue in the issue tracking log.
·         Defect Priority: Priority of the Defect. How soon the problem report submitter requires a solution. The accepted values are:  1. Critical: issue will stop project progress if not resolved.  A solution is needed immediately. 2. High:  A solution is needed as soon as possible. 3. Medium: the problem should be solved in the next release. 4. Low: the problem should be solved in a future release.
·         Defect Severity: Severity of the Defect.
·         Defect Type: Type of the defect. It can be Functional, Installation or GUI issue
·         Descriptions: Any prerequisites or preconditions that must be fulfilled prior to executing the test.
·         Fixed by: Name of software engineer or developer who fixed the issue
·         GPIB (General Purpose Interface Bus):  is used to enable electronics test equipment to be controlled remotely.
·         Impact Summary: a description of the impact of the issue.  The impact may be expressed in terms of one or more of the following:  schedule, scope, resources, and space.  The impact description should also include a reference to any milestones.
·         Integration testing: is used to identify whether or not there is any bug in software.
·         LAN-local area network: it is a computer network
·         Load software testing: is used to check whether the application can load during develop software.
·         Mbps- Megabits per second: is referred to data transfer speeds and measured in megabits
·         Name of person doing test: The name of the person who executed the test
·         Open Date: Date on which the defect was initially reported and logged.
·         Planned Fix Build Version: Build version of the product where the defect was fixed
·         Problem Description: Detailed description of the defect.
·         Regression Testing is tested on the entire software and not just the new functionality and defect fixes.
·         Release/Build Version: Release/Build version of the product where the defect was detected.  
·         Reported by: The name of the person who reported the defect.
·         Requirement ID or test case ID: is the ID of the test case.
·         Smoke testing: is used to make sure the major functionality is working fine.
·         Software Build date: The date of build of the software.
·         Status: The status of the defect. 
·         Steps to Replicate: Step by steps to reproduce the defect.
·         Test Case Id: The test case that caused a bug to be raised.
·         Test date: The date of execution of the test.
·         Unit testing: It is useful to identify, determine bugs and errors
·         Validation: Step-by-step procedure to execute the test.