Monday, January 24, 2011

Friday, January 21, 2011

Software Installation/Uninstallation Testing

Have you performed software installation testing? How was the experience? Well, Installation testing (Implementation Testing) is quite interesting part of software testing life cycle.

Installation testing is like introducing a guest in your home. The new guest should be properly introduced to all the family members in order to feel him comfortable. Installation of new software is also quite like above example.

If your installation is successful on the new system then customer will be definitely happy but what if things are completely opposite. If installation fails then our program will not work on that system not only this but can leave user’s system badly damaged. User might require to reinstall the full operating system.

In above case will you make any impression on user? Definitely not! Your first impression to make a loyal customer is ruined due to incomplete installation testing. What you need to do for a good first impression? Test the installer appropriately with combination of both manual and automated processes on different machines with different configuration. Major concerned of installation testing is Time! It requires lot of time to even execute a single test case. If you are going to test a big application installer then think about time required to perform such a many test cases on different configurations.

We will see different methods to perform manual installer testing and some basic guideline for automating the installation process.

To start installation testing first decide on how many different system configurations you want to test the installation. Prepare one basic hard disk drive. Format this HDD with most common or default file system, install most common operating system (Windows) on this HDD. Install some basic required components on this HDD. Each time create images of this base HDD and you can create other configurations on this base drive. Make one set of each configuration like Operating system and file format to be used for further testing.

How we can use automation in this process? Well make some systems dedicated for creating basic images (use software’s like Norton Ghost for creating exact images of operating system quickly) of base configuration. This will save your tremendous time in each test case. For example if time to install one OS with basic configuration is say 1 hour then for each test case on fresh OS you will require 1+ hour. But creating image of OS will hardly require 5 to 10 minutes and you will save approximately 40 to 50 minutes!

You can use one operating system with multiple attempts of installation of installer. Each time uninstalling the application and preparing the base state for next test case. Be careful here that your uninstallation program should be tested before and should be working fine.

Installation testing tips with some broad test cases:

1) Use flow diagrams to perform installation testing. Flow diagrams simplify our task. See example flow diagram for basic installation testing test case.

Add some more test cases on this basic flow chart Such as if our application is not the first release then try to add different logical installation paths.

2) If you have previously installed compact basic version of application then in next test case install the full application version on the same path as used for compact version.

3) If you are using flow diagram to test different files to be written on disk while installation then use the same flow diagram in reverse order to test uninstallation of all the installed files on disk.

4) Use flow diagrams toautomate the testing efforts. It will be very easy to convert diagrams into automated scripts.

5) Test the installer scripts used for checking the required disk space. If installer is prompting required disk space 1MB, then make sure exactly 1MB is used or whether more disk space utilized during installation. If yes flag this as error.

6) Test disk space requirement on different file system format. Like FAT16 will require more space than efficient NTFS or FAT32 file systems.

7) If possible set a dedicated system for only creating disk images. As said above this will save your testing time.

8 ) Use distributed testing environment in order to carry out installation testing. Distributed environment simply save your time and you can effectively manage all the different test cases from a single machine. The good approach for this is to create a master machine, which will drive different slave machines on network. You can start installation simultaneously on different machine from the master system.

9) Try to automate the routine to test the number of files to be written on disk. You can maintain this file list to be written on disk in and excel sheet and can give this list as a input to automated script that will check each and every path to verify the correct installation.

10) Use software’s available freely in market to verify registry changes on successful installation. Verify the registry changes with your expected change list after installation.

11) Forcefully break the installation process in between. See the behavior of system and whether system recovers to its original state without any issues. You can test this “break of installation” on every installation step.

12) Disk space checking: This is the crucial checking in the installation-testing scenario. You can choose different manual and automated methods to do this checking. In manual methods you can check free disk space available on drive before installation and disk space reported by installer script to check whether installer is calculating and reporting disk space accurately. Check the disk space after the installation to verify accurate usage of installation disk space. Run various combination of disk space availability by using some tools to automatically making disk space full while installation. Check system behavior on low disk space conditions while installation.

13) As you check installation you can test for uninstallation also. Before each new iteration of installation make sure that all the files written to disk are removed after uninstallation. Some times uninstallation routine removes files from only last upgraded installation keeping the old version files untouched. Also check for rebooting option after uninstallation manually and forcefully not to reboot.

I have addressed many areas of manual as well as automated installation testing procedure. Still there are many areas you need to focus on depending on the complexity of your software under installation. These not addressed important tasks includesinstallation over the network, online installation, patch installation, Database checking on Installation, Shared DLL installation and uninstallation etc.

Hope this article will be a basic guideline to those having trouble to start with software installation testing both manually or in automation.

Regression Testing with Regression Testing Tools and methods

What is Regression Software Testing?
Regression means retesting the unchanged parts of the application. Test cases are re-executed in order to check whether previous functionality of application is working fine and new changes have not introduced any new bugs.

This is the method of verification. Verifying that the bugs are fixed and the newly added feature have not created in problem in previous working version of software.

Why regression Testing?
Regression testing is initiated when programmer fix any bug or add new code for new functionality to the system. It is a quality measure to check that new code complies with old code and unmodified code is not getting affected.
Most of the time testing team has task to check the last minute changes in the system. In such situation testing only affected application area in necessary to complete the testing process in time with covering all major system aspects.

How much regression testing?
This depends on the scope of new added feature. If the scope of the fix or feature is large then the application area getting affected is quite large and testing should be thoroughly including all the application test cases. But this can be effectively decided when tester gets input from developer about the scope, nature and amount of change.

What we do in regression testing?

Rerunning the previously conducted tests

Comparing current results with previously executed test results.

Regression Testing Tools:
Automated Regression testing is the testing area where we can automate most of the testing efforts. We run all the previously executed test cases this means we have test case set available and running these test cases manually is time consuming. We know the expected results so automating these test cases is time saving and efficient regression testing method. Extent of automation depends on the number of test cases that are going to remain applicable over the time. If test cases are varying time to time as application scope goes on increasing then automation of regression procedure will be the waste of time.

Most of the regression testing tools are record and playback type. Means you will record the test cases by navigating through the AUT and verify whether expected results are coming or not.
Example regression testing tools are:

Winrunner

Rational Functional Tester

SilkTest

Most of the tools are both Functional as well as regression testing tools.

Regression Testing Of GUI application:
It is difficult to perform GUI(Graphical User Interface) regression testing when GUI structure is modified. The test cases written on old GUI either becomes obsolete or need to reuse. Reusing the regression testing test cases means GUI test cases are modified according to new GUI. But this task becomes cumbersome if you have large set of GUI test cases.

Black Box Testing: Types and techniques of BBT

I have covered what is White box Testing in previous article. Here I will concentrate on Black box testing. BBT advantages, disadvantages and and How Black box testing is performed i.e the black box testing techniques.

Black box testing treats the system as a “black-box”, so it doesn’t explicitly use Knowledge of the internal structure or code. Or in other words the Test engineer need not know the internal working of the “Black box” or application.

Main focus in black box testing is on functionality of the system as a whole. The term ‘behavioral testing’ is also used for black box testing and white box testing is also sometimes called ‘structural testing’. Behavioral test design is slightly different from black-box test design because the use of internal knowledge isn’t strictly forbidden, but it’s still discouraged.

Each testing method has its own advantages and disadvantages. There are some bugs that cannot be found using only black box or only white box. Majority of the applicationa are tested by black box testing method. We need to cover majority of test cases so that most of the bugs will get discovered by blackbox testing.

Black box testing occurs throughout the software development and Testing life cycle i.e in Unit, Integration, System, Acceptance and regression testing stages.

Tools used for Black Box testing:
Black box testing tools are mainly record and playback tools. These tools are used for regression testing that to check whether new build has created any bug in previous working application functionality. These record and playback tools records test cases in the form of some scripts like TSL, VB script, Java script, Perl.

Advantages of Black Box Testing
- Tester can be non-technical.
- Used to verify contradictions in actual system and the specifications.
- Test cases can be designed as soon as the functional specifications are complete

Disadvantages of Black Box Testing
- The test inputs needs to be from large sample space.
- It is difficult to identify all possible inputs in limited testing time. So writing test cases is slow and difficult
- Chances of having unidentified paths during this testing

Methods of Black box Testing:

Graph Based Testing Methods:
Each and every application is build up of some objects. All such objects are identified and graph is prepared. From this object graph each object relationship is identified and test cases written accordingly to discover the errors.

Error Guessing:
This is purely based on previous experience and judgment of tester. Error Guessing is the art of guessing where errors can be hidden. For this technique there are no specific tools, writing the test cases that cover all the application paths.

Boundary Value Analysis:
Many systems have tendency to fail on boundary. So testing boundry values of application is important. Boundary Value Analysis (BVA) is a test Functional Testing technique where the extreme boundary values are chosen. Boundary values include maximum, minimum, just inside/outside boundaries, typical values, and error values.

Extends equivalence partitioning
Test both sides of each boundary
Look at output boundaries for test cases too
Test min, min-1, max, max+1, typical values

BVA techniques:
1. Number of variables
For n variables: BVA yields 4n + 1 test cases.
2. Kinds of ranges
Generalizing ranges depends on the nature or type of variables
Advantages of Boundary Value Analysis
1. Robustness Testing – Boundary Value Analysis plus values that go beyond the limits
2. Min – 1, Min, Min +1, Nom, Max -1, Max, Max +1
3. Forces attention to exception handling

Limitations of Boundary Value Analysis
Boundary value testing is efficient only for variables of fixed values i.e boundary.

Equivalence Partitioning:
Equivalence partitioning is a black box testing method that divides the input domain of a program into classes of data from which test cases can be derived.

How is this partitioning performed while testing:
1. If an input condition specifies a range, one valid and one two invalid classes are defined.
2. If an input condition requires a specific value, one valid and two invalid equivalence classes are defined.
3. If an input condition specifies a member of a set, one valid and one invalid equivalence class is defined.
4. If an input condition is Boolean, one valid and one invalid class is defined.

Comparison Testing:
Different independent versions of same software are used to compare to each other for testing in this method.

Reference- http://www.softrel.org/stgb.html

Web Testing test cases

Web Testing, Example Test cases

WEB TESTING
While testing a web application you need to consider following Cases:

• Functionality Testing
• Performance Testing
• Usability Testing
• Server Side Interface
• Client Side Compatibility
• Security
Functionality:
In testing the functionality of the web sites the following should be tested:
• Links
i. Internal Links
ii. External Links
iii. Mail Links
iv. Broken Links
• Forms
i. Field validation
ii. Error message for wrong input
iii. Optional and Mandatory fields
• Database
* Testing will be done on the database integrity.
• Cookies
* Testing will be done on the client system side, on the temporary Internet files.
Performance :
Performance testing can be applied to understand the web site’s scalability, or to benchmark the performance in the environment of third party products such as servers and middleware for potential purchase.
• Connection Speed:
Tested over various networks like Dial Up, ISDN etc
• Load:
i. What is the no. of users per time?
ii. Check for peak loads and how system behaves
iii. Large amount of data accessed by user
• Stress:
i. Continuous Load
ii. Performance of memory, CPU, file handling etc..
Usability:
Usability testing is the process by which the human-computer interaction characteristics of a system are measured, and weaknesses are identified for correction.
• Ease of learning
• Navigation
• Subjective user satisfaction
• General appearance
Server Side Interface:
In web testing the server side interface should be tested. This is done by verify that communication is done properly. Compatibility of server with software, hardware, network and database should be tested.
Client Side Compatibility:
The client side compatibility is also tested in various platforms, using various browsers etc.
Security:
The primary reason for testing the security of a web is to identify potential vulnerabilities and subsequently repair them.
• Network Scanning
• Vulnerability Scanning
• Password Cracking
• Log Review
• Integrity Checkers
• Virus Detection

Manual Testing

10 Tips to Help You Achieve Your Software Testing Documentation Goal

Note: If you missed the first part of this post please read it: Why Documentation is important in testing?

As I mention in my earlier post, in general, understanding about software testing documentation is “It can be done only by the person who has free time”. We need to change this mindset, and then only we can leverage documentation power on our projects.

It’s not that we don’t know how to do the documentation right. We just don’t think it’s important.

Everyone must have standard templates for all the kinds of documentation starting from Test strategy, test Plan, Test cases, and Test data to Bug report. These are just to follow some standards (CMMI, ISO etc.) but, when it comes to actual implementation how many of these documents are really used by us? We just need to synchronize our quality process with documentation standards and other process in an organization.

The simplest thing to follow all kind of documentation is to involve a person in project from kick off phase who understands the project dynamics, domain, objective, and technology. And who else better than a QA person for this (of course there are technical writers present to do this – but considering a general scenario of small companies where technical writers are not present).

To achieve this goal of testing and documentation I feel we need to focus on some points.

Here are the top 10 tips to help you achieve your software testing documentation goal:

1. QA should involve at the very first phase of project so that QA and Documentation work hand in hand.

2. Process defined by QA should follow by technical people, this helps remove most of the defects at very initial stage.

3. Only creating and maintaining software testing templates is not enough, force people to use them.

4. Don’t only create and leave document, Update as and when required.

5. Change requirement is important phase of project don’t forget to add them in the list.

6. Use version controlling for everything. This will help you manage and track your documents easily.

7. Make defect remediation process easier by documenting all defects. Make sure to include clear description of defect, reproduce steps, affected area and details about author while documenting any defect.

8. Try to document what is required for you to understand your work and what you will need to produce to your stakeholders whenever required.

9. Use standard template for documentation. Like any excel sheet template or doc file template and stick to it for all your document needs.

10. Share all project related documents at single location, accessible to every team member for reference as well to update whenever required.

I am not saying that by applying above steps you will get sudden results. I know this change won’t happen in a day or two, but at least we can start so that these changes start happening slowly.

After all “the documentation needs documentation”. Isn’t it?

Share your tips. How you maintain software testing documents related to your project?

Why Documentation is Important in Software Testing

In my Software Testing career, I never heard people talking much about software testing documentation. The general opinion about testing documentation is that, anyone who has free time can do the documentation like Test case, Test plan, status report, Bug report, project proposal etc.
Even I did not stress more on the documentation, but I can say it’s my habit to place all the data in black and white and to update others about that as well.
Just want to share my experience with you:
We had delivered a project (with an unknown issue in that) to one of our client (angry client). And they found issue at Client side, which was very bad situation for us, and as usual all blame was on QA’s. The
issue was something regarding compatibility of one website. When it came to me, I was having proof that I didn’t get such requirement document which state I have to check compatibility of the website also. Thank god I was safe. That was the lesson for me, I realized importance of documentation and from that day I started to work on documents and created testing documents like Test plan, Test cases, sanity testing checklist, bug report and many.

“Ink is better than the best memory” – Chinese proverb
Software Testing Documentation: What’s that?
We all read various articles on testing technologies and methods, but how many of us have seen articles on documentation? No doubt there are few, Is it that documents are not important? No, but its’ because we have not yet realized importance of documents.
But, if we observe then the fact is, projects that have all the documents have high level of maturity. Most companies do not give even a little importance to the documentation as much they give to software development process. If we search on web then we can find various templates on how to create various types of documents. But how many of them are really used by organizations or individuals?
Fact is that, careful documentation can save an organization’s time, efforts and money. While going for any type of certification, why there is stress given on documentation, it’s just because it shows importance of client and processes to individual and organization. Unless you are able to produce document that is comfortable to user no matter how good your product is, no one is going to accept it.
It’s my experience, we own one product, which is having a bit confusing functionality. When I started working on that I asked for some help documents to Manager and I got answer “No, we don’t have any documents” Then I made an issue of that, because as a QA I knew, no one can understand how to use the product without documents or training. And if user is not satisfied, how we are going to make money out of that product?
“Lack of documentation is becoming a problem for acceptance” – Wietse Venema
Even same thing is applicable for User manuals. Take an example of Microsoft, they launch every product with proper documents, even for Office 2007 we have such documents, which are very explanatory and easy to understand for any user. That’s one of the reasons that all their products are successful.
In small companies, we always heard “project rejects in proposal or kickoff phase” it’s just because proposal documentation lacks concise and expressive language, and to show the capability of the organization. It’s not that small companies can’t deliver good quality projects but it’s their inability to express their capability. (Me also working with a small organization of 80 employees, and I heard this many time)
I personally feel Quality is the only department that can make it possible. We are the only department, which can argue on this and can provide successful future to our organizations.
Let’s organize all discussion in few points in quality perspective:
- Clarify quality objective and methods
- Ensure clarity about tasks and consistency of performance
- Ensure internal co-ordination in client work
- Provide feedback for preventive actions
- Provide feedback for your planning cycle
- Create objective evidence of your quality management system’s performance
There are hundreds of documents used in software development and testing life cycle. Here I am listing few important software testing documents that we need to use/maintain regularly:
1) Test plan
2) Test design and Test case specification
3) Test Strategy
4) Test summary reports
5) Weekly Status Report
6) User Documents/ manuals
7) User Acceptance Report
8 ) Risk Assessment
9) Test Log
10) Bug reports
11) Test data
12) Test analysis
Also Software testers regularly need refer following documents:
1) Software requirement specifications
2) Functional documents
Summary:
Software Testing Documents always play an important role in Project development/testing phase. So always keep things documented whenever possible. Don’t rely on verbal communication. Be always on safe side. Documentation will not only save you but also help organization in long run saving thousands of dollars on training and more importantly on fixing issues caused due to lack of development and testing documents. Don’t document just to avoid finger pointing on you, but habit of documentation will certainly bring a systematic approach in your testing process, leaving the ad hoc testing behind.
I will soon post “Why doing it is Important” second part of this post.
About Author: Tejaswini patil
I am working with an E-learning organization as an Associate Manager QA. I think I got very huge responsibility in very short span of my career due to my dedication towards work. I love my job and my team says I am very strict, but I feel if you want good quality you have to be.

Installation Testing