Testing World 2006

Hi All,

I have attened the Testing World 2006 conference today & the response was tremendus. The session on Testing Skills That No One Talk About by Michael Bolton was too good and put me in different thought process on the areas left out to explore.

He talks about

1. General Systems thinking
2. Critical Thinking
3. Debating Skills

Testing World 2006 is the conference for building your expertise in testing projects execution & various products. It's two power-packed days of practical information you can take back to your job and put to work immediately.

These kind of conferences helps testers to know what's happening around and how can we benefit from the same.

Applying Static Analysis for Software Testing

In this post, I will be sharing my views on Static Analysis and how the same is useful for Software Testing. Using Static Analysis in Testing will also be called as Static Testing.

The generic definition for Static Testing is that

Testing of an Application at the specification or implementation level without execution of that Application.

Static Analysis helps a lot for better Code Quality and we can perform the following set activities against the source code.

1. Capture Metrics
2. Complexity of the System (eg. Cyclomatic Complexity)
3. Compliance against Standards (eg. check for the Java Coding Guidelines)
4. Exception Handling (Captures the code that contains Unhandled exceptions)
5. Infinite Loops
6. Copy Paste Dectors (Most of the developers used to reuse the code via a copy paste and introduce some issues in the process)
7. Duplicate Code
8. Dead Code

We should be able to restrict most of the above said issues at very early stage of SDLC by applying Static Testing techniques and make these part of the build process.

Finding Bugs in Java Programs with just one mouse click

Come across this cool video about Using Static Analysis For Software Defect Detection on Google Tech Talks. William Pugh talk about how can we use Static Analysis for defect detection and his tool FindBugs which is Static Analysis tool for Java Applications.

The tool is amazing and with just with one mouse click, we can uncover so many issues that we didn't even think of.

Get more info on FindBugs here

Link to the Video : Using Static Analysis For Software Defect Detection

Abstract of the Video:

Paugh talks about some of my experience in using and expanding static analysis tools for defect detection. The FindBugs tool developed at the University of Maryland is now being widely used, including inside Google.

he will give an overview of FindBugs, show some of the kinds of errors we routinely find in production code, discuss the methodology we use for enhancing and expanding FindBugs and some of the recent additions to it, discuss ways of incorporating FindBugs into your development process (such as being able to get a report of all the warnings introduced since the last release of your software), and talk about the future of static analysis, including things such as a new Java JSR to provide standard annotations for things such as @NonNull and @Tainted.

Is Counting a Bad Idea for Test Effort?

I was going through the post Why counting is a bad idea on Shrini's Blog on Software Testing. Shirni has listed a classic example on why counting is a bad idea for Testing.

Even though i like this post, I won't agree fully on the views expressed there. Probably we might need to improve our selves with the process of measuring.

Certain artifacts like test cases, requirements and bugs are not countable things and any attempt to count them can only lead to manipulations and ill-informed decisions.

Wait — Are there any things at all in testing that we can count without loss of effectiveness and usefulness of the information that a counted number might reveal?

Let me describe my stand clearly wrt to this Why counting is a bad idea.

I do agree that it's not posssible to count all the activities that we do in testing. But then that doesn't mean like all the Test related metrics are bad. May be, We can't generalize them since it leads to lot of subjective debate.

When it comes push the usefulness and effectiveness of the information by the metric, this is more of a improvement process. All these metrics are subjective to the respective projects / environments / organizations. Any attempt in making this as a genaral metric will lead to manipulations and ill-informed decisions.

1. Bad Idea for whom ?
   Is this for Customers / Management / Developers / Testers / Others
2. If Counting is a bad Idea, how do we measure the effort spent for the Project ? Do we need to go by the gut feelings of te people ? Will there be any data to justify our decisions ?

Most of the metrics used for Software Development for that matter are subjective. It varies based on the application, technology, domain, global support etc. It's tough to make them general like we do in other industries. So the same will be applicable for Testing too.

Some thing like below are very much subjective and won't be accepted in genaral.

Productivity

1. No of Test cases prepared Per Person Per hour = 5
2. No of Test cases executed per person per hour = 15

Look at the subjectivity in the above scenario.

1. What is the structure of the Test Case ?
2. How many user actions will be there in genaral per TestCase ?
3. What is the Volume of Test Data associated with each TestCase ?
4. What information available for these Test Cases (SRS, Design, UseCases etc) ?
5. What kind of test cases are they writing ? Are they writing for Unit, Integration, System, Accepatance, Performance etc..
6. What kind of Test Design Tech are they following (BVA, EP, DT, OA, CE etc)

There will be so many queries that will be come up here and it's tough to measure some one just based on the number in genaral. But then since it's subjective to the project (team), the respective stake holder should be able to take some decision based on that number.

But then, we can't make this number generic. Probably this might be useful to the similar projects to have some rough estimate.

At the end, i shouldbe able to give qualitaive (again this is subjective) to the stake holders. If some one expects 100% qualitaive info here, that's an ideal world to which we can't go as of now.

What does that mean ?

I should be able to give a number. My metrics should tell me the number of Test Cases, Issues etc wrt the requirement.

Otherwise we may not have clarity on what we are doing & what are we trying to deliver. Things will be going on the pure gutfeelings

Pradeep has come up with nice replies to my comments on the post.

@Pradeep,

Thanks for sharing your views.

Let me look at your requirement

" This field should take only alphabets and nothing else other than that "

In the first place, the requirement is not clear for me

1. What kind of Application are we taking about ?
2. What is the size of this field (Some thing like password. It should have a Min and Max length)
3. When you say Alphabets, Do you mean to support all the alphabets in all the languages or just english or some thing else?
4. ????

See Pradeep, my queries will go on like this and all these will help me in getting the clarity on the requirement. This is how i try to to understand the system.

This process helps me in knowing on what is the requirement, it's advantages, it's limitations and most possible scenarios where this can be used.

All the above will help me in capturing the required tests for the above requirement.

But this may not be generic on the whole but then it can be generic to the Organization or at least to the project people are working with.

See there should be some way to measure on what we are doing. This measure should help the stake holders to take a decision on the quality of the application under test.

I do agree on the fact that we need to improve ourselves in coming up with measures that have qualitative information for the stake holders.

@ Shrini, Thanks for coming up a post that has good debate.

That’s expected behavior and Not a Bug

I keep hering the following phrases from our Development Team here

1. That's an expected behavior and Not a Bug
2. That's by Design

Today, I have an interesting debate with our Product Manager on a Issue. He put is as "That's an Expected Behavior and Not a Bug"

I have been looking at root causes of these problems & why do run into this kind of debates always. The following limitations came

We could have avoided the same incase if

1. Clarity on the Requirements across all the teams 
2. Limitations of the Technology
3. Usability Principles (UI Guidelines)

The above two arguments can be avoided if both teams have clarity on the above three.

Tester is not a Quality Police

The perception from most of the management is that the Testers are the quality police for the products / projects that they develop. This is true for most of the companies where there is no Quality Assurance group.

What do i mean by a Quality Police ?

1. Is the requirements freeze done for the product
2. Did we write all the specs like SRS, Design Docs, Uses Cases
3. Are the developers writing Unit Tests ?
4. Are the developers testing the builds before pass on the same to Test
5. This list goes on…

There are testers who are trying to do the above and i call them as Quality Police .

Where does this attitude come from ?

The attitude of acting like a Quality Police for testers is injected by the management from the beginning of their careers. So they expects the same as they go along with their career and this attitude spoils the relationships between developers and testers.

Acting like Quality Police and trying to get things done is more of process which need to be followed by all the teams and let's not take that up. Let the management take care of the process management too.

What is Quality ?

Wikipedia describes quality assurance as follows

Quality Assurance is the activity of providing evidence needed to establish confidence among all concerned, that the quality-related activities are being performed effectively. All those planned or systematic actions necessary to provide adequate confidence that a product or service will satisfy given requirements for quality. Quality Assurance is a part and consistent pair of quality management proving fact-based external confidence to customers and other stakeholders that product meets needs, expectations, and other requirements. QA (quality assurance) assures the existence and effectiveness of procedures that attempt to make sure – in advance – that the expected levels of quality will be reached.

Read more on Quality Assurance from wikipedia

It's worth noting here that Good Quality doesn't come from less bugs in the product. I have seen people whose assumption is that if a product contains bugs it's of less quality and in case if it doesn't it's of more quality. I would say that we can't measure the quality of the products just with Bugs. I have never seen any product which doesn't have bugs in it.

Are the Testers responsible for the Product Quality ?

The Testers test the product and report bugs. Some of these bugs will get fixed for the release in case if they are import. You can't build quality into systems just by reporting bugs and helping developers to fix them.

Assuring the quality into Products under development is the responsibility of all the teams associated with Product Development

So What should be the Role of Tester then

The role of the Tester to test software, find bugs, report them so that they can be fixed. The Bug Reports should be clear, easy to reproduce, reduce the time to debug the same for developers and the report should motivate the developers to fix the issue asap.

Tester should focuses on the software product itself and gathers important information regarding what it does and doesn't do. The process of gathering should include all the teams associated with the product. Talks to management, sales, architects, development, support and gather their expectations over the release.

Don't you think that this is a big job all by itself and need lot of time. As i keep saying the Role of the Tester is to provide qualitative information on the product to the management for the better decisions. So the big challenge here is to provide accurate, comprehensive, and timely information about the product under development.

KLOC – What does it mean to Software Testing

Introduction to KLOC

Lines of Code (LOC) is one of the software metric that is used by most of the people for Software Measurement. Thousand lines of code is treated as KLOC. This metric helps us in knowing the size and complexity of the Software Application. Click here for more details on KLOC. This info is from wikipedia

What does it mean to Software Testing

We do test applications with the intention to see if the promised functionality works fine or not. Any deviation here will be considered as a Bug. So each of these bugs must be originated from some line of code in the product.

So it's understood that when the size of the code is more there is a chance for more number of bugs in the prodcut. Even most of the process to talk about some % of issues is fine or acceptable quality per KLOC(even though there is lot of subjectivity).

The Defect Density is arrived at Number of Bugs / KLOC per the product under test. The defect density is one of the metric used to measure the quality of the product. Most of the Quality Process does talk about this metric.

Concerns

The concern in this approach is that how these values are measured. The general bias with KLOC is that people try to see that only the excutable lines of code in the product.

Each every line in the product may not be code at all & we may not execute each and every one of them. So it's not taken care, then the issues related to documentation, images, installation etc might be misleading.

Since we are looking at KLOC as the size of the product it's better to include each and every entity that effects the same. Then it's helpful for both development and test teams.