Is more testing the answer?
It’s not long after you’ve released a new product, or feature, and the call comes in that something is not right. Should have done more testing, yes? Well, maybe. What if you couldn’t do more testing because you just didn’t have the time? In hindsight, should you have factored in more time for testing into the planning phase?
In order to answer these questions we need to understand what we are trying to achieve when we test our software. We also need to consider the nature of the projects we are working on, the composition of the development team, and the cost of testing.
What are the goals?
I opened this post by saying “…the call comes in that something is not right”. I chose these words carefully. By saying that something is not right, what we are actually saying is that the product is not behaving, or does not look, as was intended. It is not great software. When we test our software, we need to go beyond, simply ensuring that it doesn’t throw errors, to making sure that it does and looks as was intended. Whether it’s an error, or an image that isn’t in the correct place, all should be treated as faults. The goal of testing is to identify these faults.
How critical is the correctness of your software?
If your software doesn’t work as intended, would the results be life-threatening? If so, you will need a considerable number of quality measures in place. Testing should be high on your agenda regardless of the number of faults that have made their way into the code. If the results are more of an inconvenience then you have greater flexibility in how to prevent faults from occurring.
What are the alternatives to more testing?
In some companies, there are teams responsible solely for software testing, whilst in others the developer is responsible for testing their own code. In reality, developers perform their own ‘informal’ testing, whilst writing the code, regardless of whether there is a dedicated testing team or not. In an ideal world, this informal testing would be sufficient to remove all the faults, and the ‘formal’ testing phase (by the testing team or otherwise) would simply validate that everything is as intended. In the real world, the formal testing phase would highlight at least a few faults, which would then be passed back to the developer for correction and re-submission to testing. The time and cost required for this develop-test iterative cycle is proportionate to the scale of the software, the number of faults found, and the developer’s ability to fix them. Therefore, it should be possible to reduce costs, and the need for additional testing, by educating the developer on how to reduce the number faults in their code in the first place. Furthermore, improvements in the requirements gathering and planning phases will ensure prevention of feature faults.
In a future post I’ll discuss what can be done to reduce fault creation, but for now remember that more testing might not always be the best approach, in the pursuit of fault free great software.
In order to answer these questions we need to understand what we are trying to achieve when we test our software. We also need to consider the nature of the projects we are working on, the composition of the development team, and the cost of testing.
What are the goals?
I opened this post by saying “…the call comes in that something is not right”. I chose these words carefully. By saying that something is not right, what we are actually saying is that the product is not behaving, or does not look, as was intended. It is not great software. When we test our software, we need to go beyond, simply ensuring that it doesn’t throw errors, to making sure that it does and looks as was intended. Whether it’s an error, or an image that isn’t in the correct place, all should be treated as faults. The goal of testing is to identify these faults.
How critical is the correctness of your software?
If your software doesn’t work as intended, would the results be life-threatening? If so, you will need a considerable number of quality measures in place. Testing should be high on your agenda regardless of the number of faults that have made their way into the code. If the results are more of an inconvenience then you have greater flexibility in how to prevent faults from occurring.
What are the alternatives to more testing?
In some companies, there are teams responsible solely for software testing, whilst in others the developer is responsible for testing their own code. In reality, developers perform their own ‘informal’ testing, whilst writing the code, regardless of whether there is a dedicated testing team or not. In an ideal world, this informal testing would be sufficient to remove all the faults, and the ‘formal’ testing phase (by the testing team or otherwise) would simply validate that everything is as intended. In the real world, the formal testing phase would highlight at least a few faults, which would then be passed back to the developer for correction and re-submission to testing. The time and cost required for this develop-test iterative cycle is proportionate to the scale of the software, the number of faults found, and the developer’s ability to fix them. Therefore, it should be possible to reduce costs, and the need for additional testing, by educating the developer on how to reduce the number faults in their code in the first place. Furthermore, improvements in the requirements gathering and planning phases will ensure prevention of feature faults.
In a future post I’ll discuss what can be done to reduce fault creation, but for now remember that more testing might not always be the best approach, in the pursuit of fault free great software.
Comments
Post a Comment