SmarteSoft

One of the most angst-ridden parts of the software development cycle is deciding when your product is ready to release. Functionality, performance, security and quality must be considered relative to specifications, and more often than not a judgment call must be made in the face of strong business pressure to release. Very often, the development cycle runs long and the Quality Assurance team doesn’t get nearly as much time as was scheduled to validate product quality. Still, QA is the final gatekeeper. If QA says ‘yes,’ the product releases amidst celebration. If they say no, there is a collective groan and the developers knuckle down to fix whatever must be fixed. Unnecessary delay hurts the business, just as too hasty a release (with attendant defects and customer complaints) also hurts the business. So the pressure on the QA Director is intense.

So how do you decide when a software product is ready?

Like all things it begins with planning at the start of the project. What are your test coverage goals?  What are your defect rate goals? How many defects of each severity are acceptable? What trend line must be established before you will feel confident releasing?

If these things (and more) are established at the start of the project, while not under intense pressure, and signoff on release metrics is gained from the key stakeholders at that time, then the release decision becomes a matter of keeping good metrics and using a little common sense for judgment calls. But if those metrics have not been established up front–or at least before decision time–then the whole process is fraught with subjectivity and emotion and the chances of someone making a bad decision are greatly increased.

Trend analysis for software quality is a whole art and science in itself, and some excellent books have been written on it. Being able to chart found versus fixed defects and their ratios, total defects of each severity and their trend, and keeping track of the impact of new builds on the stats, is a good starting point. When you’re fixing more than you’re finding, and each new build is introducing proportionately fewer defects than the previous one, while steadily reducing the overall defect rate, then you’re going in the right direction. This requires keeping whatever stats you have decided will be important from day one.

To add consistency to the defect measurement, and enable a rapid test cycle and assessment of quality, test automation is invaluable. By automating regression testing, you get to rapidly measure regressions–another key indicator of the quality trend–and can often spot a bad build almost immediately and scrap it before it does too much damage. Automation of regression testing also frees up your team to use their creativity to find new and better ways to try to break the application under test, and to test new functionality immediately rather than grinding through manual regression testing first.

A good test management tool (which enables everyone to see what is happening in the project based on their role, and includes visibility into the defect database), plus pre-planned metrics/objectives, plus trend analysis, plus test automation makes the release decision relatively stress free. Everyone can see the trend, and a common expectation becomes established that removes much of the emotion, and pressure, from the discussion. The decision on when to pull the trigger and release a product is seldom an easy one, but it can be made easier with careful planning, systematic measurement, and application of the right tools.