Archive for the ‘General Comment’ Category

SmarteSoft Webinars

Tuesday, December 8th, 2009

Come January, SmarteSoft will begin offering a series of webinars, which we hope will help you better understand all of the benefits of test automation. These interactive lectures, each about an hour long, will be broken down into two basic categories: business issues and technical issues. We’ve come up with a few topic ideas (The Economics of Test Automation, Which Licensing Scheme is Right For You?, and How Can Test Automation Keep Pace with the Speed of Application Advancement?, for example), but we would like to hear your thoughts on what might make for good webinar material.

So: What do you think? Let us know by emailing our sales team.

Ticks and Plovers

Monday, September 21st, 2009

Ticks and mosquitoes prosper by sucking the blood of their victims, often causing infection, irritation, and sometimes even death to their host in the process. It is not a symbiotic relationship; when we are their victims we try to kill them.

On the other hand, plovers clean the teeth of crocodiles, and shrimp clean the grunge off of fish, and all benefit. Crocs and fish welcome these hygienists and leave them unharmed.

IT consultants would prefer to be thought of as plovers. And indeed, most of the time, they provide a beneficial service to their customer and are rewarded with something in return. The greater the benefit relative to the cost, the more in demand their services are likely to be.

However, what happens if the consultant is faced with a situation where they can greatly enhance the benefit to the customer, but with negative short-term consequences for the themselves? This presents them with something of a dilemma.

I was recently reminded of this phenomenon when chatting with the lab director at the QA Center of Excellence for a very large, global consulting firm. (So as not to embarrass anyone, let us call him Fred.)

Fred told me that initially, when they heard our claims about ~5x faster implementation and ~10x faster maintenance, and all of the other benefits we touted, he thought we were full of it — just another vendor over-hyping their products.  But, he had been directed by his boss to check us out, so he had his team set up the lab and run our stuff through its paces.

Fred’s team came back to him and said that while there is variability in the efficiency gains to be had depending on what is being tested, the experience of the test engineer,and so forth, by and large our claims checked out.
Fred then advised his company’s local practice managers of this, sharing the results of the evaluations. He expected to get a lot of questions and excitement, but instead he received … silence.

This puzzled him, so he called around the practice managers.

Some were just too busy and hadn’t read or digested the message, others didn’t have any immediately pending projects and had filed his report for future reference, some were scared to use anything they had never used before.But a couple who had read it and had near-term needs showed little interest. When Fred asked why, he was told that improving efficiency by a significant margin would lead to fewer consultants for a shorter period of time for a test automation project, negatively impacting short-term sales of their consulting services.

So what was best for the customer was bad for the consulting firm.

Or was it ?

Fred and I discussed this, and we talked through some scenarios, as follows:

-     Imagine you are quoting around $250K for a project while your competitors submit bids of $500K or so. Who would the client select ? And when the project is done on time for the stated price, who is going to have pole position for that client’s next project? And who will the client recommend to their circle ? Long term, all other things being equal, your business increases as you blow away the competition on value.

-     In the above example, there may be a lower short-term revenue number, but what about margins ?  If your cost is now $125K instead of $400K, and you bill $250K instead of $500K, you win a definite $125K profit versus a potential $100K profit. Factor that across multiple projects and the fact that you win more projects than before due to lower prices, and your bottom line is greatly improved, and, your customers are also reaping the benefits. Eventually, your competitors would catch on and use the tools that give them the efficiency gains, then the usual margin squeeze would be in, but in the meantime you increase margins, win more projects, and reduce costs for your clients.

This is germane to a truism of business: Do the right thing for your customers and they’ll do the right thing for you. Not always, but usually.

As business people, we all have a choice. We can be plovers, benefiting our business associates and gaining some benefit ourselves. Or we can be ticks, trying to reap maximum gain for ourselves with little regard – and perhaps even negative consequences – for those we do business with.

To all you plovers out there – thank you, and I hope to work with you one day.

SmarteSuite Now Available for Legacy “Green Screen” Users

Friday, September 4th, 2009

On September 3, SmarteSoft and BlueZone Software announced a combined solution that gives enterprises that develop and maintain legacy “green screen” applications on IBM System z and System i access to a game-changing suite of test automation software. By including BlueZone in SmarteSoft’s SmartScript and SmartX applications, the firms are extending support of testing applications which help to break down the walls between software development, quality assurance, and business analysis teams for “green screen”-reliant projects.

For more details, please click here.

Breaking Down the Walls

Monday, August 24th, 2009

There was a time in the semiconductor industry where the test assets created during the process were not re-used.  This meant that test engineers were forced to start from scratch each time they developed and implemented a test plan. To make matters worse, there was a giant wall between design and test teams.

There are a number of reasons why this wall was there but, for the most part, it persisted thanks to the very different ways in which design engineers and test engineers then approached their job. As a result, the way in which such accompanying technologies as simulators and programmable test equipment worked echoed the split between design and test. These differences were of course driven by the specific needs and focus of each test environment, a situation which resulted in test data being structured completely differently for designers than it was for testers.

To this day, this difference in mind-set and the way test data is represented persists, but the SemiCon industry has realized the value of design-to-test re-use and has found ways to bridge this gap. The result?  An entire support industry that specializes in technologies designed to convert test data from the formats used by design engineers to a format that can be used by test engineers. This conversion is often an extremely complex task, but tools and methodologies have evolved to the point where re-use of design test data by test engineers is the norm throughout the industry, and has resulted in dramatic savings in test engineering costs and reductions in time-to-market for products that are often characterized by an extremely small market window.  While the wall is still very much intact for now, the results seen here still represented a dramatic, positive shift in the way technology firms view the separation between design and test. As such, it should serve as an example for the high tech sector.

At SmarteSoft, we’re taking the lessons learned in the semiconductor industry and applying them to the wall that divides software development teams from the QA folks who test their applications. With such revolutionary products as SmarteStudio (look for a full release in September) and the forthcoming SmarteScript 6.0, our clients will find themselves suddenly able to integrate development and QA, cutting both costs and time to market, thus improving their ROI. In ten years this may well be the norm. For now, you can only find such revolutionary implementation here.

What we can learn from Tom Watson

Friday, July 24th, 2009

Like many middle aged men in the US, I was enthralled by Tom Watson’s drive to win the British Open last week. At almost 60, a hero and icon of the golf world in the 70’s and 80’s was once more in pole position to win.

This win would have set some records too – including oldest major winner (by far) , and most British Open wins (tied).

On the final hole Tom hit a perfect 8-iron into the green, which landed in the perfect position on the front of the green and then ran straight for the hole. The shot was flawless, but unfortunately for Tom, and millions of fans, the ball kept rolling and rolling and ran off the back of the green.

Long story short, Tom lost.

What could he have done differently to avoid this fate ? Well, nothing that I could see. If a perfect shot is not good enough I don’t know what is.

Once I got past the agony of seeing such a worthy champion be defeated by bad luck at the last instant, this got me to thinking about the software world.

How many times have we seen everything done well leading up to software release, only to see some random event throw everything horribly off course ?

Like the time my development team crunched for months to get a release ready on time, and the QA team tested it every which way, and the Operations team prepared and tested the deployment infrastructure and did practice installations.

Everything was prepared as well as we all knew how. Then, one fine Sunday evening, we pulled together our best and brightest and deployed the software. The DBMS locked up, and nothing we did could fix it, except for a roll-back.

Weeks later that deployment was done successfully. In the interim, we discovered there was an obscure bug in Oracle’s software that caused it to fail under a very specific set of circumstances which we happened to hit during live deployment. This was not Oracle’s fault – no software is perfect. It was not our fault – we tested it successfully in the same configuration many times. It was just bad luck.

Unit testing, manual system testing, ad hoc testing, load testing, regression test automation, and pre-deployment infrastructure testing all failed to spark this problem.

Candidly, I don’t know what else we could have done. Like Watson’s approach on 18 last Sunday, we performed as close to perfectly as is possible and yet still ended up with a bogey on our hands.

So what did we learn ? How did this change our operating processes and procedures ? Well, it didn’t.

I did consider being even more cautious with the rest of the executive team with expectation setting. But if you go around telling your executives that no matter what you do everything may still go to hell, that’s generally not well received. The zone between doomsayer and pollyanna is pretty broad, and you don’t want to skate too far away from center when dealing with the executive team.

At the end of the day, stuff happens. We can improve our odds immeasurably with great preparation, frequent regression testing, etc., but there are no guarantees in life or in software development. (Something I try not to think about every time I get on a fly-by-wire airplane)

The best I can say of that episode, is that there was no finger-pointing and everyone went about their business professionally until we solved the problem.

I hope Tom feels the same way. I hope he was able to accept that luck was just against him on that hole, close the chapter without beating himself up, and move on.

The Agony and the Ecstasy: Deciding When Software is Ready to Release

Monday, July 13th, 2009

One of the most angst-ridden parts of the software development cycle is deciding when your product is ready to release. Functionality, performance, security and quality must be considered relative to specifications, and more often than not a judgment call must be made in the face of strong business pressure to release. Very often, the development cycle runs long and the Quality Assurance team doesn’t get nearly as much time as was scheduled to validate product quality. Still, QA is the final gatekeeper. If QA says ‘yes,’ the product releases amidst celebration. If they say no, there is a collective groan and the developers knuckle down to fix whatever must be fixed. Unnecessary delay hurts the business, just as too hasty a release (with attendant defects and customer complaints) also hurts the business. So the pressure on the QA Director is intense.

So how do you decide when a software product is ready?

Like all things it begins with planning at the start of the project. What are your test coverage goals?  What are your defect rate goals? How many defects of each severity are acceptable? What trend line must be established before you will feel confident releasing?

If these things (and more) are established at the start of the project, while not under intense pressure, and signoff on release metrics is gained from the key stakeholders at that time, then the release decision becomes a matter of keeping good metrics and using a little common sense for judgment calls. But if those metrics have not been established up front–or at least before decision time–then the whole process is fraught with subjectivity and emotion and the chances of someone making a bad decision are greatly increased.

Trend analysis for software quality is a whole art and science in itself, and some excellent books have been written on it. Being able to chart found versus fixed defects and their ratios, total defects of each severity and their trend, and keeping track of the impact of new builds on the stats, is a good starting point. When you’re fixing more than you’re finding, and each new build is introducing proportionately fewer defects than the previous one, while steadily reducing the overall defect rate, then you’re going in the right direction. This requires keeping whatever stats you have decided will be important from day one.

To add consistency to the defect measurement, and enable a rapid test cycle and assessment of quality, test automation is invaluable. By automating regression testing, you get to rapidly measure regressions–another key indicator of the quality trend–and can often spot a bad build almost immediately and scrap it before it does too much damage. Automation of regression testing also frees up your team to use their creativity to find new and better ways to try to break the application under test, and to test new functionality immediately rather than grinding through manual regression testing first.

A good test management tool (which enables everyone to see what is happening in the project based on their role, and includes visibility into the defect database), plus pre-planned metrics/objectives, plus trend analysis, plus test automation makes the release decision relatively stress free. Everyone can see the trend, and a common expectation becomes established that removes much of the emotion, and pressure, from the discussion. The decision on when to pull the trigger and release a product is seldom an easy one, but it can be made easier with careful planning, systematic measurement, and application of the right tools.

Werewolves and silver bullets

Thursday, July 2nd, 2009

Cards on the table, 100% test automation is impossible. Not only that, it shouldn’t even be a goal. Human intervention is required to provide context to the process.

It is way too easy to believe the promise you often hear… "You will now only have to push 1 button" …Because we want to believe it. This type of solution will occasionally work in very strict circumstances; however those same circumstances may also kill your test automation project. The blinders of strict circumstances will prevent you from seeing the forest for the trees.

"Record & Playback" sucks

Don’t buy the hype! Record and Playback as an automation strategy breaks down within a short timeframe. If it worked we could all just use macro tools and be on our merry way. Many a group who has followed this model has found out the hard way, you lose the benefit of automation, being able to re-run your tests once a code change has been implemented, you need data paramatization and workflow modification. These groups then have to start again, redeveloping the Record and Playback tests at a greater total expense than just testing manually. How many regressions do you run a year, how many incremental defect patches do you test, as the numbers approach zero you receive more benefit from record and playback, but nowhere else.

It is development

Test automation, at its core, is creating an application that is designed to exercise another application. The test itself is an application and as such requires the same steps, develop, test, re-factor, etc… Unnecessary complexity, lack of clarity, etc… All have the same negative effect for an automated test as they do for any application. So, make sure that you know how to write code efficiently, or use a tool that enforces standards, before initiating your automation project.

Automation requires time

Automation is frontloaded. Day one you will receive little, to negative, benefit. As additional regressions and test cycles are completed the cumulative effect starts to take place. Tests that were previously written create benefit that can be used to write additional tests, this repeats at the next cycle, until, eventually, you are realizing a net positive benefit to your automation effort.

Smart Staff

The implementation of any automated testing takes not only time, but skilled personnel. There is always a requirement for the project to be led by one who is experienced in test automation concepts and tool usage. The depth of your required bench is determined by the skill set required to use the tool. If you are planning on using a heavily scripted tool to automate your project, expect to have several higher end developers to implement. If you choose to go with a tool like SmarteScript by SmarteSoft, you can typically plan on the rank and file being experts in the usage of the application under test and head the team with an automation/testing expert.

Automated testing is rule driven

An automated test is an application; it will only do what you have instructed it to do. It will only compare those values you instruct it to compare, it will only use those expectations you set for it as measures of success. This is part of the reason that automated testing is no silver bullet, there are werewolves that abound. People can sense issues that are not explicitly directed. The best automation projects should account for manual as a sanity check. An automated test can pass with flying colors, and the application will still fail if the test has not been designed to make the appropriate validation.

Automation tools are not easy

Getting started may be easy, but sending staff out wily-nily to "automate" is a recipe for disaster. There is an old axiom "you don’t plan to fail, you fail to plan" and it holds very true. Having your lead, or senior testing engineer, the one with experience, draw up even a simple plan will reap benefits. Having a plan allows you to provide some insurance of success. Again, having an automation/testing plan directs the efforts to the appropriate areas.

Apply ROI to test selection

First task, what do we automate. How do we get started? Hopefully you are considering this as you initiate a project to apply test automation. The earlier in the development process that you can inject test automation as a requirement, the more chances of success. Early injection of automation allows you to start developing test plans and to start enforcing procedures that will lead to successful automation. Another benefit of early integration of testing is that the test team can start to gather workflows and required data before it is necessary, thereby speeding the effort.

Your selection effort should focus on those tests the meet criteria of frequency of use and impact on the business. If a case is used daily by customers, definitely attempt to automate it, if a case is imperative for function, there again, attempt to automate.

Conclusion

Automation is not a silver bullet, due to the "werewolves" that exist and require manual effort. With appropriate planning and mitigation of risk factors many of these can be alleviated. Plan for Success and understand the hurdles to come as you enter into the world of automated testing.

SQA Technical Seminar Topics

Thursday, June 25th, 2009

We are in the process of deploying technical seminars in various cities in the U.S.  We want to make sure we choose the right topics, as well as, where to deploy.

Currently we are thinking about the following topics:

  1. Sarbanes Oxley and Automation
  2. Best Practices in Automation
  3. Software Testing and the ADA (508 Compliance)

Please let us know in your comments which seminar you think is best and what city to have it in.  Also, what additional topics would you liked covered.

Welcome to the SmarteSoft Blog!

Thursday, June 11th, 2009

Hello!

Check here for early hints about new products, product updates, tips and tricks, and general reflection from the test gurus at SmarteSoft.