SDTimes‘ David Rubistein has a nice overview of our limited release of SmarteStudio in yesterday’s online edition of that publication. Click here to read it.
Archive for July, 2009
SmarteSoft in SDTimes!
Thursday, July 16th, 2009The Agony and the Ecstasy: Deciding When Software is Ready to Release
Monday, July 13th, 2009One of the most angst-ridden parts of the software development cycle is deciding when your product is ready to release. Functionality, performance, security and quality must be considered relative to specifications, and more often than not a judgment call must be made in the face of strong business pressure to release. Very often, the development cycle runs long and the Quality Assurance team doesn’t get nearly as much time as was scheduled to validate product quality. Still, QA is the final gatekeeper. If QA says ‘yes,’ the product releases amidst celebration. If they say no, there is a collective groan and the developers knuckle down to fix whatever must be fixed. Unnecessary delay hurts the business, just as too hasty a release (with attendant defects and customer complaints) also hurts the business. So the pressure on the QA Director is intense.
So how do you decide when a software product is ready?
Like all things it begins with planning at the start of the project. What are your test coverage goals? What are your defect rate goals? How many defects of each severity are acceptable? What trend line must be established before you will feel confident releasing?
If these things (and more) are established at the start of the project, while not under intense pressure, and signoff on release metrics is gained from the key stakeholders at that time, then the release decision becomes a matter of keeping good metrics and using a little common sense for judgment calls. But if those metrics have not been established up front–or at least before decision time–then the whole process is fraught with subjectivity and emotion and the chances of someone making a bad decision are greatly increased.
Trend analysis for software quality is a whole art and science in itself, and some excellent books have been written on it. Being able to chart found versus fixed defects and their ratios, total defects of each severity and their trend, and keeping track of the impact of new builds on the stats, is a good starting point. When you’re fixing more than you’re finding, and each new build is introducing proportionately fewer defects than the previous one, while steadily reducing the overall defect rate, then you’re going in the right direction. This requires keeping whatever stats you have decided will be important from day one.
To add consistency to the defect measurement, and enable a rapid test cycle and assessment of quality, test automation is invaluable. By automating regression testing, you get to rapidly measure regressions–another key indicator of the quality trend–and can often spot a bad build almost immediately and scrap it before it does too much damage. Automation of regression testing also frees up your team to use their creativity to find new and better ways to try to break the application under test, and to test new functionality immediately rather than grinding through manual regression testing first.
A good test management tool (which enables everyone to see what is happening in the project based on their role, and includes visibility into the defect database), plus pre-planned metrics/objectives, plus trend analysis, plus test automation makes the release decision relatively stress free. Everyone can see the trend, and a common expectation becomes established that removes much of the emotion, and pressure, from the discussion. The decision on when to pull the trigger and release a product is seldom an easy one, but it can be made easier with careful planning, systematic measurement, and application of the right tools.
SmarteSoft CEO Co-Authors Article for TEST Magazine
Friday, July 10th, 2009The most recent edition of TEST magazine features a piece on the history and application of test automation tools, co-authored by SmarteSoft’s Gordon MacGregor. In it, MacGregor and Steven Christer of Strategic Systems Solutions (a SmarteSoft partner) discuss how they’ve seen the testing industry develop, and where they feel readers should look for future trends.
To read the entire article please click here and turn to page 10.
SmarteStudio is Coming!
Tuesday, July 7th, 2009On July 15th, SmarteSoft will be offering a limited release of its new SmarteStudio application. With it, users will be able to take advantage of our innovations in record and editing technology, an enhanced debugger, a JScript interpreter, and the powerful automated reporting that they have come to expect from SmarteSoft. Key features include:
- Learn and Go technology: SmarteStudio can record your interactions with an application and then translate these into a script, which you can execute as is or modify.
- Script Editor: The script editor provides syntax highlighting, syntax checking, and code folding.
- Interpreter: SmarteStudio uses Microsoft’s JScript interpreter, so you have access to all of Microsoft’s script writing utilities.
- Debugger: The SmarteStudio debugger includes Control Execution, Persistent Breakpoints, a dialog for viewing the call stack and variable values, and a watch view, which allows you to view the value of any arbitrary Javascript statement as the program executes.
- Automated reporting: SmarteStudio generates a report automatically every time you execute a test. The report details the steps of the test, input, and results.
- Test API: SmarteStudio includes an API specifically designed for creating tests. Use it to manipulate images, spreadsheets, and customize the report.
Keep your browsers tuned to this space for more details!
Werewolves and silver bullets
Thursday, July 2nd, 2009Cards on the table, 100% test automation is impossible. Not only that, it shouldn’t even be a goal. Human intervention is required to provide context to the process.
It is way too easy to believe the promise you often hear… "You will now only have to push 1 button" …Because we want to believe it. This type of solution will occasionally work in very strict circumstances; however those same circumstances may also kill your test automation project. The blinders of strict circumstances will prevent you from seeing the forest for the trees.
"Record & Playback" sucks
Don’t buy the hype! Record and Playback as an automation strategy breaks down within a short timeframe. If it worked we could all just use macro tools and be on our merry way. Many a group who has followed this model has found out the hard way, you lose the benefit of automation, being able to re-run your tests once a code change has been implemented, you need data paramatization and workflow modification. These groups then have to start again, redeveloping the Record and Playback tests at a greater total expense than just testing manually. How many regressions do you run a year, how many incremental defect patches do you test, as the numbers approach zero you receive more benefit from record and playback, but nowhere else.
It is development
Test automation, at its core, is creating an application that is designed to exercise another application. The test itself is an application and as such requires the same steps, develop, test, re-factor, etc… Unnecessary complexity, lack of clarity, etc… All have the same negative effect for an automated test as they do for any application. So, make sure that you know how to write code efficiently, or use a tool that enforces standards, before initiating your automation project.
Automation requires time
Automation is frontloaded. Day one you will receive little, to negative, benefit. As additional regressions and test cycles are completed the cumulative effect starts to take place. Tests that were previously written create benefit that can be used to write additional tests, this repeats at the next cycle, until, eventually, you are realizing a net positive benefit to your automation effort.
Smart Staff
The implementation of any automated testing takes not only time, but skilled personnel. There is always a requirement for the project to be led by one who is experienced in test automation concepts and tool usage. The depth of your required bench is determined by the skill set required to use the tool. If you are planning on using a heavily scripted tool to automate your project, expect to have several higher end developers to implement. If you choose to go with a tool like SmarteScript by SmarteSoft, you can typically plan on the rank and file being experts in the usage of the application under test and head the team with an automation/testing expert.
Automated testing is rule driven
An automated test is an application; it will only do what you have instructed it to do. It will only compare those values you instruct it to compare, it will only use those expectations you set for it as measures of success. This is part of the reason that automated testing is no silver bullet, there are werewolves that abound. People can sense issues that are not explicitly directed. The best automation projects should account for manual as a sanity check. An automated test can pass with flying colors, and the application will still fail if the test has not been designed to make the appropriate validation.
Automation tools are not easy
Getting started may be easy, but sending staff out wily-nily to "automate" is a recipe for disaster. There is an old axiom "you don’t plan to fail, you fail to plan" and it holds very true. Having your lead, or senior testing engineer, the one with experience, draw up even a simple plan will reap benefits. Having a plan allows you to provide some insurance of success. Again, having an automation/testing plan directs the efforts to the appropriate areas.
Apply ROI to test selection
First task, what do we automate. How do we get started? Hopefully you are considering this as you initiate a project to apply test automation. The earlier in the development process that you can inject test automation as a requirement, the more chances of success. Early injection of automation allows you to start developing test plans and to start enforcing procedures that will lead to successful automation. Another benefit of early integration of testing is that the test team can start to gather workflows and required data before it is necessary, thereby speeding the effort.
Your selection effort should focus on those tests the meet criteria of frequency of use and impact on the business. If a case is used daily by customers, definitely attempt to automate it, if a case is imperative for function, there again, attempt to automate.
Conclusion
Automation is not a silver bullet, due to the "werewolves" that exist and require manual effort. With appropriate planning and mitigation of risk factors many of these can be alleviated. Plan for Success and understand the hurdles to come as you enter into the world of automated testing.
The SmarteScript Virtual TE Keyboard
Wednesday, July 1st, 2009Do you ever encounter a field–like username or password–that your developers have added security to? One way we have seen this done is to disallow the pasting of information from the clipboard. To test for this, type your password (or username) into a text editor. Select the password text, then try to paste it into your login page. Now select another field. Does you app accept the pasted data? If not there may be a security method preventing this method of data input.
When this is the case it is necessary to enter data “manually”. By this I mean using keystrokes. For anything involving more than a couple of keystrokes this process is tedious, and it also means that you can use only one input word. This is unacceptable and there is a good way of working around the limitation created by security.
In SmarteScript you can do this by creating commands to strike individual keys. Thankfully, the application includes several commands/functions in a file called the init file. Within this file is a command called virtual_TE_keyboard. This function emulates typing the individual keys as opposed to pasting in the value.
To use virtual_TE_keyboard:
Create a variable for your input by right clicking on a window
Select add input column
Name the input column appropriately
Create a click command
Right click on the window name in the tree
Select add command
Select click object command
Fill out parameters for the object in question to left click on it
Create virtual_TE_keyboard command
Right click the window name
Select add command
Select init file commands
Select virtual_TE_keyboard
Select the window name
Input the variable for the input
This will allow you to create commands to input data via the keyboard where necessary. The command will type the letters of the string it is passed and allow you to move forward with your test. The only caveat is that your data should not include special characters.
What we can learn from Tom Watson
Friday, July 24th, 2009Like many middle aged men in the US, I was enthralled by Tom Watson’s drive to win the British Open last week. At almost 60, a hero and icon of the golf world in the 70’s and 80’s was once more in pole position to win.
This win would have set some records too – including oldest major winner (by far) , and most British Open wins (tied).
On the final hole Tom hit a perfect 8-iron into the green, which landed in the perfect position on the front of the green and then ran straight for the hole. The shot was flawless, but unfortunately for Tom, and millions of fans, the ball kept rolling and rolling and ran off the back of the green.
Long story short, Tom lost.
What could he have done differently to avoid this fate ? Well, nothing that I could see. If a perfect shot is not good enough I don’t know what is.
Once I got past the agony of seeing such a worthy champion be defeated by bad luck at the last instant, this got me to thinking about the software world.
How many times have we seen everything done well leading up to software release, only to see some random event throw everything horribly off course ?
Like the time my development team crunched for months to get a release ready on time, and the QA team tested it every which way, and the Operations team prepared and tested the deployment infrastructure and did practice installations.
Everything was prepared as well as we all knew how. Then, one fine Sunday evening, we pulled together our best and brightest and deployed the software. The DBMS locked up, and nothing we did could fix it, except for a roll-back.
Weeks later that deployment was done successfully. In the interim, we discovered there was an obscure bug in Oracle’s software that caused it to fail under a very specific set of circumstances which we happened to hit during live deployment. This was not Oracle’s fault – no software is perfect. It was not our fault – we tested it successfully in the same configuration many times. It was just bad luck.
Unit testing, manual system testing, ad hoc testing, load testing, regression test automation, and pre-deployment infrastructure testing all failed to spark this problem.
Candidly, I don’t know what else we could have done. Like Watson’s approach on 18 last Sunday, we performed as close to perfectly as is possible and yet still ended up with a bogey on our hands.
So what did we learn ? How did this change our operating processes and procedures ? Well, it didn’t.
I did consider being even more cautious with the rest of the executive team with expectation setting. But if you go around telling your executives that no matter what you do everything may still go to hell, that’s generally not well received. The zone between doomsayer and pollyanna is pretty broad, and you don’t want to skate too far away from center when dealing with the executive team.
At the end of the day, stuff happens. We can improve our odds immeasurably with great preparation, frequent regression testing, etc., but there are no guarantees in life or in software development. (Something I try not to think about every time I get on a fly-by-wire airplane)
The best I can say of that episode, is that there was no finger-pointing and everyone went about their business professionally until we solved the problem.
I hope Tom feels the same way. I hope he was able to accept that luck was just against him on that hole, close the chapter without beating himself up, and move on.
Tags: British Open, software deployment, software test, test automation, Tom Watson
Posted in General Comment | 1 Comment »