| Subject |
SmarteSoft SmarteScript
|
HP/Mercury Interactive
IBM/Rational
|
| Testing Methodology |
Automated Scripting Technology
Foundation
|
Point and Learn Technology
Fully automated process. No coding required.
|
Record and Playback
Manual Process of creating scripts and tedious manual coding required |
| Test Suite Optimization |
Highly optimized scripts that use one centralized data facility.
|
Not optimized, numerous scripts and data files. |
Number of Scripts and Tables
to Maintain
|
Eliminates complexity; minimum scripts and data tables with all test
cases.
|
Large numbers of scripts tied to numerous data files. |
| Uses Multiple Scenarios and Test
Scripts Modularity |
Mirror-image of the application business flow in Grid-driven test view.
Well-organized test cases, scenarios and script modularity. |
Poorly defined, hard-coded script structure with no minimal standard.
Every script is recorded & enhanced differently subjective to user's
coding skill level. |
| Consistency |
Highly consistent methodology; tool drives consistency throughout testing
cycles. |
Inconsistent, varies, subjective to scripters and programmers. |
| Time-To-Test Cycles |
Short and predictable throughout the test / software lifecycle
|
Lengthy script maintenance periods required, unpredictable, not enough
testing time. |
Codeless Keyword Testing and
Conditional Branching
|
No programming, short time to implement.
Keyword testing driven from spreadsheets.
|
Keyword functions require advanced programming.
Very long ramp up to implement. |
| Traceable Coverage |
Allows all objects to be tested with all data permutations, measurable
up to 95%. |
Subjective to plan and testers. Difficult to measure. |
| General Features |
Does Not Require Programming
|
|
|
| Scripting is Automated |
|
|
| Scheduling Tool |
With SmarteTime |
|
Regression Manager Tool
|
|
|
Bug Tracking Tool
|
Integration with TestDirector/Quality Center and SmarteQM |
|
Bug Reporting Via Email
|
|
|
| Data Generation Tool |
|
|
| Execution Progress Bar |
|
|
Automated Test Case Documentation
|
Fully automated process |
Except QTP |
| Dynamic Objects Learning and
Recognition Using Ordinal Methodology |
|
|
Compatibility with IE8/IE9/Win7/64
Bits Systems
|
|
|
Learn Context (right-click) Menus
and all right-click menus
|
Fully automated process |
|
Dynamic Test Branching and Dynamic
Data Generation
|
Built in. Tool can make multiple decisions of the functional/regression
test direction based on dynamically changing data, and dynamically alter
test data inputs according to multiple conditions.
|
Requires extensive ramp up time and advanced level programming to
customize this feature into a script. |
Test Flow Driver
|
Built in. Allows changing the flow of a test based on returned
value/variable, conditions and acts upon it when testing any object.
|
Requires advanced level programming to customize this feature
into a script |
Global Objects Change Manager
(GCM)
|
GCM allows global object management across hundreds/thousands of tests
using Distributed Object Model (DOM); includes Relearn, Add, Copy, Delete,
and Undo options. Detect and Update feature allows scan, validate, detect
and updating of objects in each window. |
Uses Object Repository. Central point of failure.
Size of repository affects the tool's performance.
|
Re-learn Objects Tool (GUI Maintenance)
|
Built in ease of maintenance: learns revised object, compares
and reports attributes against the original object in the script. Reports
data/text items changes in combo boxes, lists and tables. During script
execution, SmarteScript can automatically update the lists when they
dynamically change.
|
No objects re-learn, dynamic update or detect and update. |
Script Maintenance
|
Built in ease of maintenance: in case of script/data corruption,
a Detect and Repair features compares script against known script structure
to determine corruption and fix these problems.
|
|
Dynamically-generated Application
Data Storage
|
Built in with the ability to store dynamic data and dynamic variables
received from the application under test during test execution. The
dynamic data and variables are stored in the Grid for future usage and
data-correlation.
|
Requires script programming and building functions |
Error Detection, Error Validation
and Error Recovery
|
Built in with the ability to make decisions based on the type of errors
received from the application under test during test execution. |
Tedious manual process to implement. Requires script programming
and building functions.
|
Text Synchronization for Terminal
Emulation (TE)
|
Built in feature for various Mainframe Emulators.
|
Requires script programming and building functions. |
| Hot Key Mapping |
|
Only a few tools have it. |
| Exception Handling |
Built in. In addition, Window and objects of
exceptions may be learned to detect exceptions that appear on the screen
during test execution.
The script can be setup to decide how to handle the recovery as well
as flow.
|
Only a few tools have it. If not built in, this feature
requires programming.
|
| Simulated Objects |
Built in. In addition, allows learning objects in difficult environments
without the need for virtual objects. This makes it a more reliable
process.
|
Only a few tools have it. Other tools it is a manual process that requires
programming implementation.
|
Script Debugging Tools
|
All features plus: Watch variable list is automated
with script auto-compiler features.
|
Spy, manual watch variable list, breakpoints, step-run.
|
Open and Capture Clipboard for
Text Verification
|
Built in functions to handle and auto-check the text that is
captured.
|
Not available. Needs to be programmed into function library. |
| Script Generation |
Automated Script Generation
|
Instantly generated. Scripts are ready to execute as the user
learns the objects in the application.
No programming required. Data-driven functions and reporting built in.
Scripts can be migrated to Performance/Load testing with SmarteLoad.
|
Manual Record, playback and programming of scripts, building
functions, data files and reporting in addition to customization programming.
|
Built-in Function Libraries
|
Built in. Automatically generated and updated to support application. |
Need to be created from scratch and maintained for every application. |
Built-in Random and Sequential
Testing
|
Automated through lists, tables and combo-box data selections
in the grid.
|
Requires programming and custom functions. |
Data Flow and Permutation Patterns
are Visible (Before,
During and After the Tests)
|
Positive, negative and boundary data reside in one centralized visible
grid. Columns can be connected to external files. In addition, variable
data can be viewed during/after test. |
No. Data resides in numerous data files, not visible during testing.
|
Average Time to Create a Reliable
Data-Driven-Ready
Script
|
Minutes to generate working scripts. 30 minutes to
create complex Data-Driven-Ready scripts with complete data selection
and test cases.
|
2-4 hours. More time required depending on application
complexity and engineering expertise.
|
Highly Scalable and Flexible
|
High. Create/update business process, automatically scales the Grid,
script and reporting - no user intervention required. |
Low. Due to the numerous scripts that must be created or updated. |
| Readability Consistency |
Built-in automatically generated test documents, consistent, easy to
read and follow. Defines pass/fail criteria for each step of every test
case executed. |
Dependent on testers skills and experience in generating scripts and
documenting them. This is a tedious process, highly uncontrolled. |
| High Portability |
High. Scripts are self-contained (with their objects), generated to
run everywhere. Same script processes can be migrated to Load testing. |
Low. Scripts must be modified with path to GUI Maps or Object Repositories.
Load testing requires re-creation of scripts in a different tool. |
| Time to Script Reliability |
Reliable scripts are generated with known proven (pre-tested) structures,
optimized and compiled. |
Significant time spent to achieve reliable scripts in large quantity.
|
| High Productivity |
Highly productive scripting methodology, all test steps are streamlined
including detailed embedded reporting of each step of the test. |
Low. Need to record/playback, debug, re-run, revise, add reports for
every test step and test case as well as functions and customization.
|
| Script Generation |
Automated process generating fully parameterized and
optimized scripts. These scripts are also correlated and ready for Load/Performance
testing therefore, one process takes care of two different test objectives.
|
Manual process to record, playback, debug, enhance and maintain. |
| Checkpoints, Comparisons |
Windows/Objects GUI Checkpoints
are Automated?
|
Automatically generated into scripts. Performs checks without any programming.
Also, checkpoints before and after the actions on objects. |
Manually inserted into the scripts. Very tedious process per test case.
|
Bitmap Checkpoints are Automated?
|
Automatically generated into scripts. Performs checks without any programming.
Also, checkpoints before and after actions on objects/bitmaps. |
Manually inserted into the scripts when needed. |
Database Checkpoints with Dynamic
Checks?
|
Dynamic checkpoints for every test case and iteration available. |
Some tools have it but, no dynamic checks. |
| Text is Automated? |
Automatically generated into the script reading text from object, screen
areas, no programming required. |
Manually inserted and programmed into the scripts. |
| Actions on Windows and Objects |
Automatically generated into script. The tool performs
the correct actions according to classes and roles (behavior) of the
objects. In addition to the mirrored actions, SmarteScript attempts
to verify the behavior of the objects by their physical descriptions.
|
Recorded in the script. Only mirrors the action when
executed.
|
| Data Input |
Key Entries automated?
|
All data values are generated into the Grid.
No need to type. |
All values have to be typed into files. |
Test Data is Centralized?
|
Grid-driven - view with all data for all test cases of the business
process in one centralized area. |
Spreadsheets and text files dispersed into different directories.
Every test case' data may reside in a different location. |
| Import and Export Test Data |
Both Import and Export
|
Import, most tools
|
| Accuracy of Data Input |
Most accurate. Generates a Grid to enter the data and acquires data
from the objects. Learns all items from lists, combo-boxes and tables
(user- selectable in the Grid. All data items automatically saved in
the Grid for further selection.
|
Prone to typos and syntax when creating scripts. |
| Data Entry |
One Grid view for all data elements; edit fields require
data that can be automatically generated or imported from Excel, CSV
or text files. Other objects have pull-down menus to select the data.
|
Manual process; either recorded or uses many data
files/spreadsheets to connect.
|
Dynamic Application Data Entry
is Automated?
|
Automated data captures which is built into the Grid |
Manual process, requires to program function libraries. |
| Reporting |
Synchronization Reporting
|
Built in synchronization and performance measurement that reports the
exact timing down to milliseconds - automatically. |
None. Tester has to add it manually and build a reporting mechanism
for every step in every script and test case. |
| Automated Step Pass/Fail Reporting |
Automatically generated and reported pass and fail for each and every
step. Report is highly detailed identifying every piece of information
that was performed on objects or captured during execution; all object
attributes and data expected versus actual is automatically reported. |
Manual and extremely tedious process that must be added to each and
every test step in every test case. Detailed reporting must be built
into every step to capture all required information for the test i.e.,
data expected versus actual as well as object attributes. |
Automated Checking and Reporting
|
Complete reports automatically generated for every step when tests are
executed. Reports include Pass/Fail criteria, window/object operation
performed, iterations, duration, expected results, actual results as
well as an analysis. Reports iteration number, correct/incorrect indication.
Two rows of reports included for each object. User can filter what to
view in the reports. |
Tedious manual process to implement: Checkpoints have to be inserted
into the scripts. Because multiple files are created, it becomes unreliable
and cumbersome to maintain. No synchronization and iteration time reported
unless manually inserted into every step of every script. |
| Report Types |
Standard HTML report with Analysis, Summary report.
|
Varies, depending on tool used.
|
| Regression Summary Report |
|
|
| Report Filtering |
|
|
| Export Reports |
|
|
| Test Planning |
| Test Scripts Optimization |
Scripts generated by Business Processes to cover all modules/test
cases.
|
Create as many scripts as test cases. |
| Automatically Parameterized Test
Cases and Steps |
Test cases are automatically parameterized and visible in the Grid view
at all times - scripts are generated Data-Driven, ready for test
data inputs. Data is not hard-coded into any script.
|
All test cases recorded hard-codes in scripts, then manually
parameterized with data. Highly tedious and time consuming process. |
| Test Case Generation and Documentation |
Built in. Automates creation of test cases once it learns the
Business Process test flow, as well as test data to be used in every
test case. Document test cases, viewable in the Scenario view.
|
QTP only |
| Manageability |
Easily managed test suites using built-in Regression
Manager to execute test sets.
|
Users build a variety of different scripts and functions
which makes it harder to manage the testing environment.
|
Random and Sequential Testing
|
Built in. Automated through lists, tables and data selections as
well as an embedded data generation tool.
|
Not available. Extensive programming required. Advanced engineers
needed as well as time required for development.
|
Provides Data-Driven Tests
Efficiency and Productivity
|
Built in.
Automatically generated while learning the objects. Includes all objects
that use data. Combo boxes, tables and lists are learned with all data
items inside. Provides dynamic import
of data from files and databases.
|
Manual and lengthy process, requires programming
to construct a test using only edit fields and combo boxes.
|
| Overall Testing Results |
| Development Cycle |
Defined by specific modules based on Business Processes.
Easy to predict timelines since scripts are automatically generated.
|
Lengthy, difficult to organize and predict timelines
due to the enormous number of scripts created.
|
| Maintenance Cycles |
Short, optimized, easy to scale, debug, update and
predict times which increases productivity.
|
Lengthy, difficult to debug and update all scripts
as well as predict timeline.
|
| Predictability |
Object and window management make it easy to measure
and predict.
|
Difficult to measure and forecast reliability and
coverage.
|
| Level of Performance |
High for both functional and performance tests: scripts
are automatically Parameterized and Correlated.
|
Depends on implementation, generally low: everything
is a manual process.
|
| Test Execution |
Efficient. Uses pre-created function libraries compiled
into DLLs, debugged, tested and ready to run. Functions executed out
of memory produce reliable, consistent and predictable response times.
No hard coded data in scripts.
|
Slow in general. To accelerate execution process,
function libraries must be programmed manually by advanced QA engineers
to remove hard-coded data from the recorded scripts.
|
| Bottlenecks |
None; only a few scripts to scale and/or update for
application changes. This process is productive and performed automatically.
|
Numerous; many scripts to modify and debug every time
application changes - not enough time left for testing.
|
| Testing Reliability |
High, easy to predict and measure due to a consistent
methodology.
|
Unknown, subject to implementation, plan and testers.
|
| Liability |
Low with fewer defects. Maximum quality level possible
in the ASQA industry.
|
Untested areas may increase defects and liability.
|
| Effectiveness and Efficiency |
Highly predictable.
|
Difficult to measure.
|
| Subjectivity |
One unified methodology, high consistency, no subjectivity.
|
Highly subjective to QA engineers own methods.
|
| Investment |
Resource Needs
|
Minimum due to fewer scripts,
low maintenance and high productivity.
|
Maximum due to the creation of
many scripts, execution, maintenance and low productivity.
|
Resource Skill Level
|
Basic QA knowledge is sufficient
to operate. Business analysts can operate in hours. Minimal training,
operational within hours. Significantly short ramp up time to
create and run a full regression set - a collection of integrated scripts.
|
Require lengthy training, advanced
testers and programmers to create functions and customize. At least
3 month ramp up time required just to learn to script a single
script well.
|
| QA Test Resources |
Focused mostly on their main
goals: testing and analyzing of results.
|
Focused on the wrong tasks: they
debug and maintain scripts before they can actually test anything
so - no time remains to test and analyze the results.
|
| Budget/Costs |
Low costs per project and time-to-test cycles throughout
the software lifecycle.
|
High costs per test project and time-to-test cycles.
|
| Infrastructure for SQA |
Long-term solution with consistent builds of an asset
without the loss to SQA. Every tester uses the same methodology and
process - there are no hidden methods to script and perform the testing.
|
If testers leave the environment, testing has to be
created all over again - huge loss, no asset built - not a long term
solution.
|
| ROI |
Significantly High which
increases over time due to testers increase in productivity.
|
Low due to loss of skilled trained engineers, ramp
up time and rapid, time-consuming tedious maintenance.
|
| Integration |
Integration with Test Management
Tool
|
Integration with SmarteQM for test execution and reporting.
|
|
| API External Function Calls |
Wizard-driven to insert API functions into the Grid.
|
Need to manually write the API functions into script.
|
| Text Verifications |
Wizard-driven to insert functions into the Grid.
|
Manually inserted into every script. Some tools cannot
perform text verification.
|
| Environment Variables Setup |
Automatically loaded for
the whole environment so there is no additional configuration each time
tests are being executed.
|
Manually inserted into every script.
|
Application Default Data Appearing
in the AUT
|
Default data automatically
appears in the grid data selection when objects are learned. User can
then add as many data values to the grid.
|
None available. Requires manual scripting to check
the default data.
|
| Technical Support |
Technical Support On-site Available
|
|
|
Technical Support Over the Phone
|
|
|
| Technical Support Over the Web |
|
|
* The marked features are available only in a separate tool which the client
would have to purchase in addition to the testing tool.
** Only QuickTest Pro has a Re-learn feature however, it doesn't display the
difference in object attributes between the older and the new objects. User
has to manually perform the analysis. In addition, no dynamic learn of combo
boxes items during script execution.
All entries in the comparison table have been made on the basis of information
available on respective product websites. The analysis and views expressed
in this section and the information made available are purely those of SmarteSoft,
Inc.. It is possible that competing products have additional features not
mentioned on the product websites.
©2008-2011 SmarteSoft, Inc. SmarteSoft and SmarteQM are trademarks of
SmarteSoft, Inc. All other company, brand, or product names are marks of their
respective holders.
