| Subject |
SmarteLoad
|
LoadRunner
|
| Performance Methodology |
| Technology Foundation |
- Record and Playback - well-organized user-friendly
grid script with server requests and responses in easy to read format
- Automated script conversion from existing SmarteScript
scripts
- Automated conversion from other vendors functional
tools
- No coding required
|
- Record and Playback - Tree and script views
- No conversion or migration
- Coding required for most tools
|
| Test Suite Optimization |
Conversion and migration use consistent flow which
was implemented in the functional test but enhanced for performance testing
usage. |
Users have to re-record scripts in a Load testing
tool even if they have existing functional scripts - "Re-inventing
the wheel…" |
| General Features |
Execution Scheduler tool (per
Group - ramp up, Ramp down, duration)
|
|
|
| Synopsis/Scenario Execution |
|
|
| Run-Time Settings |
|
|
Automated Transaction Measurements
|
|
|
Enter Transaction Names During
recording
|
|
|
Record Init, Actions & End
Components
|
|
|
| Pacing and Iteration Configurations |
|
|
| Monitor Users Progress |
|
|
Monitor Synopsis Status &
Execution Progresses
|
|
|
| Real-Time Analysis Graphs |
|
|
Real-Time Analysis Summary Display
Statistics upon Graph Selection
|
|
|
| Real-Time Statistics |
|
|
| Host Generator Limitation |
Use as many Host Generator Agents as needed |
Several vendors charge more for additional Load Generator agent installations |
Does Not Required Script Coding
|
|
Script enhancements require coding |
Script Conversion From Functional
/ Regression tool
|
|
|
| Data Generation Tool |
|
Except LoadRunner |
Automated Remote Agent Installations
|
Remote Host Agent installation is performed from remote through the
Host Management facility |
User must manually install or schedule system admin to install; this
causes delays in test environment readiness |
Dynamic Data Correlation Methods
|
Manual and fully automated process with undo option for all
or specific correlation
|
Manual process; automation is limited (doesn't find many correlations)
which requires programming |
Complex Dynamic Correlation (boundary
change)
|
Fully automated process with advanced Regular Expression pattern matching
approach |
Requires complex programming |
Complex Sequential Dynamic Correlations
with Numerous Embedded Correlations
|
Fully automated process with advanced Regular Expression pattern matching
approach |
Requires complex programming |
| Controller Automated Transaction |
Automated transactions in the Controller; no need to go back to the
script
|
Manual Inserts of Start and End Transaction commands into the script |
Implementing SyncPoints (multi-user
intersect points for Stress Testing)
|
Automated SyncPoints within the Controller; no need to go back to the
script
|
Manually insert rendezvous commands into the script, going back from
controller to script |
Multi-Protocol Oracle RCS Protocol
+ HTTP/S
|
Includes latest Oracle RCS protocol (New Educational Protocol)
|
|
Multi-Protocol WinSock + HTTP/S
|
Included |
Requires complex programming |
| .NET Protocol Recording |
Included |
Require additional costs for extension and Virtual Users for .net protocol |
Java Protocol Libraries &
Customization (With Java)
|
Easily integrates with Java libraries for additional Java custom protocols
using Java (included) |
Requires complex programming |
| Analysis & Statistics |
Multiple Real-Time Graphs View
|
|
|
Maximum Number of Graphs in Real-Time
|
8
|
8
Some have fewer RT graphs
|
| Monitors |
Included in the tool's price - no hidden costs |
Additional costs per monitor |
| Analysis View |
|
|
| Analysis Summary Report |
|
|
Analysis Graphs Add and Merge
|
Automatic merge and scale |
Some have but not all features |
| Reporting |
| Reports View |
|
|
Export Report to various Formats
|
Also export to PowerPoint |
Some have but not all formats |
| Transaction Summary Report |
|
|
| Report Filtering |
|
|
| Export Reports |
|
|
| Script Generation |
Ease-of-Use, Readability &
Consistency
|
Grid Visualization system allows user-friendly interface
with easy to read scripts and flow. Grid script displays server requests
and responses in a clear page by page format. This allows consistent
script generation for all users.
|
Relies on engineers recording and updating the scripts.
The process is inconsistent and subjective to the users.
Each engineer may record and program in his/her own format.
|
Script Customization
(Uses JavaScript)
|
Does not require programming, uses pre and post conditions JavaScript
commands for functional and data validation as well as script enhancements
|
Requires complex programming |
| Data Input Methods |
Random & Sequential and Unique
Data Testing
|
|
|
| Import & Export Test Data |
Both Import and Export
|
Import, most tools
|
| Investment |
| Virtual Users limitation |
All virtual user pricing is the same for all
protocols
No hidden or additional costs for additional protocols
|
The cost for virtual users depends on the type
of protocol required for testing
|
| All Inclusive Price |
Out of the box price includes all protocols, monitors
and extensions
No hidden or additional costs for additional components
|
Requires additional costs for Protocols, Monitors
and Extensions
|
| Resources Skill Level |
Basic QA knowledge is sufficient to operate. Testers,
Developers and even Business Analysts can operate within hours. Minimal
training, significantly short ramp up period to create and run
a Synopsis, a suite of multi-user integrated performance/load scripts.
|
Requires lengthy training, ramp up period, advanced
testers and programmers to create functions and customize. At least
3 months ramp up time just to learn how to script and use the
tool properly.
|
| Budget/Costs |
Overall low costs per virtual user,
project and time-to-test cycles. Example: SmarteLoad list price is a
fraction of the LoadRunner price.
|
Overall high costs per virtual user,
test project and time-to-test cycles
|
Performance/Load Test
QA Infrastructure
|
Long-term solution with consistent performance testing
methodology. Every tester/developer uses the same methodology and process
|
If testers leave the environment, performance testing
has to be recreated (subjective to testers). Not a long term solution
|
| ROI |
|
|
| Overall Testing Results |
Test Development Cycle
Effectiveness & Efficiency
|
Highly predictable script development throughout the
performance test cycles. Scripts can be automatically generated from
another tool.
|
Difficult to measure and predict script development
and performance test cycles
|
| Subjectivity |
One unified methodology, high consistency, no subjectivity
|
Highly subjective to QA engineer's own scripting methods
|
| Technical Support |
Technical Support On-Site Available
|
|
|
Technical Support Over the Phone
|
|
|
| Technical Support Over the Web |
|
|
