|
Independent Evaluations of Networking Products and Tools |
NETWORK TESTING LABS REVIEW:Transaction Monitors – An UpstartVs. the EstablishedArgent Defender, Mercury (HP) SiteScope and CA Wily Introscope
Executive Summary
Argent Defender, a relative newcomer in the field of Web-based transaction performance monitoring and management, easily won top honors in our competition. Defender gave us more information – and gave us the right information – to make tuning Web-based transaction environments quick and effective. Defender was easier to use, requiring no changes to the application. It was more robust and reliable. And it gave us a fine-grained look at exactly what our application was doing.
Discussion
An upstart has emerged in the complex and critically important realm of Web-based transaction performance monitoring. While Argent Software has in the past focused on monitoring and managing whole networks of routers, switches, servers, WAN links and specific applications (such as Exchange, Notes, SQL Server and Oracle), the vendor has recently turned its attention to helping customers quickly detect and resolve transaction-level performance problems in run-your-business-on-it, Web-based application systems. The ideal transaction monitor for Web-based applications can accurately discover and graphically depict the components and paths in a network that are associated with transaction creation, processing and response. It works well with virtually any type of application, from ASP-based to J2EE-based. It can detect connection problems and slowdowns in real time. It can show network administrators and developers exactly where the bottlenecks are. The perfect transaction monitor sends alerts via e-mail, pager and other means, and it can escalate these alerts as necessary until the problem is fixed. It produces useful reports, it has a responsive, easy-to-navigate user interface and the ideal transaction monitor is scalable, robust and reliable. Moreover, the perfect tool offers a user-editable scripting environment. To weigh the value and merit of two of the more established, market-leading transaction monitoring tools against the upstart, we decided to test Mercury Interactive SiteScope, CA Wily Introscope and Argent Defender at both our Alabama lab and at various customer sites. Argent Defender earned flying colors in the competition. Defender more accurately identified to us where and why the application was slowing down. Defender gave us a finer level of precise detail regarding application performance. It alerted us on a more timely basis, and Defender escalated the alerts as necessary to make sure every problem received the attention it deserved. Finally, Defender was easier to use, more responsive, more robust and more scalable. Argent Defender wins our Network Testing Labs World Class award for Best Transaction Monitor.
Argent DefenderArgent Defender quickly alerted us to the bottlenecks and outages we created in the lab. With breathtaking clarity and precision, Defender revealed exactly the network components involved in the performance problems we subjected it to. In one test, Defender notified us that a user clicking a particular pushbutton on the third screen of our application caused a spike in our database server’s CPU utilization – we verified that, yes, indeed, Defender had found a deliberately-coded application bug. In another test, in which we stressed an application by bombarding it with a dense stream of transactions, Defender correctly and easily discovered that our application server machine was underpowered. Remarkably, yet just as we’d expected, Defender works equally as well in a production environment as it does in a test environment. Argent Defender rather smartly leverages the DHTML Interface, the Active Accessibility Interface and raw mouse and keyboard events to instrument (our term for Argent’s approach, not the vendor’s) Web-based transactions. Defender uses agent modules at client PCs to make sure a transaction is properly instrumented. A central Defender engine stores the transaction performance data in an ODBC-compliant database, such as SQL Server, Oracle or, for very small tests, Access. Defender analyzes the data and displays the resulting statistics on a targeted basis. Defender provides network administrators, troubleshooters and capacity planners with fine-grained detail that shows exactly those computing environment components that participate in the processing of a transaction and the precise extent of that participation. Our tests showed that Argent Defender itself uses fewer resources (e.g., CPU, memory and disk space) to accomplish its work than either Wily Introscope or SiteScope. Our tests also showed that Argent Defender uses network bandwidth more frugally. Furthermore, we found Defender to be much more robust and reliable. Impressively, Defender works equally well with application systems based on ASP, Java, .NET, SOAP (Web Services), PHP, AJAX and even Citrix. If scalability is important to you, be aware that Argent Defender has an n-tier architecture that allows it to monitor transactions on even the largest network. Telling Defender which transactions to monitor is as simple as turning on its recording facility and then issuing one or more transactions. The recording facility is a simple but powerful Browser Helper Object, or BHO. The automatically-generated playback script can then be used to monitor those transactions on an ad hoc or, if you wish, scheduled basis. To our delight, we found we could even edit the playback scripts to our heart’s content. In its SQL Server or Oracle database, Defender stores event detail, not just statistics. This means that investigating a problem that happened yesterday or even several days ago is a breeze. Defender’s archive becomes a treasure trove of performance data. You’ll never again have to tell users, “Let us know if the problem happens again and we’ll look at it then.” When Defender detects a problem, based on alerts you’ve set up (note that the ones out of the box work just fine), it can take any one of a number of actions, including sending e-mail messages, paging people and even issuing SQL commands. Argent Defender did a better job than either SiteScope or Introscope of escalating an alert, via e-mail addresses and pager numbers, when the front line people did not immediately respond to the alert. One of Defender’s more significant advantages is its ability to launch a script when it detects a problem. These scripts run in a high-level VBScript environment and, although they can be as complex and powerful as you wish, the scripts are child’s play to construct. These Root Cause Analysis scripts can be either Windows (WMI)-based scripts or UNIX-based scripts. Argent supplies several extremely useful scripts with Defender. These can reveal the exact nature of the culprit behind a particular slowdown or outage. Defender’s intelligent analysis of problems saved us many hours of head-scratching as we went through the process of figuring out how to solve each specific performance problem. For example, Defender easily and clearly showed us when poor response from a Web site was caused not by the primary Web site but by its references to a second, separate Web site. The primary Web site was perfectly healthy. Defender informed us that the second Web site was memory-starved. Defender also correctly pointed out slow network (bandwidth) problems, slow Web server problems, broken connection problems and crashed/hung Web server software problems. Defender scripts not only correctly detected when the IIS service failed, but, remarkably, the scripts also parsed the IIS log, listed the processes running on the Web server and, at our behest, restarted the IIS service in order to restore the Web server to full availability. Argent Defender uses an Argent SuperMap to graphically depict the network. The underlying maps can contain whatever images you wish and that best express the locations of network components, the business functions of those components, or some other representation that makes sense to you. When Argent Defender alerts you to a problem, drilling down through the various Argent SuperMaps to see exactly what’s happening takes just a few mouse clicks. Argent Defender reveals a wealth of data regarding the trouble, and it highlights those statistics associated with the outage or slowdown. Argent supplies its Defender customers with a run-time version of Crystal Reports. With Crystal Reports, designing new custom reports is a breeze. You won’t spend all your time designing reports, however. We found Argent Defender’s out-of-the-box reports perfectly suitable for tracking problem resolutions, following historical trends and planning for future network capacity. Argent Defender’s documentation is comprehensive, easy to follow and accurate. Defender can be installed in less than 30 minutes.
Mercury Interactive (H-P) SiteScopeJust a couple of years ago, SiteScope was a general-purpose network monitoring tool for detecting server, router and WAN link connectivity problems. From these humble beginnings, Mercury Interactive (now a subsidiary of H-P) grew SiteScope into an application-oriented transaction performance monitoring product. In our tests, SiteScope failed to deliver Defender’s level of detail regarding each of the application bottlenecks we put in its path. Furthermore, SiteScope was more cumbersome to use, its user interface was less responsive and its learning curve was steeper. SiteScope, which is written in Java, is an object-oriented tool consisting of modules that display SiteScope’s browser-based interface, schedule other modules’ executions, gather performance statistics, send alerts, run scripts and produce reports. H-P uses the term Business Process Management (BPM) to denote SiteScope’s transaction monitoring. Using what was formerly known as the Topaz Watchdog, SiteScope’s monitoring feature even monitors itself. Once every day, or if SiteScope crashes, the watchdog restarts SiteScope. SiteScope’s monitoring components stand guard over systems, applications, Web servers and network devices. The systems monitors collect basic server metrics, such as CPU utilization, memory usage, DNS/DHCP, file sizes vs. free disk space and TCP/IP services (FTP, Telnet, etc.). The applications monitors watch over specific software products, such as WebLogic, CiscoWorks, Citrix MetaFrame, IBM DB2, WebSphere, ColdFusion, Exchange, Internet Information Server, SQL Server, Oracle and SAP. The Web server monitors keep an eye on Web servers (Apache, Netscape Enterprise, Netscape FastTrac, Microsoft IIS or O'Reilly WebSite), Web site links, e-business servers and SOAP-based Web Service sites. The network service monitors detect, for example, device and port availability via ping operations. We noted SiteScope’s ping monitor sometimes missed detecting devices that were indeed up and running. Although easy to get used to and navigate, SiteScope’s user interface is somewhat sluggish. We were also put off by SiteScope’s automatic daily restart – a monitoring tool should be robust enough to run more than 24 hours without causing problems. It used e-mail and pager messages to alert us, and its reports summarized overall network activity levels and error events. SiteScope has clear and comprehensive online documentation. It installed in less than an hour.
CA Wily IntroscopeWily Introscope is primarily a tool for J2EE-based software, although it has also gained some support for .NET-based applications. Wily Introscope works by insinuating its agents into the run-time environment. These agents intercept events, measure the resulting event data and report the event statistics back to a central console. We were disappointed that Wily Introscope gave us little or no help with our PHP, AJAX, ASP, non-J2EE Java and Citrix testbed applications. Not every application has a perfect J2EE, EJB-based design and implementation. in a .NET system, Wily Introscope basically collects PERFMON data. It augments this data with interpolated data that Wily Introscope’s agents glean from the passing stream of events. To its credit, the .NET version of Introscope does have the ability to automatically discover .NET software that’s running on the network. We found Wily Introscope to be fairly robust and reliable, although its user interface wasn’t as responsive as we would’ve liked. Wily Introscope’s CPU/memory/disk/bandwidth footprint was larger than Defender’s but smaller than SiteScope’s. Moreover, Wily Introscope’s alerting feature lacked many of the options and capabilities of Argent Defender. Wily Introscope uses a proprietary SmartStor database as a repository for its transaction performance data. Its reports are easily customized, with the out-of-the-box report formats seemingly geared mostly toward Service Level Agreement (SLA) monitoring. Wily Introscope’s documentation, which is online, lacks an in-depth tutorial guidebook on how to use the tool. It installs in less than an hour.
ConclusionArgent Defender supplies network administrators, troubleshooters and developers with exactly the right information needed to solve performance problems and stay ahead of emerging challenges. Defender is robust, responsive, easy to use and scalable. We recommend you evaluate it for use in your own computing environment. You’ll be glad you did.
Test bed and methodologyWe evaluated each transaction monitoring product's ability to discover, manage, administer, monitor, alert, report on, troubleshoot and automatically fix any and every kind of Web-based transaction system. Virtually all our testing took place across WAN links. Our lab's computing environment included Windows NT, 98, XP, 2000, 2003 and Vista, along with Solaris 8.0, Macintosh OS X and Red Hat Linux. Relational databases on the network were Oracle 8i, IBM DB2 Universal Database, Sybase Adaptive Server 12.5 and Microsoft SQL Server 2005. Web servers were Microsoft IIS, Netscape Enterprise and Apache. We ran each product's server components on a Windows 2003 Advanced Server-based Compaq Proliant ML570 with four 900 Mhz CPUs, 2 GB RAM and 135 GB of SCSI disk. We used an XP-based Dell Latitude D505 notebook computer for client access. In the performance testing, we measured transactions per second and, for sheer throughput, mb per second of data flow. We varied the types and densities of transaction streams in a comprehensive battery of tests. We also deliberately created bottlenecks, both in hardware and software. We expected the transaction monitoring tools to accurately detect and report on the transaction rates, transaction data flow and bottlenecks.
Transaction Monitoring Report CardGrade scale is A through F, with F = Failing and A = Perfect
Vendor Details
|