Thursday, September 26, 2013

Load testing your cloud app - Tools comparison

If you’re planning to load test your cloud app, the comparison below might be handy.
The research I have done is on these service providers:
  1.          Soasta CloudTest,
  2.          Blitz,
  3.          BlazeMeter,
  4.          LoadStorm,
  5.          and MicrosoftCloud-based Load test tool.


Soasta – CloudTest

Pros

  •          A complete environment to setup tests
  •          Add test users locations seamlessly
  •          a good coverage around the globe
  •          A comprehensive real-time interface to metrics
  •          Drill down analysis on web requests
  •          A Lite (Free) version is available with a limit of 100 concurrent users

Cons

  •          Requires some setup: you have to spin up a VM to load the test environment

Pricing

“CloudTest is sold based on server hours, including support. With no limits on the number of testers or software access, you select
Plans based on how many test server hours you’ll run. Plans are available for coverage of a few tests each month up to thousands of tests each year” - http://www.soasta.com/wp/wp-content/uploads/2012/10/DS-CloudTest-100812.pdf

Blitz

Pros

  •          Capacity planning tools, Optimization, Performance monitoring are out-of-the-box
  •          Nothing to install/setup, all cloud-based
  •          Simulate up to 5,000 concurrent virtual users
  •          It has a chrome/firefox plugin to monitor app performance on develop environment
  •          Integrated with Google Analytics
  •          Performance Monitoring Alerts  (Email, Twitter, SMS, PagerDuty)

Cons

  •          Only 8 test locations around the globe (Australia, Brazil, California Ireland, Japan, Oregon, Singapore, Virginia)
  •          Relatively less sophisticated interface (compared to Soasta)

Pricing

  •          Works with “Credits”
  •          1 credit = 1 minute and 1,000 users - https://www.blitz.io/pricing
  •          40 Credits = $40
  •          150 Credits = $135
  •          300 Credits = $240

BlazeMeter

Pros

  •          JMeter compliant
  •          Easy to use test creation interface
  •          Nice interface for test results
  •          You can export test results
  •          Compare two test runs
  •          Has a server in Australia (yes, we’re load-testing apps in Oz)
  •          Test scheduling
  •          Integrated with Google Analytics
  •          VPN is supported
  •          Chrome extension to record JMeter scripts and upload it to BlazeMeter
  •          It has a free version that can simulate up to 50 concurrent users
  •          A good customer base

Cons

  •          Limited Geo coverage (California, Oregon and Virginia USA, Tokyo, Ireland, Brazil, Singapore and Australia)

Pricing

  •          Monthly plans – Starting $199 (1,000 users, 20 hours) up to “Call Us”
  •          On-Demand (Per hour) – Starting $19 (1,000 users, 1 server) to $299 (40,000 users, 40 servers)
  •          http://blazemeter.com/pricing



LoadStorm

Pros

  •          Scalable up to 300,000 users
  •          A testing data centre in Sydney, Australia
  •          Supports automatic crawling (Spider)
  •          No scripting language is involved
  •          Nothing to install

Cons

  •          Currently they only cover Virginia, California, Oregon, Ireland, Sydney, Tokyo, Singapore, Sao Paulo
  •          No built-in feature to collect server performance metrics

Pricing

  •          Monthly plans – Starting Free (100 users, 25 users per test) up to “Call Us”
  •          On-Demand (Per hour) – Starting $39 (1,000 users per test) to “Call” (10,000+ users)
  •          http://loadstorm.com/load-testing-cost/



Microsoft Cloud-based Load test tool

Pros

  •          MS offers a free version of TFS services (up to 5 developers)
  •          Reuse on-premise tests
  •          Easy to set-up

Cons

  •          Requires Visual Studio™ Ultimate (Pricy)

Pricing

  •          It’s still vague - varies according to your MSDN subscription
  •          If you don’t have an MSDN subscription, you’ll have to buy VS Ultimate (around AU$19,274) and then subscribe for TFS Services



8 comments:

  1. Thanks for the awesome post.
    At TechTrendsIT we believe that Cloud Application Development Service will drive future technology strategy and help business grow effectively. We suggest and give a strong recommendation to our clients to consider a “cloud first policy” while designing and deploying web application and mobile applications.

    ReplyDelete
  2. Nice information about the load testing!!! I prefer Loadrunner automation testing tool to validate the performance of software application/system under actual load. Best Loadrunner training institute in Chennai

    ReplyDelete
  3. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic..
    Selenium Training in Chennai | QTP Training in Chennai

    ReplyDelete
  4. Thanks for Information Oracle Apps Technical is a collection of a bunch of collected applications like accounts payables, purchasing, inventory, accounts receivables, human resources, order management, general ledger and fixed assets, etc which have its own functionality for serving the business
    Oracle Apps Training In Chennai

    ReplyDelete
  5. Oracle Training in chennai | Oracle D2K Training In chennai
    This information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic..

    ReplyDelete