The key goals of load testing are:

1. Find out whether your website can support the expected # of concurrent users.
2. At what load does the app break?

To do that, you normally follow these steps:

1. Identifying the primary user path
2. Identifying the expected # of concurrent users. (Both now and future)
3. Set up virtual users to hit the app (load generation capability is the key factor to pick the right tool. You don't want too much hardware investment to generate the load you want, right? )
4. Run the test
5. Analyze the result (throughput under load, avg response time under load)

Challenges to load test Flex app

We have a web application that has Flex frontend talks to J2EE web application backend via AMF. How do we load and stress test this system? We certainly can just perform load test against our backend. However, it may need to expose our service via Servlet and load test it as typical restful web service. If we want to simulate Flex and load test through AMF. We need to find a way to capture the AMF requests from Flex client and replay it in our load testing suite. To do that, we can use Charles http proxy to capture the AMF request and tell JMeter to load test our application via replaying this AMF request. This  article can give us the detail. However, Charles is a commerical product. If you want a free solution, you can try this.

I know that JMeter comes with proxy to record the request. You can try it out to see whether you can exam the AMF message. Let me know if it works!

JMeter

Load is all of the users using your web site at a point in time. Load includes users making requests to your web site as well as those reading pages from previous requests. However, we do need some way of differentiating between all clients and those clients actually making requests to our web site. We use the terms concurrent load and active load (making request) to make this distinction. We use the JMeter to help us to generate the load to our system. In term of load generation, we should make sure that we can simulate the peak load. JMeter is a great load testing tool. I have heard that Google is using this to load and stress test its application.

To use JMeter, follow the steps below:

1. Set up Thread Group - use to model concurrent virtual users and decide how you want the load be generated.
   • Number of threads.
   • The ramp-up period (it tells JMeter the amount of time for creating the total number of threads). At the beginning of a load test, if the ramp-up period is zero, JMeter will create all the threads at once and send out requests immediately, thus potentially saturating the server and, more importantly, deceivingly increasing the load. That is, the server could become overloaded, not because the average hit rate is high, but because you send all the threads' first requests simultaneously, causing an unusual initial peak hit rate. The rule of thumb for determining a reasonable ramp-up period is to keep the initial hit rate close to the average hit rate.
   • The number of times to execute the test.
   • If the client machine running JMeter lacks enough computing power to model a heavy load, JMeter's distributive testing feature allows you to control multiple remote JMeter engines from a single JMeter console.
2. Introduce user think time
3. Specify response-time requirements and validate test results.

To get familiar with it, here are some articles I found it useful

1. Stress testing with JMeter by Daniel Rubio - Linux.com
2. Loading testing with Apache JMeter by Kulvir Singh Bhogal, - Devx.com
3. JMeter distributed testing
4. JMeter recording testing
5. Load test your drupal app scalability with JMeter - Part 1, Part 2
6. Load testing with JMeter - powerpoint - very good!
7. Scalability Factors of JMeter in Performance Testing - response size, response time, protocol, hardware configuration, load generating tool architecture and configuration, complexity of client-side processing.
8. JMeter Tips - Javaworld

 Reference

1. Someone has written a nice article in sys-con to talk about how to load test Flex application.
2. Use FlexUnit for stress testing
3. Test Remote Data Service via FlexUnit
4. Load testing with log replay - interesting idea

Testing Strategies Overview

I believe that I don't need to stress out how important testing is. So, lets cut the crap and talk about the challenging areas related to this subject. Here are the things I want to talk about

1. Write your unit test first then code
   
   • It is the core concept from Test Driven Development (TDD). Everyone tried that will tell how great it is. However, the difficult part of it is to enforce it across your team especially under the time constraint. Everyone wants to get something out right away and we are all getting so much stresses from our business side about features release to stay competitive in the market. How can we allocate time for this "luxury"? The key here is whether you believe it will save you tons of time in a long run. If not, it is hard to carry out this idea.
   • What are the areas that could save my time? Agile development may face one big challenge from the product side. The problem is mainly due to the way they break down the project into mini-specs to fit into a sprint of work. Sometimes they find it difficult to think of a project in a micro level. To deliver the mini-spec in a agile process, they may leave out the detail in the spec and try to fill them up in an iterative fashion. Write your test first forces you to think API, input, output and use cases ahead of time. This will help you to feedback the product team earlier in the loop. Otherwise, it will cause you more time to change your logic or patch your codes later.
   • If you have a suite of unit tests, you are more confidence to do refactoring because you can perform regression tests to make sure nothing is broken. Apart from that, you can add new features to someone's code without worrying that it will break the old features as you have the unit tests from the owner to verify.
   • Things you need to pay attention are: (1) Don't commit code to the trunk that breaks the build (compile), but it is ok that it doesn't pass the tests temporary as you are in the middle to fill up the code and you don't want to have your code sitting in your local box too long. (2) Do code coverage to make sure you have your unit test exercises your codes in good %.
   • Tools I used to write my unit test: TestNG/ JUnit for Java code, FlexUnit for flex code
   • How do we unit test our DAO? Test with your local database (use same vendor and version of the database as the one in production. I don't like to use any memory databases like Hypersonic or SQLite b/c it doesn't really test our your SQL).
2. Functional testing - use Selenium to automate it.

Performance/load testing

Basic workflow

1. Create use case scripts for your application via recording. For Flex, we want a tool that can convert the binary AMF message to readable text as part of our script.
2. Introduce natural variability - varying the login credentials, records opened, records saved, and other user actions including thinking time. This step is important because it provides the real-world variability that keeps your application working hard and prevents you from missing performance problems because of caching at various layers on your server.
3. Define test data sets to be used by your scripts as you ramp up hundreds or thousands of simulated users executing your tests.
4. Define profiles of load on your application. You might decide to have 20% of your users entering information slowly, 20% entering information very quickly, 40% looking up information, 10% producing reports, and 10% making account changes. The breakdowns and level of detail are entirely up to you.
5. Specify how to run the test. You control such things as the speed of ramping up user load, how fast each user executes the test, and whether there should be natural variability between users and sessions. Each tool provides different fine-grained control over this step.

Tips

• Don't carry performance test without passing all the functional tests.
• Test against single box first and you should not run JMeter on the same machine running the application to be tested.
• Make sure that the testing is affected as little as possible by network traffic (do it in the subnet to take out network IO factor).
• Create a test plan that contains thread group(s) that controls the threads that will be created by JMeter to simulate simultaneous users. Determine # of threads and the ramp-up period. Add HttpRequest then listener to view result in table and/or graph format. Each thread will issue an HttpRequest and response time will be recorded. Then avg response time and standard deviation will be calculated. Another way people uses JMeter is to use its HTTP session recording to produce load tests.
• On the other hand, you need to monitor your target in terms of CPU, memory and IO consumption (use OpenNMS) during the perfomance testing.
• Create a scheduled integration build that runs JMeter tests (use CruiseControl). Read this article for integration detail.
• How to achieve the load via distribution?
• How to obtain meaningful information from the raw data?
• Look at throughput (QPS), resources consumption (CPU, Memory, IO) over time and over users/ load.
• Need to set up the database with good amount of data for testing.
• Performance profiling
  • Obtain the rough data from performance test first
  • Identify the bottleneck and fix the problem.
  • Don't waste too much time to over-tune your system. Just fix the bottlenecks that impact the SLA.
• Stress testing
• Scalability testing - testing on scaling out or up your system.
• Availability testing

Reference

Below are the references I read to write this article

1. http://www.opensourcetesting.org
2. JMeter User Manual