How to test my system?

Here is a good video from Google 

Testing Strategies Overview

I believe that I don’t need to stress out how important testing is. So, lets cut the crap and talk about the challenging areas related to this subject. Here are the things I want to talk about

1. Write your unit test first then code
   • It is the core concept from Test Driven Development (TDD). Everyone tried that will tell how great it is. However, the difficult part of it is to enforce it across your team especially under the time constraint. Everyone wants to get something out right away and we are all getting so much stresses from our business side about features release to stay competitive in the market. How can we allocate time for this “luxury”? The key here is whether you believe it will save you tons of time in a long run. If not, it is hard to carry out this idea.
   • What are the areas that could save my time? Agile development may face one big challenge from the product side. The problem is mainly due to the way they break down the project into mini-specs to fit into a sprint of work. Sometimes they find it difficult to think of a project in a micro level. To deliver the mini-spec in a agile process, they may leave out the detail in the spec and try to fill them up in an iterative fashion. Write your test first forces you to think API, input, output and use cases ahead of time. This will help you to feedback the product team earlier in the loop. Otherwise, it will cause you more time to change your logic or patch your codes later.
   • If you have a suite of unit tests, you are more confidence to do refactoring because you can perform regression tests to make sure nothing is broken. Apart from that, you can add new features to someone’s code without worrying that it will break the old features as you have the unit tests from the owner to verify.
   • Things you need to pay attention are: (1) Don’t commit code to the trunk that breaks the build (compile), but it is ok that it doesn’t pass the tests temporary as you are in the middle to fill up the code and you don’t want to have your code sitting in your local box too long. (2) Do code coverage to make sure you have your unit test exercises your codes in good %.
   • Tools I used to write my unit test: TestNG/ JUnit for Java code, FlexUnit for flex code
   • How do we unit test our DAO? Test with your local database (use same vendor and version of the database as the one in production. I don’t like to use any memory databases like Hypersonic or SQLite b/c it doesn’t really test our your SQL).
2. Functional testing – use Selenium to automate it.

Performance/load testing

Basic workflow

1. Create use case scripts for your application via recording. For Flex, we want a tool that can convert the binary AMF message to readable text as part of our script.
2. Introduce natural variability - varying the login credentials, records opened, records saved, and other user actions including thinking time. This step is important because it provides the real-world variability that keeps your application working hard and prevents you from missing performance problems because of caching at various layers on your server.
3. Define test data sets to be used by your scripts as you ramp up hundreds or thousands of simulated users executing your tests.
4. Define profiles of load on your application. You might decide to have 20% of your users entering information slowly, 20% entering information very quickly, 40% looking up information, 10% producing reports, and 10% making account changes. The breakdowns and level of detail are entirely up to you.
5. Specify how to run the test. You control such things as the speed of ramping up user load, how fast each user executes the test, and whether there should be natural variability between users and sessions. Each tool provides different fine-grained control over this step.

Tips

• Don’t carry performance test without passing all the functional tests.
• Test against single box first and you should not run JMeter on the same machine running the application to be tested.
• Make sure that the testing is affected as little as possible by network traffic (do it in the subnet to take out network IO factor).
• Create a test plan that contains thread group(s) that controls the threads that will be created by JMeter to simulate simultaneous users. Determine # of threads and the ramp-up period. Add HttpRequest then listener to view result in table and/or graph format. Each thread will issue an HttpRequest and response time will be recorded. Then avg response time and standard deviation will be calculated. Another way people uses JMeter is to use its HTTP session recording to produce load tests.
• On the other hand, you need to monitor your target in terms of CPU, memory and IO consumption (use OpenNMS) during the perfomance testing.
• Create a scheduled integration build that runs JMeter tests (use CruiseControl). Read this article for integration detail.
• How to achieve the load via distribution?
• How to obtain meaningful information from the raw data?
• Look at throughput (QPS), resources consumption (CPU, Memory, IO) over time and over users/ load.
• Need to set up the database with good amount of data for testing.
• Performance profiling
  • Obtain the rough data from performance test first
  • Identify the bottleneck and fix the problem.
  • Don’t waste too much time to over-tune your system. Just fix the bottlenecks that impact the SLA.
• Stress testing
• Scalability testing – testing on scaling out or up your system.
• Availability testing

Reference

Below are the references I read to write this article

1. http://www.opensourcetesting.org
2. JMeter User Manual