1. Production Tomcat Architecture
2. Tuning tomcat for performance
3. Resolving problems which affect availability

 Production Tomcat Architecture

In production Tomcat relies on a number of resources which can impact its overall performance. Understanding the overall system architecture is key to tuning performance and troubleshooting problems.

1. Hardware: CPU(s), memory, network IO and file IO
2. OS: SMP (symmetric multiprocessing) and thread support
3. JVM: version, tuning memory usage, and tuning GC
4. Tomcat: version (example, Tomcat 6 supports NIO)
5. Application: Application design can have the largest impact on overall performance
6. Database: concurrent db connection is allowed (pooling and object caching)
7. Web Server: Apache can sit in front of Tomcat and serves the static content. It also can do load balancing across multiple Tomcat instances.
8. Network: Network delays.
9. Remote Client: How fast is the communication protocol? Content can be compressed. 

Performance Tuning

How to measure and test performance

• Request latency is key b/c it reflects the responsiveness of your site for visitors.
• Test environment should match production as closely as possible.
• The data volume is important to simulate in database side.
• Test HTTP requests with different request parameters (test corner cases)
• Use load test to simulate the traffics (ex. JMeter)
• Final tests should be over longer periods like days because JVM performance changes over time and can actually improve if using HotSpot. Memory leaks, db temporary unavailable, etc can only be found when running longer tests.

JVM version, memory usage and GC

• Sun Java 1.3 and later releases inlcude HotSpot profiling optimizer customized for long running server application.
• Tomcat will freeze processing of all requests while the JVM is performing GC. On a poorly tuned JVM this can last 10′s of seconds. Most GC’s should take < 1 second and never exceed 10 seconds
• Tune the -Xms (min) and -Xmx (max) java stack memory (set them to the same value can improve GC performance)
• Make sure the java process always keeps the memory it uses resident in physical memory and not swapped out to virtual memory.
• Use -Xincgc to enable incremental garbage collection
• Try reducing -Xss thread stack memory usage

Tomcat version and configuration

• Tomcat 6 supports NIO.
• Set “reloadable” false – remove unnecessary detection overhead
• Set “liveDeploy” to false – liveDeploy controls whether your webapps directory is periodically checked for new war files. This is done using background thread.
• Set “debug” to 0
• Set “swallowOutput” to true – This makes sure all output to stdout or stderr for a web application gets directed to the web application log rather than the console or catalina.out. This make it easier to troubleshoot problems.
• Connector configuration – minProcessor, maxProcessor, acceptCount, enableLookups. Don’t set the acceptCount too high b/c this sets the number of pending requests awaiting processing. It is better deny few requests than overload Tomcat and cause problems for all requests. Set “enableLookups” to false b/c DNS lookups can add significant delays.

Database connection pool

• We use connection pool provided by Spring instead
• Using middleware to persist and cache objects from your database can significantly improve performance b/c of fewer db calls, less thrashing of the JVM for creation and subsequent GC of object craeted for resultset.

Application design and profiling

• If the data used to generate a dynamic page rarely changes, modify it to a static page which you regenerate periodically.
• Cache dynamic page
• Use tool like JProble to profle your web applications during development phase
• Look for possible thread synchronization bottlenecks
• Date and Time thread synchronization bottleneck 

Troubleshooting

Collecting and analyzing log data

Common problems

• Broken pipe – For HTTP Connector indicates that the remote client aborted the request. For web server JK Connector indicates that the web server process or thread was terminated. These are normal and rarely due to a problem with Tomcat. However, if you have long request, the connectionTimeout may close the connection before you send your response back.
• Tomcat freezes or pauses with no request being processed – usually due to a long pause of JVM GC. A long pause can cause a cascading effect and high load once Tomcat starts handling requests again. Don’t set the “acceptCount” too high and use java -verbose:gc startup argument to collect GC data.
• Out of Memory Exception – look into application code to fix the leak (profile tool can help). Increase available memory on the system via -Xmx. Restart tomcat!
• Database connection failure – connection used up when traffic is spike.
• Random connection close exception - when you close your connection twice. First close(), the connection returns to the pool. It may be picked up by another thread. Now, second close() may close a connection that is being used by other thread. Don’t close connection twice, use JDBC Template from Spring to avoid this problem. 

Reference

1. JavaWorld GC Article
2. Sun HotSpot Performance Document
3. Tomcat Performance Slides

Isolation and encapsulation are the keys. Isolating concerns opens the door to accommodating change, because it puts boundaries on the ripple effects of changes within the code. Isolation makes modular development and testing easy. Isolation makes it easy to have different people work on different pieces. Finally, isolation makes it easier to envision ways of improving the user experience that are generalized and powerful.

The first rule I learnt in component design is “Reduce coupling and increase cohesion”. The goal of this is to reduce the amount of interactions among components and in turn promote reusability and reduce complexity. This concept proven to be useful in my career life. The tricky part of it is to draw a line around a component and provide a clear responsibilities (ie. via interface) and interactions of it (ie. communication). A good architect can make it very clean and simple whereas a novice designer cannot make a component reusable without associated with different dependencies. In this article, the author focuses on the design in UI, but the rules apply to all aspects of design. A term “MVCS” is mentioned that adds “Services” to the common known MVC pattern. Service is responsible to encapsulate the communication with the external world. Here is the diagram of the MVCS:

In MVCS, Model captures the state of the application. It doesn’t know any of the non-Model classes and it carries no intelligence (ie. no function but read/write for properties). According to the author:

In general, avoid functions, since they introduce more complexity, and stray interactions with non-Model objects can creep in. Complex logic for model objects usually belongs in the Controller.

However, from what I have learnt, there are certainly logics belongs to the model classes. A domain model without any behavior is termed as “Anemic Domain Model” by Martin Fowler. “…this is an anti-pattern because it’s so contrary to the basic idea of object-oriented design; which is to combine data and process together.” When I look closer, this is the model that supports the UI but not the domain model in the server side. Is the same rule applied in this area as well? On the other hand, how the model notifies the View if its state is changed? In Flex, you can define your properties as “Bindable”. Whenever the state of the model is change, an event will be fired to notify the associated View.

View references objects in Model via data binding. Bindings are used to move data from sources that change into destinations that need to be change-aware. Besides,View captures users interaction (via event handler registration) and hands off action to Controller (via calling Controller’s function in event handler). Controller, like Session, is application-wide object that should be obtained through a static instance variable to avoid having to pass all such objects all over the application.

Controller is the center of the application. It is the place where application logic and behavior are implemented. It is the only part of the architecture that communicates with all the other parts, thus relieving the View, Model, and Service packages from needing to know too much about each other.

Service objects make up a layer of objects that handle all communication with the outside world like Web Service and Http calls. Besides, Services act as factories for Operation objects; it is actually these Operations that do the work of communication by initiating requests, handling responses, and dispatching status and completion events. Why not just do it internally rather than generate the Operation for controller to call? According to the author, it gives more flexibility. However, I don’t see such much usage of this flexibility. So, I still prefer doing simple Service’s method call…