Traditional web application is based on request and response model that information is delivered as a single payload and then immediately close the connection to the client. To keep the client in sync, we normally pull the server periodically. This approach may generate unacceptable load to the server. To solve this problem, we want to have a push mechanism from server to client. This is why Comet is defined. Comet is a generic term describing various approaches to send data asynchronously from a Web server to a client without the need for the client to explicitly request the data. It is an essential technique for any real-time event-driven web applications, where the majority of events occur on the server and data must be "pushed" frequently to the client. To achieve this, Comet servers must maintain a continuous connection to each client for the duration of the session.

OK. How to maintain a continuous connection to each client for the duration of the session?

If you try to adapt traditional server to the Comet methodology, it may not scale and often fails after a few thousand simultaneously open connections. A true Comet implementation requires a very different kind of server architecture to be efficient and scalable - Liberator (a solid Comet server that are used by the financial industries. However, it is written in C and not open source although it has FREE edition distributed).

To understand this statement a little bit more, we need to know how traditional web containers handle the request. They are under one request per thread model.

1. The client , typically , a browser sends request for resource to a web server.
2. The server has a listening thread that keeps track of incoming connections.
3. When a request arrives , the server uses one process or thread to process the request.
4. The resource is returned to the client and the connection is closed.

In this model, the number of requests that can be served in a second would depend on two things

1. How many threads are there to handle the client requests
2. How long it takes to serve one request.

If all threads of server are busy, then the incoming requests are put in a queue. The server would return to the requests in queue when server threads become free. The number of requests handled per second is always greater than the number of allowed simultaneous connections. All this is made possible because the time required to process a request is very short. In other words you can server more requests in a second than you have threads.

However, there are one breed of applications that need to hold onto the connections. Think of applications that require real time data coming to clients (stock tickers)  or think of applications where low-latency is required. In the above traditional web model, the browser has to re-connect to get the new data. (Polling). If the new updates "can"  happen with high frequency (e.g. a chat application) then the polling frequency also has to increase .  An alternative to high frequency polling is to use push based applications. For push based application, once the browser connects to server, the server will maintain the connection till the browser time-out (server response stream is not closed) and keeps flushing data down the connection as and when they become available. In servlet container, to hold the connection, your thread in the service method cannot exit the method. Otherwise, the response stream will be closed. So what you do is, you block the thread on some condition within the service method. So the thread will block for your condition. When push data becomes available , this thread writes to response stream and again enters a blocked state. So as long as you hold onto the connection, you can not return this thread to the thread pool. And as more and more "push" connections are established you would run out of threads! To remedy the problem, the possible solutions are:

1. Increase # of server threads.

Flex Push

There is confusion that whether BlazeDS supports real time messaging. Yes it does . In fact, BlazeDS has a full spectrum of channel types ranging from simple polling, to near-real-time polling, to real-time streaming.

1. Simple polling - ping the server from Flex client using the traditional request and response model
2. Near-real-time polling (long polling) - Instead of acknowledging right away, the server could hold the polling request until there’s a message for the client. This ensure the messages are delivered to the client as soon as they become available. The caveat for using long-polling is the thread limitation in most application servers. At this moment, BlazeDS could not support more than a few hundred long-polling clients on most application servers. However, this problem could be resolved once servers like Tomcat start to support asynchronos, non-blocking connection threads. Update: Now Tomcat 6 supports NIO.
3. Real-time streaming - BlazeDS supports real-time message streaming over AMF and HTTP. Unlike long polling, which closes and reopens the connection upon receiving a message, streaming keep the connection open at all times. Streaming suffers from the same thread blocking issue as long polling. A cap must be set so the server is not hang by idle threads.

The reason why people are confused is that Adobe doesn't release its proprietary push solution RTMP to BlazeDS. So, RTMP isn't available as a channel in the BlazeDS configuration files. BlazeDS lives in a Servlet container and hence constrained by one-thread-per-connection limit whereas LCDS has NIO-based channels that can scale up to 1000s of requests. On the other hand, BlazeDS has the advantage that it'll work over port 80/443, whereas LCDS will use some port for persistent connections that would require a firewall configuration. Once the servlet that implements BlazeDS is revved to support Comet Events under Tomcat 6, and then Jetty Continuations, then the long polling technique will be fine.

UPDATE: We are waiting for a solution that supports Comet Events under Tomcat 6. Then BlazeDS can be coupled to the Tomcat NIO HTTP listener and be able to scale as well as any NIO based server software.

I have learnt from this article that you can create a channel set in client side. So Flex can fail-over to other channels until it gets connected or the list is exhausted.

Marc has put an effort to build a better data grid like a spreadsheet in Flex. (check this out)

Reference

Here are the references I used for this article

1. Tuning Apache and Tomcat for Web 2.0 comet application
2. Performance of Grids for Streaming Data - This shows you the performance numbers on various frontend technologies. Again, Flex shows us a good result.
3. Are raining comets and threads? - Comet Daily
4. Comet & Java: Threaded Vs Nonblocking IO
5. JDK 1.6 uses epoll to implement NIO
6. BlazeDS dev guide
7. Achieve performance breakthrough using BlazeDS - Farata System put an effort to write its NIO channel that runs on Jetty 7 and receive promising result.

1. Production Tomcat Architecture
2. Tuning tomcat for performance
3. Resolving problems which affect availability

 Production Tomcat Architecture

In production Tomcat relies on a number of resources which can impact its overall performance. Understanding the overall system architecture is key to tuning performance and troubleshooting problems.

1. Hardware: CPU(s), memory, network IO and file IO
2. OS: SMP (symmetric multiprocessing) and thread support
3. JVM: version, tuning memory usage, and tuning GC
4. Tomcat: version (example, Tomcat 6 supports NIO)
5. Application: Application design can have the largest impact on overall performance
6. Database: concurrent db connection is allowed (pooling and object caching)
7. Web Server: Apache can sit in front of Tomcat and serves the static content. It also can do load balancing across multiple Tomcat instances.
8. Network: Network delays.
9. Remote Client: How fast is the communication protocol? Content can be compressed. 

Performance Tuning

How to measure and test performance

• Request latency is key b/c it reflects the responsiveness of your site for visitors.
• Test environment should match production as closely as possible.
• The data volume is important to simulate in database side.
• Test HTTP requests with different request parameters (test corner cases)
• Use load test to simulate the traffics (ex. JMeter)
• Final tests should be over longer periods like days because JVM performance changes over time and can actually improve if using HotSpot. Memory leaks, db temporary unavailable, etc can only be found when running longer tests.

JVM version, memory usage and GC

• Sun Java 1.3 and later releases inlcude HotSpot profiling optimizer customized for long running server application.
• Tomcat will freeze processing of all requests while the JVM is performing GC. On a poorly tuned JVM this can last 10's of seconds. Most GC's should take < 1 second and never exceed 10 seconds
• Tune the -Xms (min) and -Xmx (max) java stack memory (set them to the same value can improve GC performance)
• Make sure the java process always keeps the memory it uses resident in physical memory and not swapped out to virtual memory.
• Use -Xincgc to enable incremental garbage collection
• Try reducing -Xss thread stack memory usage

Tomcat version and configuration

• Tomcat 6 supports NIO.
• Set "reloadable" false - remove unnecessary detection overhead
• Set "liveDeploy" to false - liveDeploy controls whether your webapps directory is periodically checked for new war files. This is done using background thread.
• Set "debug" to 0
• Set "swallowOutput" to true - This makes sure all output to stdout or stderr for a web application gets directed to the web application log rather than the console or catalina.out. This make it easier to troubleshoot problems.
• Connector configuration - minProcessor, maxProcessor, acceptCount, enableLookups. Don't set the acceptCount too high b/c this sets the number of pending requests awaiting processing. It is better deny few requests than overload Tomcat and cause problems for all requests. Set "enableLookups" to false b/c DNS lookups can add significant delays.

Database connection pool

• We use connection pool provided by Spring instead
• Using middleware to persist and cache objects from your database can significantly improve performance b/c of fewer db calls, less thrashing of the JVM for creation and subsequent GC of object craeted for resultset.

Application design and profiling

• If the data used to generate a dynamic page rarely changes, modify it to a static page which you regenerate periodically.
• Cache dynamic page
• Use tool like JProble to profle your web applications during development phase
• Look for possible thread synchronization bottlenecks
• Date and Time thread synchronization bottleneck 

Troubleshooting

Collecting and analyzing log data

Common problems

• Broken pipe - For HTTP Connector indicates that the remote client aborted the request. For web server JK Connector indicates that the web server process or thread was terminated. These are normal and rarely due to a problem with Tomcat. However, if you have long request, the connectionTimeout may close the connection before you send your response back.
• Tomcat freezes or pauses with no request being processed - usually due to a long pause of JVM GC. A long pause can cause a cascading effect and high load once Tomcat starts handling requests again. Don't set the "acceptCount" too high and use java -verbose:gc startup argument to collect GC data.
• Out of Memory Exception - look into application code to fix the leak (profile tool can help). Increase available memory on the system via -Xmx. Restart tomcat!
• Database connection failure - connection used up when traffic is spike.
• Random connection close exception - when you close your connection twice. First close(), the connection returns to the pool. It may be picked up by another thread. Now, second close() may close a connection that is being used by other thread. Don't close connection twice, use JDBC Template from Spring to avoid this problem. 

Reference

1. JavaWorld GC Article
2. Sun HotSpot Performance Document
3. Tomcat Performance Slides