Class and ID Selector

/* The difference between an ID and a class is that an ID can be used to identify
one element, whereas a class can be used to identify more than one. */

/* class selector */
.intro

/* id selector*/
#top

/* built-in html element (no need to put symbol) */
p

/* locate specific element - paragraph with class=jam
   (for example <p class=jam>xxx</p>) */
p.jam {xxxx} 

/* apply grouping */
h2, .thisClass, .thatClass {...}

/* nested selector */
#top{...}
#top h1{...}

/* state of an element (ex. link) */
a.snowman:link
a.snowman:visited
a.snowman:active
a.snowman:hover

/* shorthand : margin, padding, border-width where th
p {
    border-top-width: 1px;
    border-right-width: 5px;
    border-bottom-width: 10px;
    border-left-width: 20px;
}

//can be
p {
    border-width: 1px 5px 10px 20px;
}

//border-width, border-color and border-style shorthand
p {
    border: 1px red solid;
}

margin: 1em 10em; // top/bottom = 1em, left/right = 10em

//background-color, background-image, background-repeat, background-position
body {
    background: white url(http://www.htmldog.com/images/bg.gif) no-repeat top right;
}

Box Model

All html tags/ elements in html can be think of within its own box/ container. And it can be aligned on the screen in accordance with the CSS box model. Here is the box model in diagram:

I keep forgetting where is margin and padding. So, I use the word "MBP" to help me out where M=margin, B=border, P=padding. ;)

At the first glance, it is all about spacing between an element and its neighbors. However, when I see how creative people leverage negative margin. I think it is good to write about it. For margin like "margin: 0px -30px 0px 0px;",

Display Flow

Once you understand how the spacing of an element, you need to under the display flow. Block element (div, p, h1) will display in an newline while Inline element (span, a, img) will display without open up an newline. You can take an element out from its flow via explicitly setting its display properties: inline, block and none. "none" here means not showing it at all. The difference between "display:none" and "visibility:hidden" is "display:none" will take the element out from the flow, so it will not leave an empty space on screen. For example when you want to print an article, you want to just print its content. So, you will do:

#navigation, #seeAlso, #comments, #standards {
    display: none;
}

To take an element out from the display flow but still showing it, there are 2 common ways:

1. Floating
2. Positioning

I will cover these in the layout section.

Layout

margin: -30px 0 15px 0; (clockwise: top, right, bottom, left)
Imagine margin is like a photo white border that make your photo bigger in size and it is often enforced.
So, if a photo has margin inside a div like "right 50" and it is
floating right as well. Because external container doesn't moved by internal element, margin-right 50
will push the photo leftward by 50px instead. And -ve margin here will do the opposite.

When static element (ie. no float) is given a -ve margin on top, left, it pulls the element in that specified direction.
But for bottom and left, it doesn't move but move the content surrounding towards it (ie. overlap with it).

Possibility:
(1) use list to build a table
(2) Overlap to add emphasis
(3) Smashing 3D effect for Bevel text
(4) Layout - Flexible Document Structure

Using negative margins with floats sometimes pisses off older browsers. Some symptoms include:

    * Making links unclickable
    * Text becomes difficult to select
    * Tabbing any links disappears when you lose focus

Solution: Just add position:relative and it works!

Layout

Positioning

position has 4 options: absolute, relative, static and fixed

• absolute will take the element out from the flow and use top, left, buttom and left coordinate to locate it
• fixed did the same as absolute but it will not move even u scroll the browser.

Floating

floating an element will shift it to the right or left of a line, with surrounding content flowing around it. If you don't want the next element to wrap around the floating object, you can apply the clear property like: clear:left, clear:right and clear:both

@ Rules

/* attach a set of css to another css file */
@import url(xyz.css); 

/* apply its content to a specified media (ex. all, aural, handheld, print,
   projection, screen): IE just takes all, screen, print */
@media print{
    body {xxxx}
    ...
}

/* embedded external font (not all browser supports it) */
@font-face {
    font-family: somerandomfontname;
    src: url(somefont.eot);
    font-weight: bold;
}

p {
    font-family: somerandomfontname;
    font-weight: bold;
}
 

Reference

ref: http://www.smashingmagazine.com/2009/07/27/the-definitive-guide-to-using-negative-margins/
ref: http://www.htmldog.com/guides/cssbeginner/
ref: http://www.htmldog.com/reference/cssproperties/
http://devzone.zend.com/article/627

Tip 1: Embedding

Many Adobe Flex applications use external assets like images, sounds, and fonts. Although you can reference and load assets at run time, you often compile these assets into your applications. The process of compiling an asset into your application is called embedding the asset. Flex lets you embed image files, movie files, MP3 files, and TrueType fonts into your applications... When you embed an asset, you compile it into your application's SWF file. The advantage of embedding an asset is that it is included in the SWF file, and can be accessed faster than when the application has to load it from a remote location at run time. The disadvantage of embedding an asset is that your SWF file is larger than if you load the asset at run time - Adobe

There are 3 ways to embed asset to Flex. In Flex code, you can use directive @Embed for direct use or you can associate the embedded asset with a variable by using the [Embed] metadata tag. In Style, you can use Embed to associate asset as well. Go check the syntax in detailed here.

Flex also have an option to embed any kind of file at compile time. The trick is with the ‘mimeType‘ option of the [Embedd] tag. while embedding other kind of files like text (or any) we need  to specify ‘application/octet-stream‘ for mimeType option. Here is the sample:

[Bindable]  
[Embed(source="MyFile.txt", mimeType="application/octet-stream")]  
private var myFileClass:Class;
...

var MyFileByteArray:ByteArrayAsset = ByteArrayAsset(new myFileClass());
var story:String = MyFileByteArray.readUTFBytes(MyFileByteArray.length); 

You must specify that the MIME type for the embedding is application/octet-stream, which causes the byte data to be embedded "as is", with no interpretation. It also causes the autogenerated class to extend ByteArrayAsset rather than another asset class. For example, if you embed a PNG file without specifying this MIME type, the PNG data will be automatically transcoded into the bitmap format used by the player, and a subclass of BitmapAsset will be autogenerated to represent it. But if you specify the MIME type as application/octet-stream, then no transcoding will occur, the PNG data will be embedded as is, and the autogenerated class will extend ByteArrayAsset.

Tip 2: Binding

To make Flex datagrid completed, I would like to have the following featues. AutoCompleted Search - Locate the data I want quickly if there are too many rows in my grid. Internationalization - Handle currency, number and date format. Data Export - Output the data in csv format, so users can import to Excel. Pagination - If I give the total number of records, the subset of the data rows and the number of rows per page, the grid should be able to do pagination and fire the events when user clicks on other pages. This article I will show you how to make these happen.

AutoCompleted Search

After you obtain the resultset from the database and pass it back to Flex as list of value objects, Flex, as usual, will convert it to ArrayCollection of value objects and bind it to the datagrid. If you want to filter out the record in the datagrid, you don't need to remove the records from the ArrayCollection. All you need is to provide the implementation of the ArrayCollection's filterFunction. Here is the example: 

...
[Bindable]
private var _data:ArrayCollection;

private function init():void
{
	_data = new ArrayCollection(
	[
	{ "name":"One" },
	{ "name":"Two" },
	{ "name":"Three" },
	{ "name":"Four" },
	{ "name":"Five" }
	]);

	_data.filterFunction = filterFunction;
}

private function filterFunction( item:Object ):Boolean
{
	var name:String = String( item.name ).toLowerCase();
	var searchStr:String = textInput.text.toLowerCase();
	return searchStr == name.substr( 0, searchStr.length );
}
//trigger when user type in search string in the text box
private function handleChange():void
{
	//this will iterate the dataset against the filterFunction
	_data.refresh();
}
...

Here is the caveat. When using a filterFunction in an ArrayCollection, the Flash Player will always search every item. It is not an effective way. To speed it up, Hillel Coren has documented an approach that the subsequent filters will search on the filtered list instead. His approach is elegant and well-documented. Go to his article for detailed. The demo is here. I found Hillel solution quite elegant. Apart from making the search more efficient, his design also modulizes the search code so that it can be unit tested easily.

The idea is to keep the last failed search string in each record. So, if the next search string begins with last search string, it is unnecessary to check each fields in the record before you can say it won't be matched.

//---- SearchDemo.mxml ---
private function filterFunction( item:Person ):Boolean
{
	return SearchUtils.isMatch( item, textInput.text );
}

//--- SearchUtils.as ---
public static function isMatch( item:ISearchable, searchStr:String ):Boolean
{
	if (_enableFasterSearch && !quickCheck( item, searchStr )){
		return false;
	}
			
	var orSearchStrs:Array = searchStr.split( "," );
			
	for each (var orSearchPart:String in orSearchStrs)
	{
		var andSearchStrs:Array = orSearchPart.split( " " );
		var isMatch:Boolean;
				
		for each (var andSearchStr:String in andSearchStrs){
			isMatch = false;					
			for each (var field:String in item.getSearchFields()){
				if (item.matchesField( field, andSearchStr )){
					isMatch = true;
				}	
			}					
			if (!isMatch){break;}
		}	
				
		if (isMatch){
			item.setLastFailedSearchStr( "" );
			return true;
		}
	}			
	item.setLastFailedSearchStr( searchStr );
	return false;			
}
...

There are several things worth to mention here. The search routine is generic b/c it is against the item that implements the ISearchable Interface. The item implements this interface will provide the implementation of the matchesField method. So, the generic search routine can focus on providing features on top of it like 'AND', 'OR' filter and quick check algorithm.

NOTE: After you set the filterFunction, you will only get the filtered record if you iterate the ArrayCollection. If you want to get the full dataset, you need to do: 

for each (var obj:Object in arrayCollection.source)  
{  
     // do stuff  
} 

Again, thanks for Hillel's tip.

I will talk about Data Export in Part 2 of this series.

In our daily life, we deal with lots of data. The data normally is stored in text format for the ease of human to read. With the large amount of data we have, we need ways to deal with it. There are several things we frequently do on the data: Search, Filter, Sort and Analysis. In Linux, there are some powerful commands that I can use: cat, grep, find, sort, unique and etc. I found those commands quite powerful. So, I decide to put these down as my reference. This tutorial I will go over the basic text processing commands and how we use them together to achieve the tasks we often encounter in our workplace. 

cat

The power of "cat" is not just output a file to screen but to concatenates a list of file content and stream through the pipe to another program as input.

cat * | sort

find

The power of find is to list out the matched filenames based on metadata of the files like type, size, create date...

grep

"grep" helps you to list out the file(s) with the content that match the pattern(s) in regular expression. You can use it as content search across the files in your file system.

grep -H -R --color -n -P abc *

option:

1. --color (highlight matching part in content with color)
2. -n (show line number)
3. -P PATTERN (perl regular expression pattern)
4. -R (recursively)
5. -l (only list out the filenames that match the pattern)
6. -H show filename that matched.

cut

"cut" extracts sections from each line of input. (example of usage). Below the command will extract the 5th field and the rest from each line of file A using delimiter colon.

cut -d ":" -f 5- fileA

option:

1. -c (character)
2. -b (byte)
3. -f 5- (field if the line can be broken down by delimiter)
4. -d | (delimiter is pipe character)

sort 

The sort command sorts a file according to fields--the individual pieces of data on each line. By default, sort assumes that the fields are just words separated by blanks, but you can specify an alternative field delimiter if you want (such as commas or colons). Output from sort is printed to the screen, unless you redirect it to a file.

donor.data
Bay Ching 500000 China
Jack Arta 250000 Indonesia
Cruella Lumper 725000 Malaysia

Let's take this sample donors file and sort it according to the donation amount. The following shows the command to sort the file on the second field (last name) and the output from the command:

sort +1 -2 donors.data
Jack Arta 250000 Indonesia
Bay Ching 500000 China
Cruella Lumper 725000 Malaysia

If the file is delimited by comma, you can use -t , to tell the sort the delimiter. You can use -u to output the uniqueness as well.

sort -t: +1 -2 company.data
Nasium, Jim:031762:Marketing
Jucacion, Ed:396082:Sales
Itorre, Jan:406378:Sales
Ancholie, Mel:636496:Research

To sort the file on the third field (department name) and suppress the duplicates, use this command:

sort -t: -u +2 company.data
Nasium, Jim:031762:Marketing
Ancholie, Mel:636496:Research
Itorre, Jan:406378:Sales

Note that the line for Ed Jucacion did not print, because he's in Sales, and we asked the command (with the -u flag) to suppress lines that were the same in the sort field.

option:

1. -f Make all lines uppercase before sorting (so "Bill" and "bill" are treated the same).
2. -r Sort in reverse order (so "Z" starts the list instead of "A").
3. -n Sort a column in numerical order
4. -tx Use x as the field delimiter (replace x with a comma or other character).
5. -u Suppress all but one line in each set of lines with equal sort fields (so if you sort on a field containing last names, only one "Smith" will appear even if there are several).
6. Specify the sort keys like this: +m Start at the first character of the m+1th field. -n End at the last character of the nth field (if -N omitted, assume the end of the line)

uniq

uniq - line level uniqueness. It prints the unique lines in a sorted file, retaining only one of a run of matching lines. Optionally, it can show only lines that appear exactly once, or lines that appear more than once. uniq requires sorted input since it compares only consecutive lines.

option:

1. -u (print the unqiue lines only - lines only appear once)
2. -d (print the duplicate lines only - lines appear more than once)
3. -c (prefix each line with occurrence)

bash$ cat testfile
This line occurs only once.
This line occurs twice.
This line occurs twice.
This line occurs three times.
This line occurs three times.
This line occurs three times.


bash$ uniq -c testfile
1 This line occurs only once.
2 This line occurs twice.
3 This line occurs three times.


bash$ sort testfile | uniq -c | sort -nr
3 This line occurs three times.
2 This line occurs twice.
1 This line occurs only once.

wc

wc - word count. Apart from word count, it also does the following

1. wc -w gives only the word count.
2. wc -l gives only the line count.
3. wc -c gives only the byte count.
4. wc -m gives only the character count.
5. wc -L gives only the length of the longest line.

tr

"tr" translate or delete characters. It is used for data cleaning job. Can we do pattern replacement?

tr '[:lower:]' '[:upper:]'

The above command will convert all the lowest case to upper case.

tr '.' '/'

The above will convert all the . character to /. And for translation, you cannot have -d option on. You may be asking when would we do that. Here is the common use case - convert window files to unix formatted file:

tr -d '\r' < input_dos_file.txt > output_unix_file.txt

option:

1. -s (squeeze the repeated characters into one character. eg. tr -s '\n' )
2. -d (delete characters eg. tr -d '\000')

sed

"tr" can do character replacement. But if you want to do pattern replacement, you need to use sed. usage: sed -e s/pattern/replacement/flags

sed -e s/one/another

sed -e s/[aeiou]/_/g

 Note the use of the "g" flag so that you apply the pattern/replacement to every match instead of just the first one.

awk

Put them all together

cat * |grep lucene-core|cut -f2 -d' '|uniq|tr '.' '/'| awk '{printf "%s.class\n", $1}'

Behavior

Quite often in threaded applications we have a producer-consumer situation where some threads want to add jobs onto a queue, and some other worker threads want to remove jobs from the queue and then execute them. It is quite useful in such circumstances to write a queue which blocks on pop when there is nothing on the queue. Otherwise the consumers would have to poll, and polling is not very good because it wastes CPU cycles. When clients attempt to retrieve elements but none are available, a BlockingQueue encapsulates code that waits until an element becomes available.

Usage

The server runs infinitely, accepting requests from a client. Clients can send requests asynchronously and rapidly—the server only adds a client request to the server's queue (ie. blocking queue), then returns immediately to the client. A separate server thread loops indefinitely, asking the BlockingQueue to return when a new request becomes available. Once a request is available, it's removed from the queue. The server then handles the request.

Reference

1. Java Specialist - Blocking Queue

ReentrantLock

Behavior

ReentrantLock supports all of the lock-acquisition modes defined by Lock, providing more flexibility for dealing with lock unavailability than does synchronized. Intrinsic locking (ie. synchronized) works fine in most situations but has some functional limitations: It is not possible to interrupt a thread waiting to acquire a lock. If you want the lock, you need to prepare to wait for it forever. Using ReentrantLock, you have more flexibility and scalability:

1. Timed and Polled lock acquisition - Use tryLock(). One form of this method returns immediately if the lock is already held and the other can wait for some period of time for the lock to become available before giving up. In both cases, we could effectively loop and retry the tryLock() until it succeeds.
2. Interruptable lock acquisition - Use lockInterruptabily() allows locking to be used within cancellable activities
3. Non-block structured locking

Usage

Reference

1. Choosing between synchronized and ReentrantLock
2. More flexible, scalable locking in JDK 5 - Brian Goetz

Caching is a crucial performance tuning strategy, especially your system has high read to write ratio. You can perform caching strategy at different levels from client browser cache all the way to disk cache at server side. Lets take a brief look at where we can cache based on the invocation path for a request to be fulfilled:

1. Client browser cache
2. CDN network
   • A CDN is a network, like Akamai, where a web site such as JustProposed.com can offload high-bandwidth static files like photos and videos to another network, so that my web site doesn't need to have such huge bandwidth to run. Since bandwidth is a major expense, especially as we grow or when we get slashdotted (in which case we run out of bandwidth), a CDN has looked interesting. However, Akamai is too expensive for us to use. So, we will go for the free network, Coral CDN.
   • Apart from the bandwidth, JustProposed.com has lots of non-USA users who sometimes find my site slow to use. So, CDN network gives us proximity advantages.
   • To use Coral CDN, you simply append nydu.net:8080 to the end of the hostname in the URL of your expensive resources. For example, http://www.justproposed.com/raydoris/myphoto.jpg to http://www.justproposed.com.nydu.net:8080/raydoris/myphoto.jpg
   • Coral looks great, the only problem I have with it is that it's running on a high port, so that people behind proxy servers that don't automatically support http over anything bug port 80 will have problems. To use Coral, follow this instruction.
3. Reverse proxy server and content accelerator - Squid 
   •  Why not use Apache as reverse proxy instead of putting Squid in front of Apache? Here are some of the benefits of this setup. The main reason is that Apache spawns out a new process per request that eats up lots of resources.
   •  

There are several things that you need to look at when you go for caching approach:

1. What to cache? The data used by most web applications varies in its dynamicity, from completely static to always changing at every request. Everything that has some degree of stability can be cached. However, I always pick the ones that are most frequently access and/or expensive to compute and retrieve to cache because of the limited resource (ie. memory).

Application level caching (for J2EE)

JCS - Java Caching System

1. Configuration
   • To understand the power of JCS, the best way is to look at its configuration file. To find out what is each configurable parameter does, take a look at this article.
2. Integrate with Spring
   • To use JCS with Spring, take a look at this article. It talks about how to create a wrapper or Interceptor for your DAO and inject it to your service for caching purpose. To implement cache as an aspect with full control of what and how to cache, it doesn't use the declarative Spring module caching approach. Regular dependency injection can do the trick!
3. Distributed caching
   • JCS is a front-tier cache that can be configured to maintain consistency across multiple servers by using a centralized remote server (client-server) or by lateral distribution (peer-to-peer) of cache updates. 

Reference

1. Speed up your LAMP stack with lighhttpd
2. Squid and Apache on the same server - have squid listened on port 80 and apache listened on port 8080
3. Squid configuration variable

Traditional web application is based on request and response model that information is delivered as a single payload and then immediately close the connection to the client. To keep the client in sync, we normally pull the server periodically. This approach may generate unacceptable load to the server. To solve this problem, we want to have a push mechanism from server to client. This is why Comet is defined. Comet is a generic term describing various approaches to send data asynchronously from a Web server to a client without the need for the client to explicitly request the data. It is an essential technique for any real-time event-driven web applications, where the majority of events occur on the server and data must be "pushed" frequently to the client. To achieve this, Comet servers must maintain a continuous connection to each client for the duration of the session.

OK. How to maintain a continuous connection to each client for the duration of the session?

If you try to adapt traditional server to the Comet methodology, it may not scale and often fails after a few thousand simultaneously open connections. A true Comet implementation requires a very different kind of server architecture to be efficient and scalable - Liberator (a solid Comet server that are used by the financial industries. However, it is written in C and not open source although it has FREE edition distributed).

To understand this statement a little bit more, we need to know how traditional web containers handle the request. They are under one request per thread model.

1. The client , typically , a browser sends request for resource to a web server.
2. The server has a listening thread that keeps track of incoming connections.
3. When a request arrives , the server uses one process or thread to process the request.
4. The resource is returned to the client and the connection is closed.

In this model, the number of requests that can be served in a second would depend on two things

1. How many threads are there to handle the client requests
2. How long it takes to serve one request.

If all threads of server are busy, then the incoming requests are put in a queue. The server would return to the requests in queue when server threads become free. The number of requests handled per second is always greater than the number of allowed simultaneous connections. All this is made possible because the time required to process a request is very short. In other words you can server more requests in a second than you have threads.

However, there are one breed of applications that need to hold onto the connections. Think of applications that require real time data coming to clients (stock tickers)  or think of applications where low-latency is required. In the above traditional web model, the browser has to re-connect to get the new data. (Polling). If the new updates "can"  happen with high frequency (e.g. a chat application) then the polling frequency also has to increase .  An alternative to high frequency polling is to use push based applications. For push based application, once the browser connects to server, the server will maintain the connection till the browser time-out (server response stream is not closed) and keeps flushing data down the connection as and when they become available. In servlet container, to hold the connection, your thread in the service method cannot exit the method. Otherwise, the response stream will be closed. So what you do is, you block the thread on some condition within the service method. So the thread will block for your condition. When push data becomes available , this thread writes to response stream and again enters a blocked state. So as long as you hold onto the connection, you can not return this thread to the thread pool. And as more and more "push" connections are established you would run out of threads! To remedy the problem, the possible solutions are:

1. Increase # of server threads.

Flex Push

There is confusion that whether BlazeDS supports real time messaging. Yes it does . In fact, BlazeDS has a full spectrum of channel types ranging from simple polling, to near-real-time polling, to real-time streaming.

1. Simple polling - ping the server from Flex client using the traditional request and response model
2. Near-real-time polling (long polling) - Instead of acknowledging right away, the server could hold the polling request until there’s a message for the client. This ensure the messages are delivered to the client as soon as they become available. The caveat for using long-polling is the thread limitation in most application servers. At this moment, BlazeDS could not support more than a few hundred long-polling clients on most application servers. However, this problem could be resolved once servers like Tomcat start to support asynchronos, non-blocking connection threads. Update: Now Tomcat 6 supports NIO.
3. Real-time streaming - BlazeDS supports real-time message streaming over AMF and HTTP. Unlike long polling, which closes and reopens the connection upon receiving a message, streaming keep the connection open at all times. Streaming suffers from the same thread blocking issue as long polling. A cap must be set so the server is not hang by idle threads.

The reason why people are confused is that Adobe doesn't release its proprietary push solution RTMP to BlazeDS. So, RTMP isn't available as a channel in the BlazeDS configuration files. BlazeDS lives in a Servlet container and hence constrained by one-thread-per-connection limit whereas LCDS has NIO-based channels that can scale up to 1000s of requests. On the other hand, BlazeDS has the advantage that it'll work over port 80/443, whereas LCDS will use some port for persistent connections that would require a firewall configuration. Once the servlet that implements BlazeDS is revved to support Comet Events under Tomcat 6, and then Jetty Continuations, then the long polling technique will be fine.

UPDATE: We are waiting for a solution that supports Comet Events under Tomcat 6. Then BlazeDS can be coupled to the Tomcat NIO HTTP listener and be able to scale as well as any NIO based server software.

I have learnt from this article that you can create a channel set in client side. So Flex can fail-over to other channels until it gets connected or the list is exhausted.

Marc has put an effort to build a better data grid like a spreadsheet in Flex. (check this out)

Reference

Here are the references I used for this article

1. Tuning Apache and Tomcat for Web 2.0 comet application
2. Performance of Grids for Streaming Data - This shows you the performance numbers on various frontend technologies. Again, Flex shows us a good result.
3. Are raining comets and threads? - Comet Daily
4. Comet & Java: Threaded Vs Nonblocking IO
5. JDK 1.6 uses epoll to implement NIO
6. BlazeDS dev guide
7. Achieve performance breakthrough using BlazeDS - Farata System put an effort to write its NIO channel that runs on Jetty 7 and receive promising result.

Ext3 (successor of Ext2) is the standard file system for Linux: It is robust, fast and suitable for all fields of use. The main difference between them is that Ext3 has a journal that records the pending operations for fast recovery purpose in the event of system crash. This record guarantees a consistent file system at all times and reduces the time needed for checking a mounted file system from several hours to a few seconds b/c instead of checking the entire disk, the system can check just those areas noted in the journal as having pending operations.

Like all decent Unix file systems, Ext3 uses three general data structures: directories, inodes and data blocks. Directories only contain file names and the inode numbers assigned to them. Each file has one i-node that contains a list of disk block's starting sector addresses as a file content is normally not stored in contiguous disk blocks in disk drive due to constant add and delete and the size is dynamic (ie. external fragmentation). If the file content are scattered, it takes longer to retrieve its content as it takes more header spins physically.

Under the hood, each disk block can span multiple disk sectors and each sector has the size of 512 bytes. Disk sector is the smallest addressable unit on hard disks. Ext3 uses block sizes of 1024, 2048 or 4096 bytes. In theory, Ext3 supports block sizes up to 64 KB, but in x86 and x64 architectures, 4 KB is the maximum: This block size corresponds to that of the kernel's memory pages in RAM, which makes paging easier for the operating system. Ext3 uses 32-bit values (4 bytes as integer in Java) to assign block numbers, which means that it can only address about four billion blocks – 4 TB at a block size of 1024 bytes, 16 TB at 4096 bytes. So, larger block size allows you to create large file system. On the other hand, large blocks can waste a lot of disk space because files always use a whole block even if they only contain a few bytes: On average, every file wastes half a block - the larger the blocks and the smaller the files, the more noticeable the effect is. This effect called internal fragmentation.

Optimization with sacrifice

For an efficient file system, it needs to quickly find the data belonging to a file name. For example, for filename "abc.txt", OS needs to traverse a list of directory entries before the inode of the file is located (depends on the depth of the folder hierarchy) and then traverse all the data block pointers to retrieve the content. To optimize the speed, Ext3 writes the inodes into static tables on the disk during formatting. One consequence of this is that the number of inodes can't be altered after the file system has been set up. As every file needs to be assigned to one specific inode there can't be more files than inodes. It is not scalable for handling large number of files. By default, mke2fs creates one inode for every 4 KB in file systems up to 512 MB, otherwise one inode for every 8 KB. Although you can tune this number to increase the number of inodes, it is only changeable at setup time (not dynamic). By the way, each inode itself consumes 128  bytes in size. 

Handle large size file

How is it possible to fit the millions of data block numbers required for gigabyte-sized files into a static data structure of 128 bytes? It isn't - one Ext3 inode stores exactly 15 block numbers. The first twelve point directly to data blocks, block 13 to a data block containing block numbers (indirectly addressed blocks), block 14 to a block pointing to blocks with block numbers (double indirect), and block 15 points to triple indirect blocks. Therefore, at a block size of 4 KB (that is 1024 block numbers with 4 bytes per indirect block) one inode can handle 12 + 1024 + 10242 + 10243, around a billion block numbers. The resulting maximum file size of just over 4 TB.

Power of B-Tree Indexing

Now you know how inode uses hierarchical pointers to handle file with large size. However, if a directory has tons of files, how directory entries make it efficient to locate the inode. Ext2 originally stored the file names within a directory as a linked list. While this is a elegantly simple data structure it has the disadvantage that operations take longer and longer with a growing number of entries. Ext3 can manage directories in B-Tree+ structure if dir_index is set (not default). This drastically speeds up directory operations. Performance loss is only experienced when the directories are filled with hundreds of thousands of files. This is usually caused by a caching effect as Linux kernel doesn't use unlimited memory for caching directory structures even you add more memory.

$ sudo tune2fs -O dir_index /dev/hda1

Run the above command as root. Do note that the indexing will take up much more space, but then hard disk space is not too expensive nowadays. If you don't want to tweak the OS default setting but you still want to store large number of files. You can restructure the directory so that it does not contain that many files. Without doing this, in a default (untuned) Ext3 partition, each subsequent write degrades horribly past the 2000 file limit. So, keeping the items in a directory to within 2000 files should be fine. If you want to go this route, there are approaches to restructure your folder:

1. Date based - YEAR > MONTH > DATE > HOUR if your files is uniformly distributed across the time.
2. Hash based - break down the hash into several parts as folder name (check this)
3. Id based - reverse the id and break it down use 2 digits each to make sure it is uniformly distributed

NOTE: I don't want to use random number here as I want to locate the file via its metadata later.

Alternative solution for large number of files in a directory

ReiserFS can handle up to 2^31 files per dir (that's 2 billion), with a max of 2^32 (4 billion) files on the filesys total. It can handle up to 64000 subdirs in a directory. Ext3 has a limit of 32000 subdirs per dir. The max number of files per dir is theoretically unlimited (actually around 130 trillion), but performance becomes terrible with above 10-15 thousand files. The max number of total files on the filesys is limited by the number of inodes you have. With a 1 gig file system and a 4k block/ inode ratio (the default), you have around 260000 inodes, and that's also the max number of files you can have.

Reference

Here are some good references

1. The Unix and Internet Fundamental How to - Eric Raymond
2. Tuning Linux file system - Ext3 by Oliver Diedrich
   
3. Handling large number of files in a directory - Roopinder Singh
4. Super fast Ext4 filesystem arrives in Ubuntu 9.04 - If the benchmark is correct, it outperforms all the file system nowadays dramatically.
5. Introduction to Linux file systems and files
6. Extreme performance monitoring and tuning in Linux
7. Simple Help with simple answer - simplehelp.net

This article is written on top of the great work that Sébastien Arbogast has done. He has written 3 articles that showed you how to wire up Flex, BlazeDS, Spring, Hibernate and MySQL with Maven as build process. I have included his articles below as your reference.

1. The Flex, Spring, and BlazeDS full stack – Part 1: Creating a Flex module
2. The Flex, Spring and BlazeDS full stack – Part 2: Writing the to-do list server
3. The Flex, Spring and BlazeDS full stack – Part 3: Putting the application together

I have found Sebastien's work as a good foundation for my own project. To contribute back to the community, I will write a series of articles to show you how can customize and extend the todolist sample.

What is in the Part 1 of the series...

1. Enhancements on the Maven build process
   • Leverage RSL to factor our the framework swc, so the size of the application swf will be reduced. Apart from that, I also take advantage of Flash Player Cache that is available after version 9 update 3 to cache the framework libraries.
   • Clean up the Flex and BlazeDS dependencies in POM as the latest version of the sdk is available and the BlazeDS dependencies are officially available.
   • Include some common reports for maven site generation
   • Embed Jetty web server in the build process for quick deployment and testing
2. Document how to get the sample up on Eclipse for development
   
3. Use Mate as Flex framework
   • Restructure ToDoList sample to leverage Mate framework
   • Factor out Mate as RSL and integrate it with Maven build process via Flex-mojo plugin.

What are in the coming articles...

1. In part 2 of this series, I will show you how to use flex-mojo to build a modular Flex application.
2. In part 3 of this series, I will show you how to test your flex app via FlexUnit (Unit test) and FlexMonkey (Functional test)
3. In part 4 or this series, I will work on server side. I am planning to add monitoring, caching and security to the server side.

Review "ToDoList" sample

Before I start my journey, let me highlight what Sebastien has done first:

1. Sebastien's sample demonstrates how to use Maven as a build process. There are 3 parts or subprojects in his sample. They are:
   
   • todolist-config (configuration files shared by other subprojects)
   • todolist-ria (Flex frontend)
   • todolist-web (Server side that supports the Frontend)
2. All these subprojects are considered as modules of the main project (root POM). Finally, they are combined together into war artifact and ready to deploy to Tomcat or other J2EE webapp server.
3. Flex frontend and backend communicate through a binary RPC protocol - AMF. AMF is considered to be the simplest and fastest remoting approach available in Flex. Recently, Adobe has released BlazeDS as an open source implementation of AMF spec. In this sample, BlazeDS is used. To use BlazeDS, there are few things you need to do:
   
   • Externalize your POJO service via BlazeDS. This sample shows you how to integrate BlazeDS with Spring
   • Make BlazeDS endpoints availabe to the Net via Servlet.
   • Have frontend and backend shared the same BlazeDS configuration files.
4. In this sample, you can also find out how to use flex-mojo maven plugin to compile the Flex frontend code into swf. Apart from flex-mojo plugin, there are other two good plugins worth to mention:
   
   • maven-assembly-plugin - can be used to bundle all the files under a directory into a zip file. It is used by todolist-config to bundle all the configuration files (service-config.xml and remoting-config.xml) into a zip during the package phase.
   • maven-dependency-plugin - can be used to unpack the zip file and move to the place you want. It is used by todolist-web to unpack the config zip during the generate-resources phase.

Enhancements on maven POM

I have modified the sample's maven pom as follows:

• Link to new repository "Sonatype Forge" in the root POM. So, I can use the new version of flex-mojo and simplify the todolist-ria adobe framework dependencies. Apart from that, I also take away the private repository from Sebastein because BlazeDS libraries are available in official maven repository (Note: The BlazeDS libraries available in official maven repo are in version 3.0 instead of 3.0.0.544. So, you need to modify the webapp pom correspondingly).

    <repositories>
        <repository>
            <id>flex-mojos-repository</id>
            <url>http://svn.sonatype.org/flexmojos/repository/</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
    </repositories>

    <pluginRepositories>
        <pluginRepository>
            <id>flex-mojos-repository</id>
            <url>http://svn.sonatype.org/flexmojos/repository/</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </pluginRepository>
    </pluginRepositories>

• Because I link to Sonatype repository, I can have my todolist-ria depends on one flex-framework pom dependency instead of all the swc dependencies. Note that the pom dependency is a way to factor out all the adobe swc dependencies that makes your pom easier to maintain.

        <dependency>
            <groupId>com.adobe.flex.framework</groupId>
            <artifactId>flex-framework</artifactId>
            <version>3.1.0.2710</version>
            <type>pom</type>
        </dependency>

• I include mysql driver as dependency in my webapp pom. I think it is cleaner to bundle it in war. I have also added jetty plugin in the POM so you have a web server embedded in the build process. With this, you can run this sample application right after you check it out from svn (assume you have maven 2 installed). To start jetty, you can issue the following maven command under your webapp project.

project_root> mvn clean install
project_root/jp-web> mvn jetty:run-war

• I have included some reports that will be shown after site generation. You may not be able to do mvn site-deploy because it is linked to my web hosting site. However, you can modify it for your own sake.

Get the sample up on Eclipse

To develop on Eclipse, you can follow the steps below:

1. Create Eclipse project file via running the command below at the project root. This will create 2 eclipse projects. One for todolist-ria and one for the webapp. You noticed that I use the -Declipse.downloadSource=true to include the source files of my dependencies in my eclipse project. Therefore, I can get to the source code if needed.

mvn -Declipse.downloadSource=true eclipse:eclipse

1. Import the projects into Eclipse
2. Add new variable M2_REPO and set it equals to [home]/.m2/repository
3. If you have installed Flex Builder plugin to your Eclipse, you can Add Flex Project Nature to the todolist-ria project.
   • Select Application Server Type: J2EE
   • Put check on "Use remote object access service" with LiveCycle Data Service selected.
   • Set up the path. I have my tomcat installed under C:\tools with default 8080 as port. You should make the changes if you installed it differently.
   • 
   • Remove the generated main.mxml under the src folder.
   • Set index.mxml under src folder as default Flex application file to run.
   • Use Default Flex SDK in Flex Compiler Configuration instead of Server Flex SDK
   • Right click and select Recreate HTML Template if you see error.
   • After all these, you have configured your Flex application pointing to the webapp server and sharing the BlazeDS configuration files. You can verify in Flex Compiler Configuration's Additional Compiler Parameters. See whether you see this: -services "C:\tools\tomcat-6.0.16\webapps\jp\WEB-INF\flex\services-config.xml" -locale en_US
   • Move the war to your tomcat's webapp folder and start it under remote debugging setting. If you are using window, set DEBUG_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=8787,suspend=n under your bin/catalina.bat.
   • Start your webapp via bin/startup.bat
   • Put breakpoint under TodoServiceImpl save method and start remote debugger on localhost:8787
   • Right click the index.mxml and Run As Flex Application.
   • Add a new entry and save it on the flex app. You should see your remote debugger halt at the breakpoint for you to debug.
   • Now you can change your flex code and test it out without leaving your Eclipse. However, if you modify the service in webapp, you need to run "mvn clean install" and deploy the war to the tomcat before your flex code can call your server-side code via AMF.

Use Mate as Framework

If you are not familiar with Mate, click the image below that moves you to a nice presentation.

What did I do to restructure the todolist sample to make it Mate app?

Download

I have made my work available at: www.solutionhacker.com/wp-content/uploads/todolist-jp-modified.zip

Reference

Below are the references I used for the article:

1. Flex mojo compiler user guide
2. Flex mojo dependency scope rules
3. Flex 3 feature introduction: Flex 3 RSL
4. Improving Flex application performance using Flash Player Cache
5. FNA archetype projects 

Testing Strategies Overview

I believe that I don't need to stress out how important testing is. So, lets cut the crap and talk about the challenging areas related to this subject. Here are the things I want to talk about

1. Write your unit test first then code
   
   • It is the core concept from Test Driven Development (TDD). Everyone tried that will tell how great it is. However, the difficult part of it is to enforce it across your team especially under the time constraint. Everyone wants to get something out right away and we are all getting so much stresses from our business side about features release to stay competitive in the market. How can we allocate time for this "luxury"? The key here is whether you believe it will save you tons of time in a long run. If not, it is hard to carry out this idea.
   • What are the areas that could save my time? Agile development may face one big challenge from the product side. The problem is mainly due to the way they break down the project into mini-specs to fit into a sprint of work. Sometimes they find it difficult to think of a project in a micro level. To deliver the mini-spec in a agile process, they may leave out the detail in the spec and try to fill them up in an iterative fashion. Write your test first forces you to think API, input, output and use cases ahead of time. This will help you to feedback the product team earlier in the loop. Otherwise, it will cause you more time to change your logic or patch your codes later.
   • If you have a suite of unit tests, you are more confidence to do refactoring because you can perform regression tests to make sure nothing is broken. Apart from that, you can add new features to someone's code without worrying that it will break the old features as you have the unit tests from the owner to verify.
   • Things you need to pay attention are: (1) Don't commit code to the trunk that breaks the build (compile), but it is ok that it doesn't pass the tests temporary as you are in the middle to fill up the code and you don't want to have your code sitting in your local box too long. (2) Do code coverage to make sure you have your unit test exercises your codes in good %.
   • Tools I used to write my unit test: TestNG/ JUnit for Java code, FlexUnit for flex code
   • How do we unit test our DAO? Test with your local database (use same vendor and version of the database as the one in production. I don't like to use any memory databases like Hypersonic or SQLite b/c it doesn't really test our your SQL).
2. Functional testing - use Selenium to automate it.

Performance/load testing

Basic workflow

1. Create use case scripts for your application via recording. For Flex, we want a tool that can convert the binary AMF message to readable text as part of our script.
2. Introduce natural variability - varying the login credentials, records opened, records saved, and other user actions including thinking time. This step is important because it provides the real-world variability that keeps your application working hard and prevents you from missing performance problems because of caching at various layers on your server.
3. Define test data sets to be used by your scripts as you ramp up hundreds or thousands of simulated users executing your tests.
4. Define profiles of load on your application. You might decide to have 20% of your users entering information slowly, 20% entering information very quickly, 40% looking up information, 10% producing reports, and 10% making account changes. The breakdowns and level of detail are entirely up to you.
5. Specify how to run the test. You control such things as the speed of ramping up user load, how fast each user executes the test, and whether there should be natural variability between users and sessions. Each tool provides different fine-grained control over this step.

Tips

• Don't carry performance test without passing all the functional tests.
• Test against single box first and you should not run JMeter on the same machine running the application to be tested.
• Make sure that the testing is affected as little as possible by network traffic (do it in the subnet to take out network IO factor).
• Create a test plan that contains thread group(s) that controls the threads that will be created by JMeter to simulate simultaneous users. Determine # of threads and the ramp-up period. Add HttpRequest then listener to view result in table and/or graph format. Each thread will issue an HttpRequest and response time will be recorded. Then avg response time and standard deviation will be calculated. Another way people uses JMeter is to use its HTTP session recording to produce load tests.
• On the other hand, you need to monitor your target in terms of CPU, memory and IO consumption (use OpenNMS) during the perfomance testing.
• Create a scheduled integration build that runs JMeter tests (use CruiseControl). Read this article for integration detail.
• How to achieve the load via distribution?
• How to obtain meaningful information from the raw data?
• Look at throughput (QPS), resources consumption (CPU, Memory, IO) over time and over users/ load.
• Need to set up the database with good amount of data for testing.
• Performance profiling
  • Obtain the rough data from performance test first
  • Identify the bottleneck and fix the problem.
  • Don't waste too much time to over-tune your system. Just fix the bottlenecks that impact the SLA.
• Stress testing
• Scalability testing - testing on scaling out or up your system.
• Availability testing

Reference

Below are the references I read to write this article

1. http://www.opensourcetesting.org
2. JMeter User Manual