In fact, from his TOS, it stated clearly that the most important reason for Apple to ban Flash is because Apple doesn't want a third party layer of software come between the platform and the developer.

Allowing Flash — which is a development platform of its own — would just be too dangerous for Apple, a company that enjoys exerting total dominance over its hardware and the software that runs on it. Flash has evolved from being a mere animation player into a multimedia platform capable of running applications of its own. That means Flash would open a new door for application developers to get their software onto the iPhone: Just code them in Flash and put them on a web page. In so doing, Flash would divert business from the App Store, as well as enable publishers to distribute music, videos and movies that could compete with the iTunes Store – Brian on wired.com

Recently, I need to pick up Perl for some of my projects. After going through some websites and books, I start seeing why some of the tasks could be done much easier in Perl than in Java, the language that I am quite familiar with. Since Perl is loose in data type, not OO-enforced, non-explicit signature plus its syntax is full of symbols, some Perl programmers given the freedom and power without following the best practices could write very illegible code. Although it runs, it could be very hard to maintain. For those who feel the pains, you may like this post . However, I don't want to go extreme on this. I do see its power in text processing and system integration. After taking a look at the extensive CPAN library, I decide to give it a fair trial. Not only that, I will write a series of posts to help you to shorten your learning curve as well. This post I will put my head on Perl data type and its usages.

Data Type

Number and String

# numeric
$var = 123;
$var = 123.34;  #float or double

# very large long unsigned integer like md5 value.
$bignum = Math::BigInt->new("Ox1821231223238234234");

# string - single quote: no interpretation; double quote: interpreted at runtime
$myVar = 'abc';
$myVar = "product price is $price";

Array

@myarray = (1,2,4, 'abc', "abc", $myvar);
print $myarray[0]; #1 is shown

@coins = ("Quarter","Dime","Nickel"); # quotation could be hassle!
@coins = qw(Quarter Dime Nickel);     # remove the quotation headache easily

# To get numeric length of an array, you can use the scalar() function or
# we can redefine the array as a scalar variable.
$coinlength = @coins;
print scalar(@coins);
print $coinlength;

# array in perl can dynamic growth in size
push(@coins, "Penny");     #add element to the end of the array
unshift(@coins, "Dollar"); #add element to the front of the array
pop(@coins);               #remove last element of the array
shift(@coins);             #remove first element of the array
delete $coins[1];          #delete element with index 1 in the array

@names = split(',',$namelist); #convert csv line in string to array
$namelist = join(",",@names);  #reconstruct via joining them back

@Foods = qw(Pizza Steak chicken burgers);
@foods = sort(@Foods);         #sort the array according to ASCII Numeric

# transform to lowercase as Capital case can mess up the order
foreach $food (@Foods) {
	push(@foods,  "\L$food");
}

# sort it
@foods = sort(@foods);

my $count = 0;
for (@list) {
   $count++ if $_ eq "apple"; # $_ is assigned the current element of the loop
}

# slice the array
my @stuff = qw/everybody wants a rock/;
my @rock  = @stuff[1 .. $#stuff];      # @rock is qw/wants a rock/
my @want  = @stuff[ 0 .. 1];           # @want is qw/everybody wants/

Hash

Hash is also called associative array that is composed of a list of key-value pair entries and the keys here are unique. It is like Map in Java.

%myhash = ('key1' => 'abc', 'key2' => 5, 'key3' => $myvar);

# even number array can convert into hash
# element[i] => element[i+1] where i is 0,2,4..etc
%coins = ( "Quarter", 25, "Dime", 10, "Nickel", 5 );   

$myKey = "Dime";
$coins{$myKey} = 15;  # assign using variable
print $coins{"Dime"}; # output is 15;
print $coins{Dime};   # you can simplify it without quote there

foreach $key (sort keys %coins) {
     print "$key: $coins{$key} \n";
}

$coins{HalfDollar} = .50;   #add new element
delete($coins{HalfDollar}); #delete an element

if ( exists $hash{key} ){
   #retrieve value here
}

# slice a hash
my %table = qw/schmoe joe smith john simpson bart/;
my @friends = @table{'schmoe', 'smith'};   # @friends has qw/joe john/

Subroutine

Parameters passes to subroutine will be stored in @_.  You can create a reference of a subroutine and pass it to a method as function pointer.

sub my_sub{
  my($msg) = @_;
  print "this is my own message: $msg";
}
&my_sub(); #call with option & in front
my_sub('hello world'); #call without & in front and parameter can be dynamic here.

Reference and Dereference

So far so good? Now I am going to show you how to pass by reference rather than pass by value. The reason you want to pass by reference because it is more memory efficient. However, to do that, I have to take you to a trip of syntax mess. It is what gave me a hard time before and I hope I can make it clear so you will not be suffered the pain I had. OK. Lets start! Reference is a scalar that refers to the data stored in another variable of any type, as well as subroutine and methods. It gives you the ability to pass by reference a large variable to a function instead of pass by value.

Reference

$varRef = \$myvar; #varRef now stores the physical address of $myvar
$arrayRef = \@myarray;
$subRef = \&my_sub;
$arrayRef = [1, 2, 3]; #arrayRef is pointing to the address of an anonymous array

%hash = ('key1'=> 'var1', 'key2' => 'var2'); #regular hash
$rhash = \%hash;  #create reference to the hash
$href = {'key1'=> 'var1', 'key2' => 'var2'}; #reference to anonymous hash

Dereference

To access the content of a reference, there are special ways to do that:

my $name = "Test Users";
my $rname = \$name; 

sub my_func{
  my ($ref) = @_;
  print "Scalar ref value is $$ref";  #dereference
}

my_func($rname);
my_func(\$name);

# array can be deference in 2 ways
$arrayRef->[1];
$$arrayRef[1];
@array = @$arrayRef; #the whole array available again for regular usage

print $hash{key1};        # Ordinary hash lookup
print $$rhash{key1};      # hash replaced by $rhash
print $rhash->{key1};     # hash replaced by $rhash

# we are getting ourselves confused via using the syntax of dereference,
# we could get back the actual value via:
@{array_reference}
%{hash_reference}
${scalar_reference}

print Dumper %$rhash;     # dump the whole hash (valid syntax without the bracket)
for(keys %{$rhash})       # for more clarity, enclose it in curly brackets.
{
...
}

# map of anonymous subroutines
my %hash = (add => sub {my var1 = @_; print "Add $var1\n"},
            substract => sub {my var1 = @_; print "Subtract $var1 \n"} );
# get a function pointer and invoke it using -> if you need to pass parameter(s).
$hash{$add}->(5);

Confused! Give me a chart!!

I hope that you are still ok. Sometimes you see @varArray in code, it could mean the array itself or the size of it. It depends on the context. If @varArray is under scalar context, it means the length of the array. Otherwise, it means the array itself. To avoid confusion, you can always use scalar(@varArray) to get the size of the array. However, someone may use $#varArray+1 to represent the length of it. In fact, $#varArray means the last index of the array. Since array index is starting from 0, so you add 1 to get the actual size of the array. Below is a chart that I see it quite helpful when I get lost in the Perl syntax.

============================================================================================================
Expression 	 Context 	 Variable 	        Evaluates to
============================================================================================================
$scalar 	 scalar 	 $scalar, a scalar 	the value held in $scalar
@array 	         list 	         @array, an array 	the list of values (in order) held in @array
@array 	         scalar 	 @array, an array 	the total number of elements in @array (same as $#array + 1)
$array[$x] 	 scalar 	 @array, an array 	the ($x+1)th element of @array
$#array 	 scalar  	 @array, an array 	the subscript of the last element in @array (same as @array -1)
@array[$x, $y] 	 list 	         @array, an array 	a slice, listing two elements from @array (same as ($array[$x], $array[$y]))
"$scalar" 	 scalar          $scalar, a scalar 	a string containing the contents of $scalar
"@array" 	 scalar          @array, an array 	a string containing the elements of @array, separated by spaces
%hash 	         list 	         %hash, a hash 	        a list of alternating keys and values from %hash
$hash{$x} 	 scalar 	 %hash, a hash 	        the element from %hash with the key of $x
@hash{$x, $y} 	 list 	         %hash, a hash 	        a slice, listing two elements from %hash (same as ($hash{$x}, $hash{$y})

Real Life Usage

Now you have gone through basically the most common use of Perl syntax. If you are still reading this, congratulation! You can now move a step further to see how it applies to some real life examples. I am going to show how people use Hash because I see it more fun.

#--------- counting ---------
How you generate a histogram of term from a text file?
(1) Convert the text to a list of string term (you could eliminate the punctuation, normalize case, get rid of the stop words as you wish)
(2) Walk through the list and build a map of term with count as value. How? In Java, I will loop thru the list check if the term is there
    in the map. If not, put it there with count=1. If yes, pull the current count of the term and increment it. In Perl, the syntax is much easier:

my %histogram;
$histogram{$_}++ for @list;  # for loop on the list and each element by default is assigned to $_'
$unique = keys %histogram;   # obtain the number of unique terms
@unique = keys %histogram;   # obtain the list of the keys
@popular = (sort { $histogram{$b} <=> $histogram{$a} } @unique)[0..4];  # obtain top 5 based on the count

#--------- searching ---------
my $index;
for $index (0..@chambers) {               # linear search
   last if $chambers[$index] == $bullet;  #exist at this condition
}
print "Found at index $index" if $index < @chambers;

NOTICE: we see lot of actions in perl can be controlled by condition followed. If condition returns false, the whole statement will not be executed.

#---------- dispatch table -----------

# Suppose you have a script that does several related things: it manages your to-do list by adding, editing, listing, and deleting to-do items:

>> todo add "Email Samuel about photos"
Output: Todo item 129 created
>> todo done 129
Output: Item 129 marked as done

# You might expect the script to look like:

    my $command = shift @ARGV;
    if    ($command eq "add")  { add(@ARGV)  }
    elsif ($command eq "list") { list(@ARGV) }
    elsif ($command eq "done") { done(@ARGV) }
    elsif ($command eq "edit") { edit(@ARGV) }
    ...
    else { die "Unknown command: $command" }

# That is quite tedious. You could use hash to deal with this since we don't have 'switch' in perl:

    %commands = (
        add  => \&add,
        list => \&list,
        edit => \&edit,
        done => \&done,
    );
    my $action = shift @ARGV;
    if (!exists $commands{$action}) { die "Unknown command: $command" }
    $commands{$action}->(@ARGV);  #dereference the subroutine and use -> for argument passing

Conclusion

Thanks for reading up such a long post. Buy yourself a coffee because you are now equipped an essential skill to write Perl. You may wonder why I haven't shown you how get your environment setup and write the first Hello World program in Perl. I don't do this because there are tons of articles showing you how to do this. Apart from that, I also skip some basic things like conditional control, packaging, library usage, exception handling and etc. I don't cover this because I see the syntax of them quite easy to grasp. Next article, I am planning to cover the power of text processing with some deep dive in regular expression. Stay tune! There will be more symbols to get familiar with. By the way, if you see some weird syntax related to data structure, put a comment on my blog so we can learn from each other.

As I mentioned in my last article,  I was getting excited about the potential of Hive. Today, I decide to start my journey to learn this. I found a great introductory video that gives you a nice warm-up of using Hive (A basic knowledge of how hadoop and mapreduce work would be helpful for you to digest the material inside).

Below are some highlights from this video

Hive is an SQL interface built on top of Hadoop. It supports Web access and JDBC. I am amazed how close the SQL syntax like the regular SQL for RDBMS. Below are some SQLs used in this tutorial.

//———- Set up your tables in HIVE —————–
SHOW TABLES;

CREATE TABLE shakespeare (freq INT, word STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t’ STORED AS TEXTFILE;

DESCRIBE shakespeare;

//———- Load data into Hive table from Hadoop HDFS ——————-
LOAD DATA INPATH “shakespeare_freq” INTO TABLE shakespeare;

//———- Query against the data using hive sql interface ————–
select * from shakespeare limit 10;
select * from sakespeare where freq > 100 sort by freq asc limit 10;
select freq, count(1) as f2 from shakespeare group by freq sort by f2 desc limit 10;

//show me the plan
explain select freq, count(1) as f2 from shakespeare group by freq sort by f2 desc limit 10;

//———- Create a merge table and populate it using dataset joining by 2 different tables
insert overwrite table merged select s.word, s.freq, k.freq from shakespeare s join kjv k on (s.word = k.word);

//———- Query the merge table ———————
select word, shake_f, kjv_f, (shake_f+kjv_f) as ss from merged sort by ss limit 20;

To prepare the data for Hive to load in, the demo uses another mapreduce job to achieve. Remember to delete the log before doing Hive table load.

hadoop jar $HADOOP_HOME/hadoop-*-examples.jar grep input shakespeare_freq ‘\w+’

//remove the mapreduce job log
hadoop fs -rmr shakespeare_freq/_logs

Often time, large scale data processing system always IO bound. So for mapreduce job, your mapper is always waiting for data to load from disk. Hadoop mitigates the problem via during parallel load from lots of hard drives. However, a single hard drive is still max out at 75MB/s read as physical limit and nothing we can do about this. In order to achieve good speed, the key is to eliminate # of hadoop pass

Since Hive is on top of Hadoop’s HDFS, it will have the same restrictions as it. So, you cannot do UPDATE, DELETE and INSERT records as regular RDMS. However, you can do bulk load to add more new files (data) to the table and you can do delete a file from Hive.

Hive needs to store metadata of the tables out from the HDFS. You can use regular rdms to achieve the job. But when you start Hive locally, it will seek for the local metastore. So, in distributed environment, you may need to centralize the metastore in a remote location. There is wiki on the Hive site that documents how to set it up.

See Hive in Action

Cloudera Hadoop Training: Hive Tutorial Screencast from Cloudera on Vimeo.

Other projects similar to Hadoop

• Parallel databases: Gama, Bubba, Volcano
• Google: Sawzall
• Yahoo: Pig
• IBM Research: JAQL
• Microsoft: DryadLINQ, SCOPE
• Greenplum: YAML MapReduce
• Aster Data: In-database MapReduce
• Business.com: CloudBase

I ever worked for a display ad network company that collects over 400 million of impression/ click logs per day. With this amount of data, my ex-company bought a supercomputer and cross their fingers that it can handle the grow in both volume and analytic demand of the data. It is obviously not a scalable solution. However, what is the best solution?

Although I haven’t worked for this company anymore, it is still an interesting problem to solve. I have a great friend who proposed a shared nothing solution for this company. The solution is to partition the data across a set of Postgresql databases and put Greenplum on top of them to parallelize the query —there is no disk-level sharing or contention to be concerned with (i.e. it is a ‘shared-nothing’ architecture). I like this approach. The only thing is that Greenplum is not free and it may be difficult for a startup to face this upfront cost. Apart from that, this setting requires all the databases are running on the same network that hindered us to move this in the elastic cloud like Amazon EC2.

Later on, I joined a great company in the same industry that seeks for a solution in the cloud to host its data warehouse. So, I got a  chance to revisit this problem. During the research, I came across an interesting technology – column-based database (eg. infobright and lucid db). The idea of column-based data store is that traditional database stores and fetches data in row from data files into the memory. It is inefficient if your query only requires few columns for computation. So, column-based data stores your data in column with effective compression algorithm due to all values in it has the same data type. This solution is great but it doesn’t do MPP (ie. massive parallel processing) and it is also not ready for cloud yet.

Here comes another solution. That is Hive on top of Hadoop on top of Amazon cloud. It is an interesting idea. Check out this video to learn about this.

If you are not sure what Hadoop is and want to get some warm up in massive computing. I suggest you go through the following 5 excellent Google lectures.

• Cluster Computing and MapReduce – Lecture 1
• Cluster Computing and MapReduce – Lecture 2
• Cluster Computing and MapReduce – Lecture 3
• Cluster Computing and MapReduce – Lecture 4
• Cluster Computing and MapReduce – Lecture 5

I want to summarize some new and interesting Java 5 features in this article and how they change the way I code.

Enum

I use int constants to make my life easier b/c it can avoid typo. However, it has several drawbacks:

1. Java doesn’t provide namespace for int enum groups. I can either prefix my constant like ABC_ or using inner interfaces to organize it.
2. It is compile-time constants. So you need to recompile once changed.
3. No easy way to translate int enum constants into printable string during debugging.
4. You cannot iterate over all the int enum easily.
5. You need a way to validate the enum is an valid int

Use new enum type in Java 5:

public enum Apple {FUJI, PIPPIN, GRANNY_SMITH}

Enum is full-fledged final class that export one instance for each enumeration constant via a public static final field.

1. Namespace is provided via the enum type name.
2. You can reorder and add the enumeration constant without recompiling its client.
3. You can translate enum into printable strings via toString() method.
4. Enum type provides you values() method to iterate your enumeration constants (based on declaration order).
5. Type-checking can be used for the validation check
6. You can associate data with enum constant
7. Enum is immutable, serializable and comparable.

EnumSet

If elements of an enumerated types are used primarily in sets, it is traditional to use the int enum pattern, assigning a different power of 2 to each constant like READ = 1 << 2, WRITE = 1 <<1, EXECUTE = 1 << 0 to represent permissions per each entity in Unix. This representation lets you use the bitwise OR operation to combine several constants into a set, known as a bit field. The bit field representation also lets you perform set operations such as union and intersection efficiently using bitwise arithmetic. But bit fields have all the disadvantages of int enum mentioned above.

Now, java.util package provides the EnumSet to efficient represent sets of value drawn from single enum type. This class implements Set interface and internally use bit vector to represent set of values. For example, if you enum types has 64 values, the entire EnumSet can be represented as a single long, so its performance is comparable to the bit field.

The EnumSet class provides three benefits a normal set does not:

1. Various creation methods that simplify the construction of a set based on an Enumeration
2. Guaranteed ordering of the elements in the set based on their order in the enumeration constants are declared
3. Performance and memory benefits not nearly possible with a regular set implementation

Annotation

An annotation is a new language feature introduced in J2SE 5.0. Simply put, annotations allow developers to mark classes, methods, and members with secondary information that is not part of the operating code.You can see annotation is a way to extend Java language.

Before annotation from Java 5, you may use naming patterns to indicate that some program elements like method demanded special treatment by a tool or a framework. Like JUnit required its users to name the test methods with the pattern like testXXX(). It works but with some big disadvantages:

1. Typo problem
2. It doesn’t provide a way to associate parameter values with program elements.

Annotation can solve this problem. To use it, you can:

1. Create you own marker annotation (@interface is the keyword) or parametized annotation. You can annotate the annotation (ie. meta-annotation). Example: @Retention and @Target. And marker annotation has no parameter associated with it.
2. Annotate the program elements
3. Write processor to handle your annotated code. Generally, annotations never change the semantics of the annotated code, but enable it for special treatment by tools. Now, the metadata of Method carries additional info for your job. You can use Method’s isAnnotationPresent() to check if a method is annotated by certain annotation type. If you annotation carried parameter, you can use Method’s getAnnotation() to get the Annotaton object and use value() to obtain the parameter.

Reference

Below are some related articles I feel useful:

1. http://www.javalobby.org/java/forums/t16967.html
2. Annotation in Tiger – Part 1 Meta-Annotation
3. Annotation in Tiger – Part 2 Custom Annotation

Set up virtual office

1. Skype – Save you from long distance bill
   • FREE. If you pay a rate, it will let you connect to phone line.
   • When I am tired of typing, it will switch to this.
   • If you like face to face conversion, I would use iChat on Mac. All you need is an AOL account.
2. Yugma – Web conferencing
   • FREE up to 20 attendees.
   • It has Skype integration.
3. Free conferencing
   • FREE
   • In case you don’t have internet access or your laptop is not next to you. This is a good tool because it gives you a dedicated line to dail in. However, the number is not TOLL-FREE.
   • Why 712 area code? Check this out
4. Google tools – all FREE
   • Google doc (shareable)
   • Google calendar (it can sync with my iCal on Mac now. If you use Outlook, you need to install a plugin to do the calendar sync). Follow this guide to set it up.
   • Google email (have gmail to host your mail server – yourname@yourcompany.com)
5. MediaWiki – good wiki tool for information sharing
   
6. Posterous – create your company blog via email
7. iPhone
   • Not FREE
   • I use it for sync email, calendar and access Web.
8. VNC – Remote desktop tool
   • FREE
   • For Mac, download OSXVine Server from here.
   • If you are using DSL that assigns you IP address dynamically, it is quite a headache to keep track of it. You can obtain a domain name from DynDNS to abstract you from the IP address.
   • By default, VNC server will be listened on port 5900. If you want to do remote desktop outside your subnet, make sure your DSL router open a port for that and forward the request to your machine.
   • Here is a web-based VNC viewer. With that, you can do remote desktop anywhere. Just key in your dynDNS domain and you are done.
   • If you want to make this access security, you can password protected your box via configure the VNS Server.
   • Here is a great article for that.
   • There are people who uses LogMeIn service that provides more remote secure featuers. However, it is NOT FREE. To me, VNC Server is good enough solution.

Build your virtual dev team

1. Tracs – combine wiki, ticket system, project planning in one
   • FREE
   • A bit complicated to set it up in web hosting company
   • It integrates with Subversion as well
   • Bugzilla is pretty good for issue tracking as well
2. Dreamhost for web hosting
   • Below $10 per month
   • I currently use it for hosting my own blog and subversion repository.
   • No java support yet.
   • For VPS solution, this one is cheap and my buddy said it is great.
3. VirtualBox – have several operating system runs on your laptop. Very appealing!
   • FREE
   • I set up Ubundu on my Mac. Full screen, share folders, share mouse. I love it.
   • With this, I can ensure all developers are working on the same environment. Furthermore, I can have dev, qa and production using the same environment.
4. Omnigraffle – design graphical tool on Mac
   • NOT FREE but cheap
   • Free stencils available on here.
5.  Amazon AWS
   • Way low cost comparing to hire your own team to make sure your system 24×7
   • Cloud computing allows you scale on demand.
   • Processing power via EC2
   • Storage via S3
   • CDN via CloudFront
   • Messaging via Amazon SQS

Update

SQLite 3 is released that addressed some of its issues in version 2.

• BLOB support
• Fulltext searching
• Connection shared between threads
• Improve concurrency

However, it still doesn’t support writeable view, nested transaction and foreign key.

Presentation

Here is a nice presentation from Paul Roberson, look at it first.

 Note from the video:

1. Warm up with general SQL Tips (Join favors subquery, Avoid IN, Avoid LIKE, Specify columns name in select and insert, Avoid unnecessary join)
2. AIR SQL connection can connect up to 10 databases at a time, you can use qualifier for your tables.
3. Don’t reuse the same SQL Statement for different prepared statements.
4. Use transaction to do batch insert/ update/ delete operations (48x faster!)
5. Index columns in WHERE clause, use together index together
6. Create table structure before you add data because internally SQLite…?
7. Handle large resultset in parts for perceived performance gain (detailed)

There are several things I want to find out:

1. Can SQLite handles large dataset?
   • Yes. According to spec, it can handle terabyte of data.
2. Does SQLite support pagination?
   • Yes. Look at SQL Statement object.
3. How SQLite synchronize with the updated data from the remote database?
   • Strategies: overwrite vs delta (timestamp, field by field comparison, dirty flag)
   • Live cycle data service has built-in SQLite synchronization support including offline caching and conflict management.
   • http://coenraets.org/blog/2008/05/insync-automatic-offline-data-synchronization-in-air-using-lcds-26/

Some notes about SQLite

Below are some SQLite tips and practices I obtained from different sources:

• A big advantage of sqlite above a flat file is the possibility to index your data.
• Using parameterize query protects against sql injection, and makes the ‘ problems go away. It is also much faster because sqlite can reuse the execution plan of statements when you use parameters.
• Make sure to import the records in a transaction so that it doesn’t spend a lot of time creating indexes until everything is imported.
• The SQLite documentation states that SQLite databases can be terabytes in size, and that the primary limitation of SQLite is concurrency (many users at the same time).
• “The SQLite database is pretty damn fast. I was getting near instantaneous searching with databases that were ~100,000 records. Somewhere around 800,000 – 1,000,000 records you start losing performance, waiting a few seconds for a search” – by Daniel
• Each database is contained within a single file.

Reference

Below are some good links I have found:

• Develop air application with large dataset
• SQLite at the Lake – Peter Elst (06/26/2009)
• SQLite admin tool by David Deraedt – not open source
• Another SQLite admin tool by Christophe Coenraets (10/2007) – open source
• Firefox SQLite Manager plugin – support csv import. However, the project is no longer active.
• Adobe Air and SQLite video session (4/18/2008) – it contains 3 interesting examples.
  
• Get started with adobe air and sqlite (1/28/2009)
• Develop the connection and statement pool

What is ESB?

ESB = Enterprise Service Bus. The definition is flexible, but in general it’s a conduit for messages of multiple, different formats, between application endpoints, over more than one protocol.

Mule vs ServiceMix

1. Compared to Mule, the major difference for ServiceMix is its architectural design, which is fundamentally based on the Java Business Integration (JBI) standard. Mule provides a JBI binding so that Mule components can interact with JBI containers, including the ServiceMix JBI container. However, the internal Mule APIs are not based on the JBI standard.
2. JBI uses a notion of Message Exchanges and a Normalized Message to communicate between components, where as Mule use a “POJO / Endpoint” architecture.
3. Mule doesn’t require your service to implement any Mule interface. The components are wired up thru the mule-config.xml file.дивани

The video above can give you a good taste of Mule. But the example runs as standalone app. If you want to use Mule in your web application. I found this distributed raman amplifiervideo a good starting point.

What is EDA?

Asynchronous messaging lets two or more applications send data to each other without having to wait for receipt confirmation. The infrastructure guarantees message delivery, even if the receiving application isn’t currently running or the network connection is interrupted. It sounds simple enough, but asynchronous messaging demands a new way of thinking about system architecture.

Caveat of asynchronous messaging

1. The convenience of sending guaranteed messages literally around the world to applications running on different platforms or technologies is a huge benefit. What’s the catch? The messaging system can guarantee that the message will be delivered, but it does not guarantee when it will be delivered.
2. Worse yet, if an application sends two messages, the messaging system doesn’t guarantee that they’ll arrive in the same order.
3. Because destinations are unidirectional, the application receives responses on a different destination than the one on which it publishes requests. This means that the responses can arrive in a different order than the requests were sent. As a result, the application must explicitly correlate incoming response messages to the requests. In the asynchronous world, correlation is such a common need that the JMS API includes the methods getJMSCorrelationID and setJMSCorrelationID in the message interface.
4. How do we have sender and receiver in a transaction?

Reference

1. Enterprise Architecture Patterns – by Martin Fowler
2. An Asynchronous World
3. Errant Architecture
4.  

Tip 1: Embedding

Many Adobe Flex applications use external assets like images, sounds, and fonts. Although you can reference and load assets at run time, you often compile these assets into your applications. The process of compiling an asset into your application is called embedding the asset. Flex lets you embed image files, movie files, MP3 files, and TrueType fonts into your applications… When you embed an asset, you compile it into your application’s SWF file. The advantage of embedding an asset is that it is included in the SWF file, and can be accessed faster than when the application has to load it from a remote location at run time. The disadvantage of embedding an asset is that your SWF file is larger than if you load the asset at run time – Adobe

There are 3 ways to embed asset to Flex. In Flex code, you can use directive @Embed for direct use or you can associate the embedded asset with a variable by using the [Embed] metadata tag. In Style, you can use Embed to associate asset as well. Go check the syntax in detailed here.

Flex also have an option to embed any kind of file at compile time. The trick is with the ‘mimeType‘ option of the [Embedd] tag. while embedding other kind of files like text (or any) we need  to specify ‘application/octet-stream‘ for mimeType option. Here is the sample:

[Bindable]
[Embed(source="MyFile.txt", mimeType="application/octet-stream")]
private var myFileClass:Class;
...

var MyFileByteArray:ByteArrayAsset = ByteArrayAsset(new myFileClass());
var story:String = MyFileByteArray.readUTFBytes(MyFileByteArray.length);

You must specify that the MIME type for the embedding is application/octet-stream, which causes the byte data to be embedded “as is”, with no interpretation. It also causes the autogenerated class to extend ByteArrayAsset rather than another asset class. For example, if you embed a PNG file without specifying this MIME type, the PNG data will be automatically transcoded into the bitmap format used by the player, and a subclass of BitmapAsset will be autogenerated to represent it. But if you specify the MIME type as application/octet-stream, then no transcoding will occur, the PNG data will be embedded as is, and the autogenerated class will extend ByteArrayAsset.

Tip 2: Binding

To make Flex datagrid completed, I would like to have the following featues. AutoCompleted Search – Locate the data I want quickly if there are too many rows in my grid. Internationalization – Handle currency, number and date format. Data Export – Output the data in csv format, so users can import to Excel. Pagination - If I give the total number of records, the subset of the data rows and the number of rows per page, the grid should be able to do pagination and fire the events when user clicks on other pages. This article I will show you how to make these happen.

AutoCompleted Search

After you obtain the resultset from the database and pass it back to Flex as list of value objects, Flex, as usual, will convert it to ArrayCollection of value objects and bind it to the datagrid. If you want to filter out the record in the datagrid, you don’t need to remove the records from the ArrayCollection. All you need is to provide the implementation of the ArrayCollection’s filterFunction. Here is the example: 

...
[Bindable]
private var _data:ArrayCollection;

private function init():void
{
	_data = new ArrayCollection(
	[
	{ "name":"One" },
	{ "name":"Two" },
	{ "name":"Three" },
	{ "name":"Four" },
	{ "name":"Five" }
	]);

	_data.filterFunction = filterFunction;
}

private function filterFunction( item:Object ):Boolean
{
	var name:String = String( item.name ).toLowerCase();
	var searchStr:String = textInput.text.toLowerCase();
	return searchStr == name.substr( 0, searchStr.length );
}
//trigger when user type in search string in the text box
private function handleChange():void
{
	//this will iterate the dataset against the filterFunction
	_data.refresh();
}
...

Here is the caveat. When using a filterFunction in an ArrayCollection, the Flash Player will always search every item. It is not an effective way. To speed it up, Hillel Coren has documented an approach that the subsequent filters will search on the filtered list instead. His approach is elegant and well-documented. Go to his article for detailed. The demo is here. I found Hillel solution quite elegant. Apart from making the search more efficient, his design also modulizes the search code so that it can be unit tested easily.

The idea is to keep the last failed search string in each record. So, if the next search string begins with last search string, it is unnecessary to check each fields in the record before you can say it won’t be matched.

//---- SearchDemo.mxml ---
private function filterFunction( item:Person ):Boolean
{
	return SearchUtils.isMatch( item, textInput.text );
}

//--- SearchUtils.as ---
public static function isMatch( item:ISearchable, searchStr:String ):Boolean
{
	if (_enableFasterSearch && !quickCheck( item, searchStr )){
		return false;
	}

	var orSearchStrs:Array = searchStr.split( "," );

	for each (var orSearchPart:String in orSearchStrs)
	{
		var andSearchStrs:Array = orSearchPart.split( " " );
		var isMatch:Boolean;

		for each (var andSearchStr:String in andSearchStrs){
			isMatch = false;
			for each (var field:String in item.getSearchFields()){
				if (item.matchesField( field, andSearchStr )){
					isMatch = true;
				}
			}
			if (!isMatch){break;}
		}	

		if (isMatch){
			item.setLastFailedSearchStr( "" );
			return true;
		}
	}
	item.setLastFailedSearchStr( searchStr );
	return false;
}
...

There are several things worth to mention here. The search routine is generic b/c it is against the item that implements the ISearchable Interface. The item implements this interface will provide the implementation of the matchesField method. So, the generic search routine can focus on providing features on top of it like ‘AND’, ‘OR’ filter and quick check algorithm.

NOTE: After you set the filterFunction, you will only get the filtered record if you iterate the ArrayCollection. If you want to get the full dataset, you need to do: 

for each (var obj:Object in arrayCollection.source)
{
     // do stuff
}

Again, thanks for Hillel’s tip.

I will talk about Data Export in Part 2 of this series.