To many of Flash/Flex programmers, it could be a bad news that Steve Job openly banned Flash on his devices like iPad. His message clearly stated that HTML5 can be used to replace the rich experience of Flash and it will be our future. He may be right about the cons of Flash. Nothing is perfect. However, I am surprised that he took a step further to ban Flash totally. Why couldn't he simply provide an option for users to turn it off if they want? I really doubt about his intent. Whenever I see something like that, it only reminds me what Microsoft did in the old days.
Recently, I need to pick up Perl for some of my projects. After going through some websites and books, I start seeing why some of the tasks could be done much easier in Perl than in Java, the language that I am quite familiar with. Since Perl is loose in data type, not OO-enforced, non-explicit signature plus its syntax is full of symbols, some Perl programmers given the freedom and power without following the best practices could write very illegible code. Although it runs, it could be very hard to maintain. For those who feel the pains, you may like this post . However, I don't want to go extreme on this. I do see its power in text processing and system integration. After taking a look at the extensive CPAN library, I decide to give it a fair trial. Not only that, I will write a series of posts to help you to shorten your learning curve as well. This post I will put my head on Perl data type and its usages.
As I mentioned in my last article, I was getting excited about the potential of Hive. Today, I decide to start my journey to learn this. I found a great introductory video that gives you a nice warm-up of using Hive (A basic knowledge of how hadoop and mapreduce work would be helpful for you to digest the material inside).
Below are some highlights from this video
Hive is an SQL interface built on top of Hadoop. It supports Web access and JDBC. I am amazed how close the SQL syntax like the regular SQL for RDBMS. Below are some SQLs used in this tutorial.
//———- Set up your tables in HIVE —————–
SHOW TABLES;
CREATE TABLE shakespeare (freq INT, word STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t’ STORED AS TEXTFILE;
DESCRIBE shakespeare;
//———- Load data into Hive table from Hadoop HDFS ——————-
LOAD DATA INPATH “shakespeare_freq” INTO TABLE shakespeare;
//———- Query against the data using hive sql interface ————–
select * from shakespeare limit 10;
select * from sakespeare where freq > 100 sort by freq asc limit 10;
select freq, count(1) as f2 from shakespeare group by freq sort by f2 desc limit 10;
//show me the plan
explain select freq, count(1) as f2 from shakespeare group by freq sort by f2 desc limit 10;
//———- Create a merge table and populate it using dataset joining by 2 different tables
insert overwrite table merged select s.word, s.freq, k.freq from shakespeare s join kjv k on (s.word = k.word);
//———- Query the merge table ———————
select word, shake_f, kjv_f, (shake_f+kjv_f) as ss from merged sort by ss limit 20;
To prepare the data for Hive to load in, the demo uses another mapreduce job to achieve. Remember to delete the log before doing Hive table load.
hadoop jar $HADOOP_HOME/hadoop-*-examples.jar grep input shakespeare_freq ‘\w+’
//remove the mapreduce job log
hadoop fs -rmr shakespeare_freq/_logs
Often time, large scale data processing system always IO bound. So for mapreduce job, your mapper is always waiting for data to load from disk. Hadoop mitigates the problem via during parallel load from lots of hard drives. However, a single hard drive is still max out at 75MB/s read as physical limit and nothing we can do about this. In order to achieve good speed, the key is to eliminate # of hadoop pass
Since Hive is on top of Hadoop’s HDFS, it will have the same restrictions as it. So, you cannot do UPDATE, DELETE and INSERT records as regular RDMS. However, you can do bulk load to add more new files (data) to the table and you can do delete a file from Hive.
Hive needs to store metadata of the tables out from the HDFS. You can use regular rdms to achieve the job. But when you start Hive locally, it will seek for the local metastore. So, in distributed environment, you may need to centralize the metastore in a remote location. There is wiki on the Hive site that documents how to set it up.
I ever worked for a display ad network company that collects over 400 million of impression/ click logs per day. With this amount of data, my ex-company bought a supercomputer and cross their fingers that it can handle the grow in both volume and analytic demand of the data. It is obviously not a scalable solution. However, what is the best solution?
Although I haven’t worked for this company anymore, it is still an interesting problem to solve. I have a great friend who proposed a shared nothing solution for this company. The solution is to partition the data across a set of Postgresql databases and put Greenplum on top of them to parallelize the query —there is no disk-level sharing or contention to be concerned with (i.e. it is a ‘shared-nothing’ architecture). I like this approach. The only thing is that Greenplum is not free and it may be difficult for a startup to face this upfront cost. Apart from that, this setting requires all the databases are running on the same network that hindered us to move this in the elastic cloud like Amazon EC2.
Later on, I joined a great company in the same industry that seeks for a solution in the cloud to host its data warehouse. So, I got a chance to revisit this problem. During the research, I came across an interesting technology – column-based database (eg. infobright and lucid db). The idea of column-based data store is that traditional database stores and fetches data in row from data files into the memory. It is inefficient if your query only requires few columns for computation. So, column-based data stores your data in column with effective compression algorithm due to all values in it has the same data type. This solution is great but it doesn’t do MPP (ie. massive parallel processing) and it is also not ready for cloud yet.
Here comes another solution. That is Hive on top of Hadoop on top of Amazon cloud. It is an interesting idea. Check out this video to learn about this.
If you are not sure what Hadoop is and want to get some warm up in massive computing. I suggest you go through the following 5 excellent Google lectures.
I want to summarize some new and interesting Java 5 features in this article and how they change the way I code.
Enum
I use int constants to make my life easier b/c it can avoid typo. However, it has several drawbacks:
Java doesn’t provide namespace for int enum groups. I can either prefix my constant like ABC_ or using inner interfaces to organize it.
It is compile-time constants. So you need to recompile once changed.
No easy way to translate int enum constants into printable string during debugging.
You cannot iterate over all the int enum easily.
You need a way to validate the enum is an valid int
Use new enum type in Java 5:
public enum Apple {FUJI, PIPPIN, GRANNY_SMITH}
Enum is full-fledged final class that export one instance for each enumeration constant via a public static final field.
Namespace is provided via the enum type name.
You can reorder and add the enumeration constant without recompiling its client.
You can translate enum into printable strings via toString() method.
Enum type provides you values() method to iterate your enumeration constants (based on declaration order).
Type-checking can be used for the validation check
You can associate data with enum constant
Enum is immutable, serializable and comparable.
EnumSet
If elements of an enumerated types are used primarily in sets, it is traditional to use the int enum pattern, assigning a different power of 2 to each constant like READ = 1 << 2, WRITE = 1 <<1, EXECUTE = 1 << 0 to represent permissions per each entity in Unix. This representation lets you use the bitwise OR operation to combine several constants into a set, known as a bit field. The bit field representation also lets you perform set operations such as union and intersection efficiently using bitwise arithmetic. But bit fields have all the disadvantages of int enum mentioned above.
Now, java.util package provides the EnumSet to efficient represent sets of value drawn from single enum type. This class implements Set interface and internally use bit vector to represent set of values. For example, if you enum types has 64 values, the entire EnumSet can be represented as a single long, so its performance is comparable to the bit field.
The EnumSet class provides three benefits a normal set does not:
Various creation methods that simplify the construction of a set based on an Enumeration
Guaranteed ordering of the elements in the set based on their order in the enumeration constants are declared
Performance and memory benefits not nearly possible with a regular set implementation
Annotation
An annotation is a new language feature introduced in J2SE 5.0. Simply put, annotations allow developers to mark classes, methods, and members with secondary information that is not part of the operating code.You can see annotation is a way to extend Java language.
Before annotation from Java 5, you may use naming patterns to indicate that some program elements like method demanded special treatment by a tool or a framework. Like JUnit required its users to name the test methods with the pattern like testXXX(). It works but with some big disadvantages:
Typo problem
It doesn’t provide a way to associate parameter values with program elements.
Annotation can solve this problem. To use it, you can:
Create you own marker annotation (@interface is the keyword) or parametized annotation. You can annotate the annotation (ie. meta-annotation). Example: @Retention and @Target. And marker annotation has no parameter associated with it.
Annotate the program elements
Write processor to handle your annotated code. Generally, annotations never change the semantics of the annotated code, but enable it for special treatment by tools. Now, the metadata of Method carries additional info for your job. You can use Method’s isAnnotationPresent() to check if a method is annotated by certain annotation type. If you annotation carried parameter, you can use Method’s getAnnotation() to get the Annotaton object and use value() to obtain the parameter.
Nowadays there are many tools available on the Net ranging from IM to cloud computing that certainly lowers the barrier for entrepreneurs like us. Today, I am going to list out the tools that helps me to run my company:
Set up virtual office
Skype – Save you from long distance bill
FREE. If you pay a rate, it will let you connect to phone line.
When I am tired of typing, it will switch to this.
If you like face to face conversion, I would use iChat on Mac. All you need is an AOL account.
In case you don’t have internet access or your laptop is not next to you. This is a good tool because it gives you a dedicated line to dail in. However, the number is not TOLL-FREE.
Google calendar (it can sync with my iCal on Mac now. If you use Outlook, you need to install a plugin to do the calendar sync). Follow this guide to set it up.
If you are using DSL that assigns you IP address dynamically, it is quite a headache to keep track of it. You can obtain a domain name from DynDNS to abstract you from the IP address.
By default, VNC server will be listened on port 5900. If you want to do remote desktop outside your subnet, make sure your DSL router open a port for that and forward the request to your machine.
Here is a web-based VNC viewer. With that, you can do remote desktop anywhere. Just key in your dynDNS domain and you are done.
If you want to make this access security, you can password protected your box via configure the VNS Server.
There are people who uses LogMeIn service that provides more remote secure featuers. However, it is NOT FREE. To me, VNC Server is good enough solution.
Build your virtual dev team
Tracs – combine wiki, ticket system, project planning in one
FREE
A bit complicated to set it up in web hosting company
It integrates with Subversion as well
Bugzilla is pretty good for issue tracking as well
Recently, I am trying to build an interactive reporting tool that needs to deal with lots of data. The data is not dynamic because it is basically data from historical performance log files. However, the volume of the data is large (over few millions of rows) and I still want my clients to interact with large amount of data in ease. With this, I am looking into Adobe AIR as I heard that it comes with in-memory database “SQLite“. I believe it should have better performance than web-based application because data is local and SQLite is lightweight and fast. Apart from that, SQLite supports parameterized query, strongly-typed result, asynchronous/ synchonous processing, indexing, view, trigger,transaction and most of SQL92. On top of that, it is small footprint, cross-platform and open source. The tradeoff for SQLite is its weak support in concurrency because it is using table exclusive lock. However, it is totally fine for desktop application because it normally only serves one user. For more info of SQLite, check out my notes below.
Update
SQLite 3 is released that addressed some of its issues in version 2.
BLOB support
Fulltext searching
Connection shared between threads
Improve concurrency
However, it still doesn’t support writeable view, nested transaction and foreign key.
Presentation
Here is a nice presentation from Paul Roberson, look at it first.
Note from the video:
Warm up with general SQL Tips (Join favors subquery, Avoid IN, Avoid LIKE, Specify columns name in select and insert, Avoid unnecessary join)
AIR SQL connection can connect up to 10 databases at a time, you can use qualifier for your tables.
Don’t reuse the same SQL Statement for different prepared statements.
Use transaction to do batch insert/ update/ delete operations (48x faster!)
Index columns in WHERE clause, use together index together
Create table structure before you add data because internally SQLite…?
Handle large resultset in parts for perceived performance gain (detailed)
There are several things I want to find out:
Can SQLite handles large dataset?
Yes. According to spec, it can handle terabyte of data.
Does SQLite support pagination?
Yes. Look at SQL Statement object.
How SQLite synchronize with the updated data from the remote database?
Strategies: overwrite vs delta (timestamp, field by field comparison, dirty flag)
Live cycle data service has built-in SQLite synchronization support including offline caching and conflict management.
Below are some SQLite tips and practices I obtained from different sources:
A big advantage of sqlite above a flat file is the possibility to index your data.
Using parameterize query protects against sql injection, and makes the ‘ problems go away. It is also much faster because sqlite can reuse the execution plan of statements when you use parameters.
Make sure to import the records in a transaction so that it doesn’t spend a lot of time creating indexes until everything is imported.
The SQLite documentation states that SQLite databases can be terabytes in size, and that the primary limitation of SQLite is concurrency (many users at the same time).
“The SQLite database is pretty damn fast. I was getting near instantaneous searching with databases that were ~100,000 records. Somewhere around 800,000 – 1,000,000 records you start losing performance, waiting a few seconds for a search” – by Daniel
ESB = Enterprise Service Bus. The definition is flexible, but in general it’s a conduit for messages of multiple, different formats, between application endpoints, over more than one protocol.
Mule vs ServiceMix
Compared to Mule, the major difference for ServiceMix is its architectural design, which is fundamentally based on the Java Business Integration (JBI) standard. Mule provides a JBI binding so that Mule components can interact with JBI containers, including the ServiceMix JBI container. However, the internal Mule APIs are not based on the JBI standard.
JBI uses a notion of Message Exchanges and a Normalized Message to communicate between components, where as Mule use a “POJO / Endpoint” architecture.
Mule doesn’t require your service to implement any Mule interface. The components are wired up thru the mule-config.xml file.дивани
The video above can give you a good taste of Mule. But the example runs as standalone app. If you want to use Mule in your web application. I found this distributed raman amplifiervideo a good starting point.
What is EDA?
Asynchronous messaging lets two or more applications send data to each other without having to wait for receipt confirmation. The infrastructure guarantees message delivery, even if the receiving application isn’t currently running or the network connection is interrupted. It sounds simple enough, but asynchronous messaging demands a new way of thinking about system architecture.
Caveat of asynchronous messaging
The convenience of sending guaranteed messages literally around the world to applications running on different platforms or technologies is a huge benefit. What’s the catch? The messaging system can guarantee that the message will be delivered, but it does not guarantee when it will be delivered.
Worse yet, if an application sends two messages, the messaging system doesn’t guarantee that they’ll arrive in the same order.
Because destinations are unidirectional, the application receives responses on a different destination than the one on which it publishes requests. This means that the responses can arrive in a different order than the requests were sent. As a result, the application must explicitly correlate incoming response messages to the requests. In the asynchronous world, correlation is such a common need that the JMS API includes the methods getJMSCorrelationID and setJMSCorrelationID in the message interface.
How do we have sender and receiver in a transaction?
There are some interesting tips I found during the time I work on Flex Programming. I will cover Embedding, Binding, Event Handling, Function Pointer, Mixin and more. I hope these tips will make your life easier when you work on Flex.
To make Flex datagrid completed, I would like to have the following featues. AutoCompleted Search – Locate the data I want quickly if there are too many rows in my grid. Internationalization – Handle currency, number and date format. Data Export – Output the data in csv format, so users can import to Excel. Pagination - If I give the total number of records, the subset of the data rows and the number of rows per page, the grid should be able to do pagination and fire the events when user clicks on other pages. This article I will show you how to make these happen. » Read more…
To many of Flash/Flex programmers, it could be a bad news that Steve Job openly banned Flash on his devices like iPad. His message clearly stated that HTML5 can be used to replace the rich experience of Flash and it will be our future. He may be right about the cons of Flash. Nothing is perfect. However, I am surprised that he took a step further to ban Flash totally. Why couldn't he simply provide an option for users to turn it off if they want? I really doubt about his intent. Whenever I see something like that, it only reminds me what Microsoft did in the old days.
There are some interesting tips I found during the time I work on Flex Programming. I will cover Embedding, Binding, Event Handling, Function Pointer, Mixin and more. I hope these tips will make your life easier when you work on Flex.
To make Flex datagrid completed, I would like to have the following featues. AutoCompleted Search – Locate the data I want quickly if there are too many rows in my grid. Internationalization – Handle currency, number and date format. Data Export – Output the data in csv format, so users can import to Excel. Pagination - If I give the total number of records, the subset of the data rows and the number of rows per page, the grid should be able to do pagination and fire the events when user clicks on other pages. This article I will show you how to make these happen.