Recently, I am trying to build an interactive reporting tool that needs to deal with lots of data. The data is not dynamic because it is basically data from historical performance log files. However, the volume of the data is large (over few millions of rows) and I still want my clients to interact with large amount of data in ease. With this, I am looking into Adobe AIR as I heard that it comes with in-memory database “SQLite“. I believe it should have better performance than web-based application because data is local and SQLite is lightweight and fast. Apart from that, SQLite supports parameterized query, strongly-typed result, asynchronous/ synchonous processing, indexing, view, trigger,transaction and most of SQL92. On top of that, it is small footprint, cross-platform and open source. The tradeoff for SQLite is its weak support in concurrency because it is using table exclusive lock. However, it is totally fine for desktop application because it normally only serves one user. For more info of SQLite, check out my notes below.
Update
SQLite 3 is released that addressed some of its issues in version 2.
BLOB support
Fulltext searching
Connection shared between threads
Improve concurrency
However, it still doesn’t support writeable view, nested transaction and foreign key.
Presentation
Here is a nice presentation from Paul Roberson, look at it first.
Note from the video:
Warm up with general SQL Tips (Join favors subquery, Avoid IN, Avoid LIKE, Specify columns name in select and insert, Avoid unnecessary join)
AIR SQL connection can connect up to 10 databases at a time, you can use qualifier for your tables.
Don’t reuse the same SQL Statement for different prepared statements.
Use transaction to do batch insert/ update/ delete operations (48x faster!)
Index columns in WHERE clause, use together index together
Create table structure before you add data because internally SQLite…?
Handle large resultset in parts for perceived performance gain (detailed)
There are several things I want to find out:
Can SQLite handles large dataset?
Yes. According to spec, it can handle terabyte of data.
Does SQLite support pagination?
Yes. Look at SQL Statement object.
How SQLite synchronize with the updated data from the remote database?
Strategies: overwrite vs delta (timestamp, field by field comparison, dirty flag)
Live cycle data service has built-in SQLite synchronization support including offline caching and conflict management.
Below are some SQLite tips and practices I obtained from different sources:
A big advantage of sqlite above a flat file is the possibility to index your data.
Using parameterize query protects against sql injection, and makes the ‘ problems go away. It is also much faster because sqlite can reuse the execution plan of statements when you use parameters.
Make sure to import the records in a transaction so that it doesn’t spend a lot of time creating indexes until everything is imported.
The SQLite documentation states that SQLite databases can be terabytes in size, and that the primary limitation of SQLite is concurrency (many users at the same time).
“The SQLite database is pretty damn fast. I was getting near instantaneous searching with databases that were ~100,000 records. Somewhere around 800,000 – 1,000,000 records you start losing performance, waiting a few seconds for a search” - by Daniel
ESB = Enterprise Service Bus. The definition is flexible, but in general it’s a conduit for messages of multiple, different formats, between application endpoints, over more than one protocol.
Mule vs ServiceMix
Compared to Mule, the major difference for ServiceMix is its architectural design, which is fundamentally based on the Java Business Integration (JBI) standard. Mule provides a JBI binding so that Mule components can interact with JBI containers, including the ServiceMix JBI container. However, the internal Mule APIs are not based on the JBI standard.
JBI uses a notion of Message Exchanges and a Normalized Message to communicate between components, where as Mule use a “POJO / Endpoint” architecture.
Mule doesn’t require your service to implement any Mule interface. The components are wired up thru the mule-config.xml file.дивани
The video above can give you a good taste of Mule. But the example runs as standalone app. If you want to use Mule in your web application. I found this distributed raman amplifiervideo a good starting point.
What is EDA?
Asynchronous messaging lets two or more applications send data to each other without having to wait for receipt confirmation. The infrastructure guarantees message delivery, even if the receiving application isn’t currently running or the network connection is interrupted. It sounds simple enough, but asynchronous messaging demands a new way of thinking about system architecture.
Caveat of asynchronous messaging
The convenience of sending guaranteed messages literally around the world to applications running on different platforms or technologies is a huge benefit. What’s the catch? The messaging system can guarantee that the message will be delivered, but it does not guarantee when it will be delivered.
Worse yet, if an application sends two messages, the messaging system doesn’t guarantee that they’ll arrive in the same order.
Because destinations are unidirectional, the application receives responses on a different destination than the one on which it publishes requests. This means that the responses can arrive in a different order than the requests were sent. As a result, the application must explicitly correlate incoming response messages to the requests. In the asynchronous world, correlation is such a common need that the JMS API includes the methods getJMSCorrelationID and setJMSCorrelationID in the message interface.
How do we have sender and receiver in a transaction?
There are some interesting tips I found during the time I work on Flex Programming. I will cover Embedding, Binding, Event Handling, Function Pointer, Mixin and more. I hope these tips will make your life easier when you work on Flex.
To make Flex datagrid completed, I would like to have the following featues. AutoCompleted Search - Locate the data I want quickly if there are too many rows in my grid. Internationalization - Handle currency, number and date format. Data Export - Output the data in csv format, so users can import to Excel. Pagination - If I give the total number of records, the subset of the data rows and the number of rows per page, the grid should be able to do pagination and fire the events when user clicks on other pages. This article I will show you how to make these happen. Read more »
Share and Enjoy:These icons link to social bookmarking sites where readers can share and discover new web pages.
In our daily life, we deal with lots of data. The data normally is stored in text format for the ease of human to read. With the large amount of data we have, we need ways to deal with it. There are several things we frequently do on the data: Search, Filter, Sort and Analysis. In Linux, there are some powerful commands that I can use: cat, grep, find, sort, unique and etc. I found those commands quite powerful. So, I decide to put these down as my reference. This tutorial I will go over the basic text processing commands and how we use them together to achieve the tasks we often encounter in our workplace.
AOP is a way to modularize cross-cutting concerns. Ok, what does “modularize” really mean? Modularization is the encapsulation of a unit of functionality. It is exactly what “Class” is doing in OO world. How about “cross-cutting concerns”? Basically it means any functionalities that span multiple modules/ classes. They include TransactionManagement, Security, Caching, Performance Monitoring and etc. To understand how AOP works, we first look at the common terms in this area:
Join point - An identifiable point in the execution of a program like method invocation, exception thrown.
Pointcut - Program construct that selects join points and collects context at those points. AspectJ has a rich pointcut expression language!
Advice - Code to be executed at a join point that has been selected by a pointcut.
To me, I found it easier to understand these terms if I consider join point as event generated point in code, pointcut as a way to define what events to be captured and advice as event handler.
AOP is indeed a powerful way to factor out system or infrasturcture-related code from the business oriented code. Typically, we use it to take care of transaction, security and profiling aspects. But it doesn’t stop you putting creativity in this domain. With a bit more creativity, you can also do the following::
Exception translation - checked to runtime
Catch ConcurrencyFailureExceptions and transparently retry if an idempotent operation fails with, for example, a deadlock loser exception.
I always wonder how my Flex application displayed on the Flash Player in browser. Why decompile Flex SWF will give me 2 frames movie? What is SystemManager and how can I get a handle of it? Many of these kind of questions are at the lower level. The level that makes Flex application possible. Normally, we don’t need to dive into this to write a Flex application. However, it would make me feel more comfortable to understand this before I advocate Flex as the main part of our company’s UI strategy.
To understand how your Flex application got loaded and display on Flash Player, you can read this great article. I am going to put down the sequences in steps below:
Flex is a 2-frame movie
The first frame of a Flex SWF contains the SystemManager, the Preloader, the DownloadProgressBar and some glue helper classes. Remember, the Preloader is what creates the DownloadProgressBar control which displays the progress of a Flex application downloading and being initialized. The second frame of a Flex SWF contains the rest of the Flex framework code, your application code and all of your application assets like embedded fonts, images, etc.
Hibernate is great. However, I don’t see it fits all the data access requirements. At its core, it is an ORM tool that helps you to map your object model to relational model. If you have full control of your relational model and perform lots of CRUD operations, it is certainly a great tool for you. Its transparent persistence, 2 level caching, dirty checking, lazy/ eager data retrieval and sql generation indeed can save us lots of development time. However, one tool doesn’t fit all !! Why not?
In my current company, I have created a reporting tool that interfaces with dimensional model in data warehouse. In this setting, you will deal with star schema with denormalized dimensions. Often time, I need to tune the query performance via looking into explain plan. Without full control of SQL, my job will be hard to achieve. Apart from that, reporting tool often issues read-only set based queries to the data warehouse. The resultset returned doesn’t fit into my OO model at all. Again, Hibernate just doesn’t fit in. People in my company argue that I should use named query in Hibernate for the sake of sticking with the standard. I am like ok, whatever… I have known a tool called iBatis that I can achieve my job cleanly. Why the hell I would have motivation to try named query that basically a way to by-pass the ORM model to query database. What benefit I will get from this? The cache? We are using ETL to update our fact and dimension in the data warehouse, not by my reporting app. Unless the ETL throws me an event when the update cycle is finished so I can flush my cache, I simply don’t think it gives me lots of help.
Anyway, it is just my own little perspective. You don’t have to agree with me. The point here is not that I don’t like Hibernate but I don’t like to be pushed to use it only because it is “standard” to someone. If Hibernate could help me to construct my sql based on user input, stream my result directly to my presentation layer, populate my model automatically based on mapping I provided, detect data warehouse changes and take care my cache, then I am more happy to adopt it in my reporting app. Otherwise, I would not be eager to dump my iBatis DAO layer unless I get no choice under the political game.
Quite often in threaded applications we have a producer-consumer situation where some threads want to add jobs onto a queue, and some other worker threads want to remove jobs from the queue and then execute them. It is quite useful in such circumstances to write a queue which blocks on pop when there is nothing on the queue. Otherwise the consumers would have to poll, and polling is not very good because it wastes CPU cycles. When clients attempt to retrieve elements but none are available, a BlockingQueue encapsulates code that waits until an element becomes available.
Usage
The server runs infinitely, accepting requests from a client. Clients can send requests asynchronously and rapidly—the server only adds a client request to the server’s queue (ie. blocking queue), then returns immediately to the client. A separate server thread loops indefinitely, asking the BlockingQueue to return when a new request becomes available. Once a request is available, it’s removed from the queue. The server then handles the request.
ReentrantLock supports all of the lock-acquisition modes defined by Lock, providing more flexibility for dealing with lock unavailability than does synchronized. Intrinsic locking (ie. synchronized) works fine in most situations but has some functional limitations: It is not possible to interrupt a thread waiting to acquire a lock. If you want the lock, you need to prepare to wait for it forever. Using ReentrantLock, you have more flexibility and scalability:
Timed and Polled lock acquisition - Use tryLock(). One form of this method returns immediately if the lock is already held and the other can wait for some period of time for the lock to become available before giving up. In both cases, we could effectively loop and retry the tryLock() until it succeeds.
Interruptable lock acquisition - Use lockInterruptabily() allows locking to be used within cancellable activities
Find out whether your website can support the expected # of concurrent users.
At what load does the app break?
To do that, you normally follow these steps:
Identifying the primary user path
Identifying the expected # of concurrent users. (Both now and future)
Set up virtual users to hit the app (load generation capability is the key factor to pick the right tool. You don’t want too much hardware investment to generate the load you want, right? )
Run the test
Analyze the result (throughput under load, avg response time under load)
There are some interesting tips I found during the time I work on Flex Programming. I will cover Embedding, Binding, Event Handling, Function Pointer, Mixin and more. I hope these tips will make your life easier when you work on Flex.
To make Flex datagrid completed, I would like to have the following featues. AutoCompleted Search - Locate the data I want quickly if there are too many rows in my grid. Internationalization - Handle currency, number and date format. Data Export - Output the data in csv format, so users can import to Excel. Pagination - If I give the total number of records, the subset of the data rows and the number of rows per page, the grid should be able to do pagination and fire the events when user clicks on other pages. This article I will show you how to make these happen. 
In our daily life, we deal with lots of data. The data normally is stored in text format for the ease of human to read. With the large amount of data we have, we need ways to deal with it. There are several things we frequently do on the data: Search, Filter, Sort and Analysis. In Linux, there are some powerful commands that I can use: cat, grep, find, sort, unique and etc. I found those commands quite powerful. So, I decide to put these down as my reference. This tutorial I will go over the basic text processing commands and how we use them together to achieve the tasks we often encounter in our workplace. 
)