Reporting solution!

admin — Wed, 21 Jan 2009 08:24:28 +0000

Open source reporting

My company needs a reporting engine but it doesn’t want to go for the expensive commerical ones like MicroStrategy. In fact, I don’t know why we need to pay so much because there are tools out there for FREE. As usual, I googled the Net and found out two seemingly promising open source reporting solution.

Pentaho Reporting
Jasper Reporting

Both of them are bundled with a suite of tools related to OLAP, Data Mining, ETL.. etc. To me, I just want an non-invasive reporting engine that can easily integrate into our architecture. To my dismay, I found out Pentaho doesn’t go this route. It basically gives you a reporting server configured. You could build your reports and deploy them following the manual. However, I hardly see a reporting solution that could satisfy all the business requirements without customization. All I expected from Pentaho is a jar file with documents that shows me how to use its api to generate reports in different formats and how to integrate with our database. I have attempted to look into the code and extracted the stuff I want from Pentaho. However, I found out the engine is actually not powerful. To strip out the workflow part, it is basically a simple SQL executor that later on will render the result according to the UI info embedded in the report definition. What is wrong with that?

We want to handle pagination and data streaming as our data volume is huge. In Pentaho, you need to take care these yourself. So, you write your own sql, paginate yourself, stream it yourself if the resultset is huge. Isn’t it what we are doing without it? Apart from that, each report in Pentaho needs a report definition. It supports dynamic sql via token replacement. It is primitive as I want it to support control flow because I may decide what tables to join based on the input filters.
On the UI side, Pentaho helps you to render your result into graph, table…etc. Again, I don’t like this UI solution as well. I found that JFreeChart is not as powerful as the Flex solution. I am adopting Flex and it gives me much powerful visualization tool. All I want is to ship my Flex app the data from my query’s result.

How about Japser? Pretty much the same but the good thing of Jasper is that it gives you the jar and document of how to use it instead of a reporting server like Pentaho. So, I can use it as report renderer to generate PDF and Excel like other utility libraries I use. So, what is my final solution?

I finally decide to create my own report definition that my Flex UI can take and render out the reporting interface. So, I don’t need to create form for each report. Apart from that, in my report definition, I have iBatis SQL template embedded. So, I can leverage its dynamic sql feature that supports control flow logic and the auto result class population. Yes, I still need to handle pagination and streaming myself. But, at least, it already saves up my time. The result object populated will return to Flex via AMF. So, I don’t need to marshal and de-marshal it in xml. It saves the processing time and costs less bandwidth. At the end, my solution combines the best in the market:

Powerful reporting widgets provided by Flex
Fast streaming and RPC protocol – AMF
Good dynamic sql generation and mapping tool from iBatis
Good reporting rendering tool from Jasper that helps me to do PDF and Excel generation

My solution is more flexible. As I can plugin hibernate map if I don’t want to write my own sql at all. Apart from that, no UI work is needed to deploy a new report unless my generic reporting interface is not enough.

Later, if I really need the workflow engine provided by Pentaho, I can plug it in. Again, the document provided doesn’t give us clear instruction or APIs of how to do it.

Reference

Below are references I used to build my solution:

Flexible reporting with JasperReport and iBatis
How Kodo JPA handles large result set (its optimization guide is good reference even you may not use Kodo)
Process Large Result Sets in Java Web Application
Streaming architecture

Business Intelligence – Part 1 Pentaho

admin — Thu, 26 Jun 2008 09:00:41 +0000

Getting into Business Intelligent World

When I dig deeper in business intelligence, I found out that it is a huge topic ranging from reporting to data mining. Like all the knowledge acquisition plan, I put a series of milestones for myself. If you are interested, here is my list:

Get and prepare your data

Data collection - log processing, web services (SOAP and REST), RSS, screen scraping and more.
Data preparation and crunching - ETL (Kettle)
Data storage - data warehousing

Visualize your data

Reporting and Charting (Pentaho as server and Flex as frontend)

Analysis your data

Data modeling
Data analysis (OLAP)

Get smart of your data

Collective intelligence
Data mining

Introduction of Pentaho

Firstly, I want to see whether there is any out of the box open-source solution that captures what I am trying to do here. If so, I can reach my goal much faster. Yes. It has to be open-source b/c I don't have $$ and I don't want to be just a user. After doing my homework a bit, I found out an open-source BI tool named Pentaho that looks pretty solid. So, I decide to dive deep to this. Like all the tools I mess around with, I want to integrate Pentaho as library. However, I don't find anything on the Net that shows me how to do it. I am looking into its download and checkout its pentaho-sample project. What it shows me is how to use their tools to create a report on their systems using their UI. I definitely need more!

After few days of efforts, I managed to pull out all the unnecessary dependencies from pentaho. The heart of Pentaho is its xaction interpreter. The approach Pentaho uses is to write adapters, plug into its framework and use xaction to wire them up in a workflow fashion. In fact, most of its functionalities come from other open-source projects like quartz for scheduling, shark for workflow engine and jfreereport for reporting. I don't think their xaction is clean but I do like their architectural approach.