My company needs a reporting engine but it doesn’t want to go for the expensive commerical ones like MicroStrategy. In fact, I don’t know why we need to pay so much because there are tools out there for FREE. As usual, I googled the Net and found out two seemingly promising open source reporting solution.

1. Pentaho Reporting
2. Jasper Reporting

Both of them are bundled with a suite of tools related to OLAP, Data Mining, ETL.. etc. To me, I just want an non-invasive reporting engine that can easily integrate into our architecture. To my dismay, I found out Pentaho doesn’t go this route. It basically gives you a reporting server configured. You could build your reports and deploy them following the manual. However, I hardly see a reporting solution that could satisfy all the business requirements without customization. All I expected from Pentaho is a jar file with documents that shows me how to use its api to generate reports in different formats and how to integrate with our database. I have attempted to look into the code and extracted the stuff I want from Pentaho. However, I found out the engine is actually not powerful. To strip out the workflow part, it is basically a simple SQL executor that later on will render the result according to the UI info embedded in the report definition. What is wrong with that?

1. We want to handle pagination and data streaming as our data volume is huge. In Pentaho, you need to take care these yourself. So, you write your own sql, paginate yourself, stream it yourself if the resultset is huge. Isn’t it what we are doing without it? Apart from that, each report in Pentaho needs a report definition. It supports dynamic sql via token replacement. It is primitive as I want it to support control flow because I may decide what tables to join based on the input filters.
2. On the UI side, Pentaho helps you to render your result into graph, table…etc. Again, I don’t like this UI solution as well. I found that JFreeChart is not as powerful as the Flex solution. I am adopting Flex and it gives me much powerful visualization tool. All I want is to ship my Flex app the data from my query’s result.

How about Japser? Pretty much the same but the good thing of Jasper is that it gives you the jar and document of how to use it instead of a reporting server like Pentaho. So, I can use it as report renderer to generate PDF and Excel like other utility libraries I use. So, what is my final solution?

I finally decide to create my own report definition that my Flex UI can take and render out the reporting interface. So, I don’t need to create form for each report. Apart from that, in my report definition, I have iBatis SQL template embedded. So, I can leverage its dynamic sql feature that supports control flow logic and the auto result class population. Yes, I still need to handle pagination and streaming myself. But, at least, it already saves up my time. The result object populated will return to Flex via AMF. So, I don’t need to marshal and de-marshal it in xml. It saves the processing time and costs less bandwidth. At the end, my solution combines the best in the market:

1. Powerful reporting widgets provided by Flex
2. Fast streaming and RPC protocol – AMF
3. Good dynamic sql generation and mapping tool from iBatis
4. Good reporting rendering tool from Jasper that helps me to do PDF and Excel generation

My solution is more flexible. As I can plugin hibernate map if I don’t want to write my own sql at all. Apart from that, no UI work is needed to deploy a new report unless my generic reporting interface is not enough.

Later, if I really need the workflow engine provided by Pentaho, I can plug it in. Again, the document provided doesn’t give us clear instruction or APIs of how to do it.

Reference

Below are references I used to build my solution:

1. Flexible reporting with JasperReport and iBatis
2. How Kodo JPA handles large result set (its optimization guide is good reference even you may not use Kodo)
3. Process Large Result Sets in Java Web Application
4. Streaming architecture

SELECT name AS Name, category AS Category,
IF(winter>500, "Sells", "Slow")
AS Trend FROM sales;

http://www.roseindia.net/mysql/mysql5/flow-control-constructs.shtml

Rename tables

RENAME TABLE table1 TO table2;

Add a user

connect mysql
INSERT INTO user VALUES(‘%’,'monty’,PASSWORD(‘some_pass’);
FLUSH PRIVILEGES;

Monitoring

The number of mysql processes running at any given time should be monitored. I do this manually from time to time, using SHOW PROCESSLIST in mysql. I had a problem in the past with processes never expiring and the number of connections maxed out. Set timeout to resolve this. Put report id and user id in the comment before running the sql. So, you can trace back what the report the sql for and who issued it when you need to kill it or analyze how to fix it.

Create a table from other table

CREATE TABLE emps2 as select * from emps;

One downside to this approach on both MySQL and Oracle is that it doesn’t make an exact replica of the table, for example any index against the table will not be receated for the new table.

CREATE TABLE emp5 like emp;

The downside of this method is that the data isn’t copied across in the way it is using an SQL statement in the create. But it’s easy to do that using a command like follows.

INSERT INTO emps5 select * from emps;