As I mentioned in my last article,  I was getting excited about the potential of Hive. Today, I decide to start my journey to learn this. I found a great introductory video that gives you a nice warm-up of using Hive (A basic knowledge of how hadoop and mapreduce work would be helpful for you to digest the material inside).

Below are some highlights from this video

Hive is an SQL interface built on top of Hadoop. It supports Web access and JDBC. I am amazed how close the SQL syntax like the regular SQL for RDBMS. Below are some SQLs used in this tutorial.

//———- Set up your tables in HIVE —————–
SHOW TABLES;

CREATE TABLE shakespeare (freq INT, word STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t’ STORED AS TEXTFILE;

DESCRIBE shakespeare;

//———- Load data into Hive table from Hadoop HDFS ——————-
LOAD DATA INPATH “shakespeare_freq” INTO TABLE shakespeare;

//———- Query against the data using hive sql interface ————–
select * from shakespeare limit 10;
select * from sakespeare where freq > 100 sort by freq asc limit 10;
select freq, count(1) as f2 from shakespeare group by freq sort by f2 desc limit 10;

//show me the plan
explain select freq, count(1) as f2 from shakespeare group by freq sort by f2 desc limit 10;

//———- Create a merge table and populate it using dataset joining by 2 different tables
insert overwrite table merged select s.word, s.freq, k.freq from shakespeare s join kjv k on (s.word = k.word);

//———- Query the merge table ———————
select word, shake_f, kjv_f, (shake_f+kjv_f) as ss from merged sort by ss limit 20;

To prepare the data for Hive to load in, the demo uses another mapreduce job to achieve. Remember to delete the log before doing Hive table load.

hadoop jar $HADOOP_HOME/hadoop-*-examples.jar grep input shakespeare_freq ‘\w+’

//remove the mapreduce job log
hadoop fs -rmr shakespeare_freq/_logs

Often time, large scale data processing system always IO bound. So for mapreduce job, your mapper is always waiting for data to load from disk. Hadoop mitigates the problem via during parallel load from lots of hard drives. However, a single hard drive is still max out at 75MB/s read as physical limit and nothing we can do about this. In order to achieve good speed, the key is to eliminate # of hadoop pass

Since Hive is on top of Hadoop’s HDFS, it will have the same restrictions as it. So, you cannot do UPDATE, DELETE and INSERT records as regular RDMS. However, you can do bulk load to add more new files (data) to the table and you can do delete a file from Hive.

Hive needs to store metadata of the tables out from the HDFS. You can use regular rdms to achieve the job. But when you start Hive locally, it will seek for the local metastore. So, in distributed environment, you may need to centralize the metastore in a remote location. There is wiki on the Hive site that documents how to set it up.

See Hive in Action

Cloudera Hadoop Training: Hive Tutorial Screencast from Cloudera on Vimeo.

Other projects similar to Hadoop

• Parallel databases: Gama, Bubba, Volcano
• Google: Sawzall
• Yahoo: Pig
• IBM Research: JAQL
• Microsoft: DryadLINQ, SCOPE
• Greenplum: YAML MapReduce
• Aster Data: In-database MapReduce
• Business.com: CloudBase

Data warehouses differ from operational databases in the way they are designed; they are optimized for efficient querying and not for updating. Data warehouses provide a read-only version of the data in the operational databases, which is optimized for querying. The kind of modeling most commonly used in warehouse design is called dimensional modeling, and the schemas produced are known as star schemas. In dimensional modeling, a database is organized around a small number of fact tables. Each row in a fact table is a single measurable event: a single sale, a single hit to a web page, etc. Example: Order-Management Dimension Schema

The key benefits of data warehouse are simplication and consolidation of data. It normally gathers data from different operational databases into single dimensional model for reporting and analysis purpose. On the other hand, dimensional modeling offers a chance to reduce the level of complexity in your database. By reducing complex chains of tables into dimension tables, the schema becomes smaller and performance tends to improve. The approaches we take to reduce the complexity are (1) We try to model one aspect of the system for each DM schema. (2) We can denormalize the schema to reduce number of joins. ETL Process Once you have a data schema for your warehouse, you'll need to fill it with data. This process is known as extract, transform, and load, or ETL for short. The first step, extraction, is simply the process of selecting all the data of interest from the operational database. Then the data must be transformed into the format needed by the warehouse. This could be as simple as renaming some of the fields or as complex as cleaning dirty data and computing new fields. Finally the data must be loaded into the data warehouse. There are some areas you need to pay attention when you perform the ETL:

1. During extraction, you will put a lot of strains to the operational database. To deal with this problem we can replicate a low-cost copy of the operational database on the warehouse machine before doing extraction. The SQL output of the extraction process can be a CSV file.
2. Transformation can be computing summary data, converting postal code into geo-code (ie. lat and long) that powers"within X miles" queries. You can use Perl to do this job. The output of transformation may be another CSV file.
3. Finally, you load the data into CSV into dimensional model. To speed up the load, in MySQL, we first disable indexes with ALTER TABLE foo DISABLE KEYS, and after the load, we re-enable them with ALTER TABLE foo ENABLE KEYS. Each table needs to be cleared before loading via TRUNCATE command.
4. You may be wondering what happens to clients using the warehouse while an ETL process is running. In our case, nothing at all! This magic is achieved by actually having two warehouse databases, one in use and the other free for loading. All the data goes into the loading database, and when it's full we swap it into place with RENAME.This produces an atomic switch of all tables in the loading database with the tables in the live database. It will wait for any running queries in the warehouse to finish before performing the swap, which is exactly what we want.

Quick Tips

1. CSV format isn't a standard. Use XML can solve character issue but it might not perform as well due to formatting overhead.
2. Transform is not always needed. If not, use "SELECT … INTO TABLE" to provide a straight database-to-database extract-and-load.
3. Incremental load is highly desirable. Use trigger can achieve that.
4. Operational database uses MySQL's InnoDB backend, providing referential integrity and transactions. However, we chose MySQL's MyISAM backend for our warehouse for better performance as it is read-only and transactional feature is not needed.
5. MySQL does not support for bitmap indexes. Bitmap indexes are ideal for the kind of low-cardinality data that is commonly used in data warehouses. PostgreSQL supports bitmap indexes as of version v8.1, as do a number of commercial database systems.

When I dig deeper in business intelligence, I found out that it is a huge topic ranging from reporting to data mining. Like all the knowledge acquisition plan, I put a series of milestones for myself. If you are interested, here is my list:

Get and prepare your data

• Data collection – log processing, web services (SOAP and REST), RSS, screen scraping and more.
• Data preparation and crunching – ETL (Kettle)
• Data storage – data warehousing

Visualize your data

• Reporting and Charting (Pentaho as server and Flex as frontend)

Analysis your data

• Data modeling
• Data analysis (OLAP)

Get smart of your data

• Collective intelligence
• Data mining

Introduction of Pentaho

Firstly, I want to see whether there is any out of the box  open-source solution that captures what I am trying to do here. If so, I can reach my goal much faster. Yes. It has to be open-source b/c I don’t have $$ and I don’t want to be just a user. After doing my homework a bit, I found out an open-source BI tool named Pentaho that looks pretty solid. So, I decide to dive deep to this. Like all the tools I mess around with, I want to integrate Pentaho as library. However, I don’t find anything on the Net that shows me how to do it. I am looking into its download and checkout its pentaho-sample project. What it shows me is how to use their tools to create a report on their systems using their UI. I definitely need more!

After few days of efforts, I managed to pull out all the unnecessary dependencies from pentaho. The heart of Pentaho is its xaction interpreter. The approach Pentaho uses is to write adapters, plug into its framework and use xaction to wire them up in a workflow fashion. In fact, most of its functionalities come from other open-source projects like quartz for scheduling, shark for workflow engine and jfreereport for reporting. I don’t think their xaction is clean but I do like their architectural approach.

Enough talk, lets start! Here I would use series of articles to cut your learning curve and show you how to get yourself familiar with Pentaho as a developer rather than user. First thing first, follow the articles below to set up your environment. 

1. http://wiki.pentaho.com/display/PentahoDoc/07.+Debugging+with+the+Standalone+Platform+Project
2. http://wiki.pentaho.com/display/PentahoDoc/Building+and+Debugging+Pentaho+with+Eclipse
3. http://wiki.pentaho.com/display/PentahoDoc/Manual+Deployment+of+Pentaho

Populate it
insert into item(id) values(1);
insert into item(id) values(2);
insert into item(id) values(3);
insert into item(id) values(4);

insert into item_log(item_id, price) values(1, 100);
insert into item_log(item_id, price) values(1, 100);
insert into item_log(item_id, price) values(1, 100);
insert into item_log(item_id, price) values(1, 200);
insert into item_log(item_id, price) values(1, 200);
insert into item_log(item_id, price) values(1, 200);
insert into item_log(item_id, price) values(1, 200);
insert into item_log(item_id, price) values(1, 200);
insert into item_log(item_id, price) values(1, 200);
insert into item_log(item_id, price) values(2, 200);
insert into item_log(item_id, price) values(2, 200);

Run SQL
SELECT COUNT(il.price), i.id AS item_id, il.price
FROM item i, item_log il
WHERE i.id = il.item_id GROUP BY il.price, i.id;

Result
count | item_id | price
——-+———+——-
3 | 1 | 100
6 | 1 | 200
2 | 2 | 200

Run SQL
SELECT COUNT(il.price), i.id AS item_id, il.price, array_accum(il.id) AS item_id_array
FROM item i, item_log il
WHERE i.id = il.item_id
GROUP BY il.price, i.id;

Result
count | item_id | price | item_id_array
——-+———+——-+—————
3 | 1 | 100 | {1,2,3}
6 | 1 | 200 | {4,5,6,7,8,9}
2 | 2 | 200 | {10,11}

1. Divide the world into measurements and context.
2. Numeric measurements place in Fact table whereas context are broken down into Dimensions. A fact table in a pure star schema consists of multiple foreign keys, each paired with a primary key in a dimension, together with the facts containing the measurements.
3. Build the FK-PK pairs as surrogate keys that are just sequentially assigned integers.
4. Use a special record in Dimension to represent unknown or no because we want to avoid putting null as FK.
5. Resist snowflake the dimensional tables and leave them in flat second normal form because the flat tables are much more efficient to query. Snowflaking a dimension into third normal form, while not incorrect, destroys the ability to use bitmap indexes and increases the user-perceived complexity of the design.
6. Semi-additive fact – Most fact tables are huge, with millions or even billions of rows, you almost never fetch a single record into your answer set. Rather, you fetch a very large number of records, which you compress into digestible form by adding, counting, averaging, or taking the min or max. Bank balance and inventory levels represent intensities that are awkward to express in an additive format. You sum over balance for 1 month is not really meaningful. Normally, we still treat these semiadditive facts as if they were additive but just before presenting the results to the end user, divide the answer y the number of time periods to get the right result. This technique is called averging over time.
7. Slowly changing dimension -
8. Hierarchical Dimension – There are 2 types of hierarchies. One is “Parent-Child” relationship and the other one is “Array of Level”. Array of level like Country -> State -> City -> Store. Parent-Child like product categories that can be nested in different ways.

There are debates about whether we should choose PostgreSQL vs MySQL. Here is one case study that shows PostgreSQL is better in OLTP system. However, for select query, another study shows MySQL v5 is 2X faster the PostgreSQL v8. For data warehouse application, MySQL sounds like a better option as it is mostly read-only.

Here is the summary that I obtained from this article that compares PostgreSQL with MySQL:

1. MySQL uses traditional row-level locking. PostgreSQL uses something called Multi Version Concurrency Control (MVCC) by default. MVCC is a little different from row-level locking in that transactions on the database are performed on a snapshot of the data and then serialized.
2. MySQL supports the advanced feature of data partitioning within a database whereas PostgreSQL does not.
3. PostgreSQL has many of the database features that Oracle, DB2, or MS-SQL has, including triggers, views, inheritance, sequences, stored procedures, cursors, and user-defined data types. MySQL’s development version, version 5.0, supports views, stored procedures, and cursors. MySQL’s future version, version 5.1, will support triggers.
4. PostgreSQL supports user-defined data types, while MySQL does not.
5. Both MySQL and PostgreSQL have support for single-master, multi-slave replication scenarios. PostgreSQL offers additional support for multi-master, multi-slave replication from a third-party vendor, as well as additional replication methods.
6. MySQL uses a threaded model for server processes, wherein all of the users connect to a single database daemon for access. PostgreSQL uses a non-threaded model where every new connection to the database gets a new database process.
7. MySQL does not support for bitmap indexes. Bitmap indexes are ideal for the kind of low-cardinality data that is commonly used in data warehouses. PostgreSQL supports bitmap indexes as of version v8.1, as do a number of commercial database systems.

Installation of Pentaho

1. Download Pentaho Demo (PCI) here
2. Read Pentaho Quick Start and the Creating Pentaho Solutions pdf documents. You can get those documents from the download center above as well.
3. Unzip the download file will result in a pentaho-demo directory. This is the server root, and it is commonly referred to as the PCI root or PCI install directory or something similar. To start the server, windows users run start-pentaho.bat; *nix users run start-pentaho.sh.
4. Open an internet browser, and navigate to:http://localhost:8080/. This may take a little while – the server needs to warm up.
5. Now you should see the pentaho web front. Try this sample out to make sure the setup is correctly done.
   • Getting Started::Hello World
   • Reporting
   • Chart Examples. Shows some of the included charting capabilities
   • Analysis / OLAP Examples. Demonstrates slice and dice
   • Dashboards. There is demo “Flash Dashboard” that actually uses XML-driven free Flash chart. Look pretty good! However, I suggest to use FLEX Charting although it is not free
6. When you’re done with pentaho, locate the stop-pentaho script in the PCI installation directory. Execute the script to stop de server.
7. For more info of how to set up PCI as server and how to configure email service. Take a look at Roland blog.

Create a new sample

1. Start pentaho demo as stated above.
2. Create a folder named “MySQL” under %PCI%/pentaho-solutions/samples/mysql.
3. To make mysql folder to display at the entry page. We need to put index.xml in the mysql folder. You may notice that we are using variables for name and description. The value of the variables are defined using index.properties under the same folder. The reason to do that is to support internationalization because you can define index_cn.properties for Chinese wording. Note: Click “Update Solution Repository” under Admin tab to refresh the change.
4. Download MySQL Sample database - sakila. Here is the schema view.

<index>
      <name>%directory_name</name>
      <description>%directory_description</description>
      <icon>folder.png|dashboard.jpg</icon>
      <visible>true</visible>
      <display-type>list</display-type>
</index>

5. Download mysql v5 database and its jdbc driver.
6. Run sakila_data.sql and sakila_schema.sql against MySql database. Now you have your sample movie database ready.

7. Create a file named mysql-ds.xml in $DEMO_BASE/jboss/server/default/deploy/

<?xml version=”1.0″ encoding=”UTF-8″?>
<datasources>
      <local-tx-datasource>
      <jndi-name>sakila</jndi-name>
      <connection-url>jdbc:mysql://localhost/sakila</connection-url>
      <driver-class>com.mysql.jdbc.Driver</driver-class>
      <user-name>root</user-name>
      <password>honr</password>
      </local-tx-datasource>
</datasources>

8.  Edit the file $DEMO_BASE/jboss/server/default/deploy/pentaho.war/WEB-INF/web.xml. Add the following right below solution5 resource-ref entry

   <resource-ref>
         <description>sakila</description>
         <res-ref-name>jdbc/sakila</res-ref-name>
         <res-type>javax.sql.DataSource</res-type>
         <res-auth>Container</res-auth>
   </resource-ref>

9.  Edit the file $DEMO_BASE/jboss/server/default/deploy/pentaho.war/WEB-INF/jboss-web.xml. Add the following right below the solution5 entry.

   <resource-ref>
         <res-ref-name>jdbc/sakila</res-ref-name>
         <res-type>javax.sql.DataSource</res-type>
         <jndi-name>java:/sakila</jndi-name>
   </resource-ref>

10. Copy your mysql jdbc driver library in the following directory: $DEMO_BASE/jboss/server/default/lib
11. Create your own myFirst.xaction file.

<?xml version=”1.0″ encoding=”UTF-8″?>
<action-sequence>
      <name>myFirst.xaction</name>
      <title>%title</title>
      <version>1</version>
      <logging-level>debug</logging-level>
      <documentation>
           <author>Raymond Hon</author>
           <description>%description</description>
           <help/>
           <result-type>rule</result-type>
           <icon>SQL_Datasource.png</icon>
     </documentation>
     <inputs/>
     <outputs>
           <rule-result>
               <type>list</type>
           </rule-result>
     </outputs>
     <resources/>
     <actions>
           <action-definition>
                  <action-outputs>
                         <rule-result type=”list” />
                  </action-outputs>
                  <component-name>SQLLookupRule</component-name>
                  <action-type>rule</action-type>
                  <component-definition>
                         <jndi>sakila</jndi>
                         <query>
                                <![CDATA[select * from actor where actor_id = 1]]>
                        </query>
                 </component-definition>
          </action-definition>
     </actions>
</action-sequence>

12. Create your own myFirst.properties with title and description.
13. Restart pentaho
14. Open up Firefox (cough IE) and hit the following URL: http://localhost:8080/pentaho/ViewAction?&solution=samples&path=mysql&action=myFirst.xaction
15. That’s it!