One strength of the Java™ 2 Platform, Standard Edition (J2SE™) is that it performs automatic memory
management, thereby shielding the developer from the complexity of explicit memory management. However, it doesn’t mean that there will not be any memory leak. So, I decide to give a summary on some key areas in this topics. They are:

1. How garbage collector works?
2. How memory leak still shows up?
3. What is weak reference?
4. What stores in heap and stack?

If you want to understand this topic in detail, you can read the this article from Sun. The key points are summarized below:

1. Without automatic memory management - GC, we may face 2 common issues: dangling reference (deallocate a object while others are still referencing it) and space leak (some objects are not referenced but not been deallocated either). GC is great but it doesn’t solve all the memory allocation problem. For example, you can have object list keep growing until it uses up all the free memory.
2. Garbage collection takes time and resource to do it and sometimes it is not acceptable for the real-time mission critical system.
3. The task of fulfilling an allocation request, which involves finding a block of unused memory of a certain size in
   the heap, is a difficult one. The main problem for most dynamic memory allocation algorithms is to avoid
   fragmentation, while keeping both allocation and deallocation efficient. One approach to eliminating fragmentation is called compaction.
4. It is also desirable that a garbage collector operate efficiently, without introducing long pauses during which the
   application is not running.

ShareThis

Video Presentation

Jinesh Varia - an evangelist from Amazon. In his presentation, he will show you how to build a regular-expression based search engine called “GrepTheWeb” on top of the Amazon infrastructure - SQS, SimpleDB, EC2 and S3. The most interesting thing he mentioned in this presentation is the on-demand architecture powered by Hadoop and Amazon infrastructure. “At time t0, you have no infrastructure. At time t1, when regular expression comes in, the system reaches the execution phase and the whole infrastructure is ready for it. At time t2, the request is fulfilled, the whole infrastructure is gone…” This gives me a taste of cloud computing and how powerful it can be.

• Building GrepTheWeb in the Cloud, Part 1: Cloud Architectures
• Building GrepTheWeb in the Cloud, Part 2: Best Practices

Web Resource

High Scalability posts an article about Amazon architecture. The author follows up with different resources and consolidates key information he found.

ShareThis

B-Tree Indexing

ShareThis

1. Buffett loves insurance business. By sterring Berkshire into the insurance business, Buffett was able to get his hands on tremendous amount of float.
2. Buffett puts lots of focuses on the good-quality management that he sees it will create long-term value.
3. Buffett relies heavily on a discounted cash flow (DCF) analysis to find stocks and companies that can be purchased for less than intrinsic value. In other words, Buffett doesn’t buy cheap stock, he buys stock cheap.
4. The key here is how to accurately calculate the intrinsic value.
5. Diversification lowers your risk but it also locks you in the market rate of return. In Buffett’s mind, he believes if you know what  you are doing and really understand how to evaluate business, you just needs to hold dozen stocks to be sufficient diversification.
6. From studies, value stocks beat growth stocks (ie. high price multiple like P/E) over the long term, but that growth stocks are a better bet for investors with shorter horizons. It also shows that small cap stocks do better than large-cap stocks over the long term.
7. Buffett is a buyer of companies than a buyer of stocks.
8. Buffett actively petitioned against efforts to eliminate personal taxes on dividends, even though dividends are paid to shareholders only after corporations have paid their own taxes. In other words, the situation results in double taxation. However, lower tax rates could benefit everyone by stimulating economic growth means better job opportunities. Author believes taxing the investment class at higher rates is likely to do more harm than good.

ShareThis

There were 2 main XML parsing technologies few years ago. They were SAX and DOM.

1. SAX is event-driven and the events are fired and forget along the xml parsing. Advantages: It doesn’t need to cache the whole xml document in memory and you don’t need to wait til the whole xml been parsed before the first event emitted. Disadvantages: It uses Push API that holds the control during parsing. So clients cannot control the parsing and it doesn’t fit for xml manipulation.
2. DOM is used to convert the xml into object tree in memory before manipulation. Advantages: Easier to manipulate the xml. Disadvantages: Eat up a lot of memory that is not good for documents larger than few MBs in size or in memory constrained environment such as J2ME.

Pull API is a more comfortable alternative for streaming processing of XML. A pull API is based around the more familiar iterator design pattern rather than observer design pattern. In a pull API, the client program asks the parser for the next piece of information rather than the parser telling the client program when the next datum is available. In a pull API the client program drives the parser. In a push API the parser drives the client. That leads to the invention of StAX.

In this article, I will introduce an new object model from Axis2 named AXIOM that uses StAX underneath for xml parsing. With this, xml parsing will cost less memory with better control.

Evolution of Axis

One of the first generation SOAP engines, Apache SOAP, uses a DOM-based object model internally to represent the XML document, where the XML handling techniques force the entire XML object model to be built at once. The second generation Apache Axis shifted to SAX to avoid keeping the complete information in the memory. SAX, however, has a major constraint - it is built around a "push" technique, and once the parsing of the XML document starts it cannot be stopped. To jump over this hurdle, Apache Axis has to record SAX events. So, effectively, the XML message has to be kept in the memory in the form of SAX events, thus making Apache Axis yet another memory intensive programming model.

Axis2 avoids keeping the complete SOAP message in the memory by introducing a new Object Model for representing the SOAP message AXIOM. AXIOM takes a dramatically new approach. Although AXIOM has an "external" resemblance to DOM, the difference lies in that it generates objects only when required. This "on-demand building" feature gives AXIOM the edge needed to overcome the memory barrier that early SOAP engines failed to pass.

An interesting feature of AXIOM is that it is based on Pull parsing. It is capable of generating pull events from the Object Model that is built. Further, if the Object Model happens to be half built, AXIOM is capable of shifting to the underlying pull parser to generate pull events directly from the stream. The heart of AXIOM is the XML Pull parser since it is the only parsing model that supports the pausing of the parsing process. AXIOM uses the Streaming API for XML (StAX), making it easy to manipulate and utilizing only a fraction of the memory used by a conventional object model. Combined with the speed of the streaming pull parser, AXIOM pushes Axis2 leaps ahead of its predecessors in terms of efficiency and speed.

Apart from new parser, Axis2 also has other new add-ons. They are:

1. Pluggable Data Binding - you can pick and choose JAXB, Castor and XMLBean for xml - java conversion.
2. Improved Support for Message-style interaction (RPC vs Message-based)
3. Improved handlers

The goal of this article is to focus on parsing technology, so I will not discuss in detail the new features on Axis2. If you want to find out more, read this.

Reference

An Introduction to StAX

Fast and lightweight object model for XML

ShareThis

After getting a taste of its power, lets set it up and try it ourselves.

ShareThis

Solr is a standalone enterprise search server with a web-services like API. You put documents in it (called "indexing") via XML over HTTP (RESTful). You query it via HTTP GET and receive XML results.

• Advanced Full-Text Search Capabilities
• Optimized for High Volume Web Traffic
• Standards Based Open Interfaces - XML and HTTP
• Comprehensive HTML Administration Interfaces
• Scalability - Efficient Replication to other Solr Search Servers
• Flexible and Adaptable with XML configuration
• Extensible Plugin Architecture

Set up Solr

 To set up Solr, you should follow this guideline. After the set up Solr, you practically have a indexing service up.

The HTTP/XML interface of the indexer has two main access points: the update URL, which maintains the index, and the select URL, which is used for queries. In the default configuration, they are found at:

• http://[hostname:port]/solr/update
• http://[hostname:port]/solr/select

To add a document to the index, we POST an XML representation of the fields to index to the update URL. In addition, you can delete, update (ie. re-post on unique). All change operations need to commit to flush to file system. On the other hand,  once we have indexed some data, an HTTP GET on the select URL does the querying. 

Powerful features Behind Solr

If you follow the guideline above, you already get yourself familiar with indexing, searching and facet browsing. Now lets get down to how to make Solr a scalable solution with great performance.

Caching

TBA

Distribution and Replication

For applications that receive large volumes of queries, a single Solr server may not be enough to meet performance requirements. Therefore, Solr provides mechanisms for replicating the Lucene index across multiple servers that are part of a load-balanced suite of query servers. The replication process is handled through a combination of event listeners enabled through the solrconfig.xml file and several shell scripts (located in solr/bin of the example application).

In a replicating architecture, one Solr server acts as the master server, providing copies of the index (called snapshots) to one or more slave servers that handle query requests. Indexing commands are sent to the master server and queries are sent to the slave servers. The master server can create snapshots manually or by configuring the <updateHandler> section of solrconfig.xml to trigger snapshot creation when commit and/or optimize events are received. In either the manual or the event-driven process, the snapshooter script is invoked on the master server, creating a directory on the server named snapshot.yyyymmddHHMMSS where yyyymmddHHMMSS is the actual time the snapshot was created. The slave servers then use rsync to copy only those files in the Lucene index that have been changed.

<listener event="postCommit" class="solr.RunExecutableListener">
    <str name="exe">snapshooter</str>
    <str name="dir">solr/bin</str>
    <bool name="wait">true</bool>
    <arr name="args"> <str>arg1</str> <str>arg2</str> </arr>
    <arr name="env"> <str>MYVAR=val1</str> </arr>
</listener>

Reference

Below are some cool references I found:

1. Search smarter with Apache Solr, Part 1: Essential features and the Solr schema
2. Search smarter with Apache Solr, Part 2: Solr for the enterprise
3. Advanced Lucene

ShareThis

After Lucene, the author created another powerful tool. Its name is Nutch. Nutch is a powerful crawler built on top of the Lucene. With Nutch, you can launch a multi-threaded crawler to obtain information from the Net. At this point of writing, Nutch is in its 0.9 version. Nutch comes with a list of cool features, including whole Web crawling, local file crawling for the
intranet, indexing all the while.

Hadoop was designed to handle the petabytes of data that Nutch could potentially store and process. In fact, Hadoop has its own file system: the Hadoop Distributed File System (HDFS), which can run on any old run-of-the-mill, low-cost hardware.
Hadoop works by storing part of the file system’s data across all the servers in the cluster. As new queries come in, HDFS follows the "moving computation is cheaper than moving data" rule — meaning that moving the processing of the query to as
close as possible to the data will be faster than placing the query at random within the cluster and moving data long distances across the network.

I have searched around to see if anyone can give me some tips on this tool. Surprisingly, I don’t see much. But don’t worry, I have found some that can at least get you start playing with it.

Set up Nutch

Here is the guideline written by Peter Wang that I followed to bring my Nutch up. Follow it and bring your Nutch before go further. By the way, if you want to run Nutch with Solr, this is a good tutorial.

Nutch Architectural Review

ShareThis

I have heard of Lucene and its powerful full text search capability many times. Today, I decide to take a look at it. Before I dive into the user guide, I went to Google Tech Talk to find a video related to Lucene first. Here is what I found: 

After I finished this video, I found Lucene a really great tool for me. So, I decided to have a deeper look at it. After a quick search,  I found a great blog that showed me how to use Lucene with Digg. With Solr on top of Lucene, you can make Lucene available as RESTful Web Service. It is so awesome, isn’t it? In this article, I will list you all the information I found during my little research on Lucene and I hope you will feel it useful.

Architecture Overview

Before we dig into the code or set up guidelines, I would like to have a high level picture of Lucene first. I borrow a diagram from this article that helps me to grasp the key components in search.

This high level picture shows you that your search keywords you entered (normally using a form) will become a HTTP search request and later been translated into a form that search engine understands by Query parser. Search engine will perform the search operation against the indexed files that was previously prepared by Indexer. After that, the result will be ranked based on predefined ranking algorithm and returned to the user. The source of the data can be from Web Service, database or documents in your file system. In this diagram, it shows you that you can launch spider or crawler like Google to obtain the data from web pages on the Internet and feed it to Indexer as your source.

Get one step deeper

Now you know the high level flow of how search works. Lets get one step into the detail.

1. What search interface to use?
2. How search interface communicates with your search engine?
3. What kind of search the search engine provides?
4. How search engine indexes the documents?
5. How result be ranked and what kind of ranking algorithms we normally use?

Below is the answers of the questions above:

1. Up to you. I would use Flex as I want to provide a rich search interface to my users.
2. Flex can talk HTTP, Web Service or RemoteObject AMF. If you put web service layer on Lucene (ie. Solr), you can use REST call (ie. HTTP) to obtain the result.
3. Lucene supports several kinds of advanced searches like:
   • Boolean operators - users can compose query using AND, OR, NOT
   • Field Search - what fields the search operates on? like title, author or content?
   • Wildcard Search - supports * and ?.
   • Fuzzy Search - Lucene provides a fuzzy search that’s based on an edit distance algorithm. You can use the tilde character (~) at the end of a single search word to do a fuzzy search. For example, the query "think~" searches for the terms similar in spelling to the term "think." The key here is the word "similar". Do we consider horse and donkey are related? Or you have hose and horse be related somehow in spelling?
   • Range Search - age, date and etc
4. Large topic. I will go back to it later.
5. Up to you. If you want to look at the popular ranking algorithm in the world, check out Google Page Rank. It is one of the algorithms that many of us interested to know. Before I want to have my wedding website - Justproposed.com be shown on at the top of the result when users type "wedding website" as search keywords, I have looked into SEO. It is a fun area to explore. Generally speaking, if the query keywords shown in the title, it weights more, If the keyword frequency is higher, it ranks higher..blah blah. However, I know Google has weighted a lot on the links. It is not just purely based on the document that you have. How to obtain the additional information during the crawling is beyond the scope of this article.

Get your hand dirty

Look into this great article.

The thing this article doesn’t mention is that you need to create you dataDir and indexDir folders under C and drag a list of html files into the dataDir before you start the web server. If you drag new htmls into it, you need to clean up your indexDir and restart your web server in order to rebuild the indexes.

I have got the application up and running. It is nice trial. My next step is to enhance this example. I will do the following:

1. Use Flex as search interface
2. Use Solr to expose the Lucene search engine as Web Service.
3. Have Flex calls my search engine via REST.
4. Display the result on Flex.

After I have my new enhancements working, I would do the following:

1. Look into how Lucene do the indexing
2. Look into Nutch,. So I can have it crawled some sites and put the htmls in dataDir for me automatically.

How Lucene Indexes the documents?

Yes. I haven’t forgot to answer the question 4. Here is the article that answers your question. To summarize, here are several key points I extracted from this article.

1. Content Extraction - Lucene only takes text for index. So, it provides different types of parsers to extract content from different types of document like word, html, doc, pdf and etc. If you have other type of document that you cannot find a parser, you take the responsibility to extract the content out for Lucene. This article shows you how to use Digester to extract content out from XML and feed Lucene. If you have a large pool of XML for content extraction, you need to pay attention on the parsing time. There is someone who has done this and obtain some performance number as reference. However, the article was a bit outdated.
2. Content Preprocessing - Analyzer is used to extract the token from your text content to be indexed. Before text is indexed, it is passed through an Analyzer. Analyzers are in charge of extracting indexable tokens out of text to be indexed, and eliminating the rest. Lucene comes with a few different Analyzer implementations. Some of them deal with skipping stop words (frequently-used words that don’t help distinguish one document from the other, such as "a," "an," "the," "in," "on," etc.), some deal with converting all tokens to lowercase letters, so that searches are not case-sensitive, and so on.
3. Indexing - IndexWriter is the key component in the indexing process. This class will use Analyzer that you passed in as parameter to create a new index or open an existing index and add documents to it. You need to set up fields and documents and feed them to the IndexWriter to do the job. Like the code below, you fetches a list of .txt files and its metadata like path from a directory and feed them for IndexWriter. IndexWriter will index them one after one.
4. Configuration - You can configure IndexWriter to achieve better performance via increasing the buffer size because the bottleneck normally happen during the IO of the index files.
5. Lucene uses inverted index concept. An inverted index is an inside-out arrangement of documents in which terms take center stage. Each term points to a list of documents that contain it. On the contrary, in a forwarding index, documents take the center stage, and each document refers to a list of terms it contains. You can use an inverted index to easily find which documents contain certain terms. Lucene uses an inverted index as its index structure.

for(int i = 0; i < textFiles.length; i++){
      if(textFiles[i].isFile() >> textFiles[i].getName().endsWith(".txt")){
        Reader textReader = new FileReader(textFiles[i]);
        Document document = new Document();
        document.add(Field.Text("content",textReader));
        document.add(Field.Keyword("path",textFiles[i].getPath()));
        indexWriter.addDocument(document);
      }
}

Lucene offers four different types of fields from which a developer can choose: Keyword,UnIndexed,UnStored,and Text.

Keyword fields are not tokenized, but are indexed and stored in the index verbatim. This field is suitable for fields whose original value should be preserved in its entirety, such as URLs, dates, personal names, Social Security numbers, telephone numbers, etc.

UnIndexed fields are neither tokenized nor indexed, but their value is stored in the index word for word. This field is suitable for fields that you need to display with search results, but whose values you will never search directly. Because this type of field is not indexed, searches against it are slow. Since the original value of a field of this type is stored in the index, this type is not suitable for storing fields with very large values, if index size is an issue.

UnStored fields are the opposite of UnIndexed fields. Fields of this type are tokenized and indexed, but are not stored in the index. This field is suitable for indexing large amounts of text that does not need to be retrieved in its original form, such as the bodies of Web pages, or any other type of text document.

Text fields are tokenized, indexed, and stored in the index. This implies that fields of this type can be searched, but be cautious about the size of the field stored as Text field.

Conclusion

To use Lucene, there are 3 main concepts you need to grasp. There are:

1. Indexer - create search engine indexes
2. Analyzer - Split text into tokens that make sense for the search engine. The structure is like document -> a sequence of fields and each field is name/value pair -> tokens. Field values may be stored, indexed or analyzed/ tokenize, (and, now, vectored). The lecture note from Doug Cutting will give you more detail.
   
3. Searcher

You may think of using grep to achieve or database to achieve what Lucene does. Grep is powerful Linux tool, however, if you want it to search on files with several MB in size, you will see that the tool is inefficient. The reason is grep doesn’t prepare the indexes of your files ahead of the time you do the search. Database can do indexing but not so sophisticated as Lucene in your varchar field. Oracle may provide one but I am not familiar with it. One key thing to remember: Lucene is open source, free and does the job extremely well. Why bother to dig into Oracle costly solution?

Lucene has given us a rich search engine capability on our web application. It has many features that I haven’t got a chance to discuss them all in this article. I will continue to write more articles on this topic as my research moves forward. Have a nice day!

Reference

The blog of the Lucene and Solr creator - Doug Cutting

ShareThis

ShareThis