In our daily life, we deal with lots of data. The data normally is stored in text format for the ease of human to read. With the large amount of data we have, we need ways to deal with it. There are several things we frequently do on the data: Search, Filter, Sort and Analysis. In Linux, there are some powerful commands that I can use: cat, grep, find, sort, unique and etc. I found those commands quite powerful. So, I decide to put these down as my reference. This tutorial I will go over the basic text processing commands and how we use them together to achieve the tasks we often encounter in our workplace. 

cat

The power of “cat” is not just output a file to screen but to concatenates a list of file content and stream through the pipe to another program as input.

cat * | sort

find

The power of find is to list out the matched filenames based on metadata of the files like type, size, create date…

grep

“grep” helps you to list out the file(s) with the content that match the pattern(s) in regular expression. You can use it as content search across the files in your file system.

grep -H -R --color -n -P abc *

option:

1. –color (highlight matching part in content with color)
2. -n (show line number)
3. -P PATTERN (perl regular expression pattern)
4. -R (recursively)
5. -l (only list out the filenames that match the pattern)
6. -H show filename that matched.

cut

“cut” extracts sections from each line of input. (example of usage). Below the command will extract the 5th field and the rest from each line of file A using delimiter colon.

cut -d ":" -f 5- fileA

option:

1. -c (character)
2. -b (byte)
3. -f 5- (field if the line can be broken down by delimiter)
4. -d | (delimiter is pipe character)

sort 

The sort command sorts a file according to fields–the individual pieces of data on each line. By default, sort assumes that the fields are just words separated by blanks, but you can specify an alternative field delimiter if you want (such as commas or colons). Output from sort is printed to the screen, unless you redirect it to a file.

donor.data
Bay Ching 500000 China
Jack Arta 250000 Indonesia
Cruella Lumper 725000 Malaysia

Let’s take this sample donors file and sort it according to the donation amount. The following shows the command to sort the file on the second field (last name) and the output from the command:

sort +1 -2 donors.data
Jack Arta 250000 Indonesia
Bay Ching 500000 China
Cruella Lumper 725000 Malaysia

If the file is delimited by comma, you can use -t , to tell the sort the delimiter. You can use -u to output the uniqueness as well.

sort -t: +1 -2 company.data
Nasium, Jim:031762:Marketing
Jucacion, Ed:396082:Sales
Itorre, Jan:406378:Sales
Ancholie, Mel:636496:Research

To sort the file on the third field (department name) and suppress the duplicates, use this command:

sort -t: -u +2 company.data
Nasium, Jim:031762:Marketing
Ancholie, Mel:636496:Research
Itorre, Jan:406378:Sales

Note that the line for Ed Jucacion did not print, because he’s in Sales, and we asked the command (with the -u flag) to suppress lines that were the same in the sort field.

option:

1. -f Make all lines uppercase before sorting (so “Bill” and “bill” are treated the same).
2. -r Sort in reverse order (so “Z” starts the list instead of “A”).
3. -n Sort a column in numerical order
4. -tx Use x as the field delimiter (replace x with a comma or other character).
5. -u Suppress all but one line in each set of lines with equal sort fields (so if you sort on a field containing last names, only one “Smith” will appear even if there are several).
6. Specify the sort keys like this: +m Start at the first character of the m+1th field. -n End at the last character of the nth field (if -N omitted, assume the end of the line)

uniq

uniq – line level uniqueness. It prints the unique lines in a sorted file, retaining only one of a run of matching lines. Optionally, it can show only lines that appear exactly once, or lines that appear more than once. uniq requires sorted input since it compares only consecutive lines.

option:

1. -u (print the unqiue lines only – lines only appear once)
2. -d (print the duplicate lines only – lines appear more than once)
3. -c (prefix each line with occurrence)

[code]]czoyMjY6XCJiYXNoJCBjYXQgdGVzdGZpbGU8YnIgLz4NClRoaXMgbGluZSBvY2N1cnMgb25seSBvbmNlLjxiciAvPg0KVGhpcyBsaW57WyYqJl19ZSBvY2N1cnMgdHdpY2UuPGJyIC8+DQpUaGlzIGxpbmUgb2NjdXJzIHR3aWNlLjxiciAvPg0KVGhpcyBsaW5lIG9jY3VycyB0aHJlZXtbJiomXX0gdGltZXMuPGJyIC8+DQpUaGlzIGxpbmUgb2NjdXJzIHRocmVlIHRpbWVzLjxiciAvPg0KVGhpcyBsaW5lIG9jY3VycyB0aHJlZSB0e1smKiZdfWltZXMuXCI7e1smKiZdfQ==[[/code]

[code]]czoxMzk6XCI8YnIgLz4NCmJhc2gkIHVuaXEgLWMgdGVzdGZpbGU8YnIgLz4NCjEgVGhpcyBsaW5lIG9jY3VycyBvbmx5IG9uY2UuPGJ7WyYqJl19ciAvPg0KMiBUaGlzIGxpbmUgb2NjdXJzIHR3aWNlLjxiciAvPg0KMyBUaGlzIGxpbmUgb2NjdXJzIHRocmVlIHRpbWVzLlwiO3tbJiomXX0=[[/code]

[code]]czoxNjA6XCI8YnIgLz4NCmJhc2gkIHNvcnQgdGVzdGZpbGUgfCB1bmlxIC1jIHwgc29ydCAtbnIgPGJyIC8+DQozIFRoaXMgbGluZSB7WyYqJl19b2NjdXJzIHRocmVlIHRpbWVzLjxiciAvPg0KMiBUaGlzIGxpbmUgb2NjdXJzIHR3aWNlLjxiciAvPg0KMSBUaGlzIGxpbmUgb2NjdXtbJiomXX1ycyBvbmx5IG9uY2UuICBcIjt7WyYqJl19[[/code]

wc

wc - word count. Apart from word count, it also does the following

1. wc -w gives only the word count.
2. wc -l gives only the line count.
3. wc -c gives only the byte count.
4. wc -m gives only the character count.
5. wc -L gives only the length of the longest line.

tr

"tr" translate or delete characters. It is used for data cleaning job. Can we do pattern replacement?

tr '[:lower:]' '[:upper:]'

The above command will convert all the lowest case to upper case.

tr '.' '/'

The above will convert all the . character to /. And for translation, you cannot have -d option on. You may be asking when would we do that. Here is the common use case - convert window files to unix formatted file:

tr -d '\r' < input_dos_file.txt > output_unix_file.txt

option:

1. -s (squeeze the repeated characters into one character. eg. tr -s '\n' )
2. -d (delete characters eg. tr -d '\000')

sed

"tr" can do character replacement. But if you want to do pattern replacement, you need to use sed. usage: sed -e s/pattern/replacement/flags

sed -e s/one/another

sed -e s/[aeiou]/_/g

 Note the use of the "g" flag so that you apply the pattern/replacement to every match instead of just the first one.

awk

Put them all together

[code]]czo4NjpcImNhdCAqIHxncmVwIGx1Y2VuZS1jb3JlfGN1dCAtZjIgLWRcJyBcJ3x1bmlxfHRyIFwnLlwnIFwnL1wnfCBhd2sgXCd7cHJpbnRmIFwiJXtbJiomXX1zLmNsYXNzXFxuXCIsICQxfVwnXCI7e1smKiZdfQ==[[/code]

I have heard of Lucene and its powerful full text search capability many times. Today, I decide to take a look at it. Before I dive into the user guide, I went to Google Tech Talk to find a video related to Lucene first. Here is what I found: 

After I finished this video, I found Lucene a really great tool for me. So, I decided to have a deeper look at it. After a quick search,  I found a great blog that showed me how to use Lucene with Digg. With Solr on top of Lucene, you can make Lucene available as RESTful Web Service. It is so awesome, isn’t it? In this article, I will list you all the information I found during my little research on Lucene and I hope you will feel it useful.

Architecture Overview

Before we dig into the code or set up guidelines, I would like to have a high level picture of Lucene first. I borrow a diagram from this article that helps me to grasp the key components in search.

This high level picture shows you that your search keywords you entered (normally using a form) will become a HTTP search request and later been translated into a form that search engine understands by Query parser. Search engine will perform the search operation against the indexed files that was previously prepared by Indexer. After that, the result will be ranked based on predefined ranking algorithm and returned to the user. The source of the data can be from Web Service, database or documents in your file system. In this diagram, it shows you that you can launch spider or crawler like Google to obtain the data from web pages on the Internet and feed it to Indexer as your source.

Get one step deeper

Now you know the high level flow of how search works. Lets get one step into the detail.

1. What search interface to use?
2. How search interface communicates with your search engine?
3. What kind of search the search engine provides?
4. How search engine indexes the documents?
5. How result be ranked and what kind of ranking algorithms we normally use?

Below is the answers of the questions above:

1. Up to you. I would use Flex as I want to provide a rich search interface to my users.
2. Flex can talk HTTP, Web Service or RemoteObject AMF. If you put web service layer on Lucene (ie. Solr), you can use REST call (ie. HTTP) to obtain the result.
3. Lucene supports several kinds of advanced searches like:
   • Boolean operators – users can compose query using AND, OR, NOT
   • Field Search – what fields the search operates on? like title, author or content?
   • Wildcard Search - supports * and ?.
   • Fuzzy Search - Lucene provides a fuzzy search that’s based on an edit distance algorithm. You can use the tilde character (~) at the end of a single search word to do a fuzzy search. For example, the query "think~" searches for the terms similar in spelling to the term "think." The key here is the word "similar". Do we consider horse and donkey are related? Or you have hose and horse be related somehow in spelling?
   • Range Search - age, date and etc
4. Large topic. I will go back to it later.
5. Up to you. If you want to look at the popular ranking algorithm in the world, check out Google Page Rank. It is one of the algorithms that many of us interested to know. Before I want to have my wedding website – Justproposed.com be shown on at the top of the result when users type "wedding website" as search keywords, I have looked into SEO. It is a fun area to explore. Generally speaking, if the query keywords shown in the title, it weights more, If the keyword frequency is higher, it ranks higher..blah blah. However, I know Google has weighted a lot on the links. It is not just purely based on the document that you have. How to obtain the additional information during the crawling is beyond the scope of this article.

Get your hand dirty

Look into this great article.

The thing this article doesn’t mention is that you need to create you dataDir and indexDir folders under C and drag a list of html files into the dataDir before you start the web server. If you drag new htmls into it, you need to clean up your indexDir and restart your web server in order to rebuild the indexes.

I have got the application up and running. It is nice trial. My next step is to enhance this example. I will do the following:

1. Use Flex as search interface
2. Use Solr to expose the Lucene search engine as Web Service.
3. Have Flex calls my search engine via REST.
4. Display the result on Flex.

After I have my new enhancements working, I would do the following:

1. Look into how Lucene do the indexing
2. Look into Nutch,. So I can have it crawled some sites and put the htmls in dataDir for me automatically.

How Lucene Indexes the documents?

Yes. I haven’t forgot to answer the question 4. Here is the article that answers your question. To summarize, here are several key points I extracted from this article.

1. Content Extraction – Lucene only takes text for index. So, it provides different types of parsers to extract content from different types of document like word, html, doc, pdf and etc. If you have other type of document that you cannot find a parser, you take the responsibility to extract the content out for Lucene. This article shows you how to use Digester to extract content out from XML and feed Lucene. If you have a large pool of XML for content extraction, you need to pay attention on the parsing time. There is someone who has done this and obtain some performance number as reference. However, the article was a bit outdated.
2. Content Preprocessing – Analyzer is used to extract the token from your text content to be indexed. Before text is indexed, it is passed through an [code]]czo4OlwiQW5hbHl6ZXJcIjt7WyYqJl19[[/code]. [code]]czo4OlwiQW5hbHl6ZXJcIjt7WyYqJl19[[/code]s are in charge of extracting indexable tokens out of text to be indexed, and eliminating the rest. Lucene comes with a few different [code]]czo4OlwiQW5hbHl6ZXJcIjt7WyYqJl19[[/code] implementations. Some of them deal with skipping stop words (frequently-used words that don't help distinguish one document from the other, such as "a," "an," "the," "in," "on," etc.), some deal with converting all tokens to lowercase letters, so that searches are not case-sensitive, and so on.
3. Indexing - IndexWriter is the key component in the indexing process. This class will use Analyzer that you passed in as parameter to create a new index or open an existing index and add documents to it. You need to set up fields and documents and feed them to the IndexWriter to do the job. Like the code below, you fetches a list of .txt files and its metadata like path from a directory and feed them for IndexWriter. IndexWriter will index them one after one.
4. Configuration - You can configure IndexWriter to achieve better performance via increasing the buffer size because the bottleneck normally happen during the IO of the index files.
5. Lucene uses inverted index concept. An inverted index is an inside-out arrangement of documents in which terms take center stage. Each term points to a list of documents that contain it. On the contrary, in a forwarding index, documents take the center stage, and each document refers to a list of terms it contains. You can use an inverted index to easily find which documents contain certain terms. Lucene uses an inverted index as its index structure.

 
for(int i = 0; i < textFiles.length; i++){
      if(textFiles[i].isFile() >> textFiles[i].getName().endsWith(".txt")){
        Reader textReader = new FileReader(textFiles[i]);
        Document document = new Document();
        document.add(Field.Text("content",textReader));
        document.add(Field.Keyword("path",textFiles[i].getPath()));
        indexWriter.addDocument(document);
      }
}

Lucene offers four different types of fields from which a developer can choose: Keyword,UnIndexed,UnStored,and Text.

Keyword fields are not tokenized, but are indexed and stored in the index verbatim. This field is suitable for fields whose original value should be preserved in its entirety, such as URLs, dates, personal names, Social Security numbers, telephone numbers, etc.

UnIndexed fields are neither tokenized nor indexed, but their value is stored in the index word for word. This field is suitable for fields that you need to display with search results, but whose values you will never search directly. Because this type of field is not indexed, searches against it are slow. Since the original value of a field of this type is stored in the index, this type is not suitable for storing fields with very large values, if index size is an issue.

UnStored fields are the opposite of UnIndexed fields. Fields of this type are tokenized and indexed, but are not stored in the index. This field is suitable for indexing large amounts of text that does not need to be retrieved in its original form, such as the bodies of Web pages, or any other type of text document.

Text fields are tokenized, indexed, and stored in the index. This implies that fields of this type can be searched, but be cautious about the size of the field stored as Text field.

Conclusion

To use Lucene, there are 3 main concepts you need to grasp. There are:

1. Indexer - create search engine indexes
2. Analyzer - Split text into tokens that make sense for the search engine. The structure is like document -> a sequence of fields and each field is name/value pair -> tokens. Field values may be stored, indexed or analyzed/ tokenize, (and, now, vectored). The lecture note from Doug Cutting will give you more detail.
   
3. Searcher

You may think of using grep to achieve or database to achieve what Lucene does. Grep is powerful Linux tool, however, if you want it to search on files with several MB in size, you will see that the tool is inefficient. The reason is grep doesn't prepare the indexes of your files ahead of the time you do the search. Database can do indexing but not so sophisticated as Lucene in your varchar field. Oracle may provide one but I am not familiar with it. One key thing to remember: Lucene is open source, free and does the job extremely well. Why bother to dig into Oracle costly solution?

Lucene has given us a rich search engine capability on our web application. It has many features that I haven't got a chance to discuss them all in this article. I will continue to write more articles on this topic as my research moves forward. Have a nice day!

Reference

The blog of the Lucene and Solr creator - Doug Cutting