Improved self-indexing inverted files for full-text retrieval

Self-index is a promising way to improve retrieval time-and-space efficiency by compression index files. An improved inverted file self-index called IFSI is proposed for full-text information retrieval. IFSI includes two level indexes: the first level index which contains a subset of the documents that are likely to be returned as top results; and the second level index which includes the surplus documents. IFSI can create a skipped index on each compressed posting list with very little or no storage overhead with efficient coding scheme. IFSI also supports efficient incremental updates with allocating free space efficiently at the tail of post lists based on statistics-based approach. Detailed simulation results and comparison with other schemes prove that the proposed IFSI can not only greatly reduce decompress time, but also simultaneously allow extremely fast query processing. © 2009 Binary Information Press March, 2009.