Improved Compressed Indexes for Full-Text Document Retrieval

We give new space/time tradeoffs for compressed indexes that answer document retrieval queries on general sequences. On a collection of D documents of total length 72, current approaches require at least vertical bar CSA vertical bar + O(n lg D/1g lg D) or 2 vertical bar CSA vertical bar + o(n) bits of space, where CSA is a full-text index. Using monotone mininum perfect hash functions, we give new algorithms for document listing with frequencies and top-k document retrieval using just vertical bar CSA vertical bar + O(n lg lg lg D) bits. We also improve current solutions that use 2 vertical bar CSA vertical bar + o(n) bits, and consider other problems such as colored range listing, top-k, most important documents, and computing arbitrary frequencies.