Fixing the Threshold for Effective Detection of Near Duplicate Web Documents in Web Crawling

被引：0
|
作者：
Narayana, V. A. [1 ]
Premchand, P. [1 ]
Govardhan, A. [1 ]
机构：
[1] CMR Coll Engn & Technol, Dept Comp Sci & Engn, Hyderabad, Andhra Pradesh, India
来源：
ADVANCED DATA MINING AND APPLICATIONS, ADMA 2010, PT I | 2010年 / 6440卷
关键词：
Fingerprint; Similarity score; Near-duplicate; Web crawling and Threshold;
D O I：
暂无
中图分类号：
TP18 [人工智能理论];
学科分类号：
081104 ; 0812 ; 0835 ; 1405 ;
摘要：
The drastic development of the WWW in recent times has made the concept of Web Crawling receive remarkable significance. The voluminous amounts of web documents swarming the web have posed huge challenges to web search engines making their results less relevant to the users. The presence of duplicate and near duplicate web documents in abundance has created additional overheads for the search engines critically affecting their performance and quality which have to be removed to provide users with the relevant results for their queries. In this paper, we have presented a novel and efficient approach for the detection of near duplicate web pages in web crawling where the keywords are extracted from the crawled pages and the similarity score between two pages is calculated. The documents having similarity score greater than a threshold value are considered as near duplicates. In this paper we have fixed the threshold value.
引用
收藏
页码：169 / 180
页数：12
相关论文
共 50 条
  • [31] Duplicate image detection in a stream of web visual data
    Gadeski, Etienne
    Le Borgne, Herve
    Popescu, Adrian
    2015 13TH INTERNATIONAL WORKSHOP ON CONTENT-BASED MULTIMEDIA INDEXING (CBMI), 2015,
  • [32] Online System for Detection of Chinese Near-Duplicate Documents
    Yang, Yang
    Chen, YuQuan
    2012 6TH INTERNATIONAL CONFERENCE ON NEW TRENDS IN INFORMATION SCIENCE, SERVICE SCIENCE AND DATA MINING (ISSDM2012), 2012, : 726 - 731
  • [33] Web Document Duplicate Detection Using Fuzzy Hashing
    Figuerola, Carlos G.
    Gomez Diaz, Raquel
    Alonso Berrocal, Jose L.
    Zazo Rodriguez, Angel F.
    TRENDS IN PRACTICAL APPLICATIONS OF AGENTS AND MULTI-AGENTS SYSTEMS, 2011, 90 : 117 - 125
  • [34] Topical web crawling using weighted anchor text and web page change detection techniques
    Yadav, Divakar
    Sharma, Ak
    Gupta, J.P.
    WSEAS Transactions on Information Science and Applications, 2009, 6 (02): : 263 - 275
  • [35] Near-Duplicate Web Page Detection: An Efficient Approach Using Clustering, Sentence Feature and Fingerprinting
    Kumar, J. Prasanna
    Govindarajulu, P.
    INTERNATIONAL JOURNAL OF COMPUTATIONAL INTELLIGENCE SYSTEMS, 2013, 6 (01) : 1 - 13
  • [36] Postal address detection from web documents
    Li, C
    Qian, Z
    Meng, XF
    Liu, WY
    INTERNATIONAL WORKSHOP ON CHALLENGES IN WEB INFORMATION RETRIEVAL AND INTEGRATION, PROCEEDINGS, 2005, : 40 - 45
  • [37] Finding event-relevant content from the web using a near-duplicate detection approach
    Chang, Hung-Chi
    Wang, Jenq-Haur
    Chiu, Chih-Yi
    PROCEEDINGS OF THE IEEE/WIC/ACM INTERNATIONAL CONFERENCE ON WEB INTELLIGENCE: WI 2007, 2007, : 291 - 294
  • [38] Web-Scale Near-Duplicate Search: Techniques and Applications
    Ngo, Chong-Wah
    Xu, Changsheng
    Kraaij, Wessel
    El Saddik, Abdulmotaleb
    IEEE MULTIMEDIA, 2013, 20 (03) : 10 - 12
  • [39] Near-Duplicate Web Page Detection: An Efficient Approach Using Clustering, Sentence Feature and Fingerprinting
    J. Prasanna Kumar
    P. Govindarajulu
    International Journal of Computational Intelligence Systems, 2013, 6 : 1 - 13
  • [40] Massively Scalable Near Duplicate Detection in Streams of Documents using MDSH
    Bogen, Paul Logasa, II
    Symons, Christopher T.
    McKenzie, Amber
    Patton, Robert M.
    Gillen, Robert E.
    2013 IEEE INTERNATIONAL CONFERENCE ON BIG DATA, 2013,
12345