【作者】 陈奋；

【导师】 吴顺祥；

【作者基本信息】 厦门大学 ， 系统工程， 2007， 硕士

【摘要】 网络爬虫是一个可以从因特网上自动提取网页的系统,它为搜索引擎从万维网上下载网页,是搜索引擎的重要组成。通用搜索引擎的网络爬虫一般是从几个种子URL链接开始进行全盘爬行,而专业领域搜索引擎的网络爬虫除了通用网络爬虫的基本功能外,还能够对链接以及页面内容进行识别,因此称作聚焦网络爬虫。聚焦网络爬虫并不追求大的覆盖,而将目标定为抓取与某一特定主题内容相关的网页,为面向主题的用户查询准备数据资源。聚焦网络爬虫已经成为搜索引擎技术领域的一个研究热点,对于专业领域的搜索产生重要的作用。本论文从聚焦的另一个角度——“过滤”上来研究网络爬虫技术,称这种类型的网络爬虫为“过滤型网络爬虫”。论文首先介绍了网络爬虫所起的作用以及网络爬虫技术的发展现状;接着在从两个方面来研究过滤型网络爬虫技术:(1)从链接过滤上,提出了链接群体的概念,根据不同的网站类型将链接群体分为单模式链接群体和多模式链接群体,同时在分析了传统的链接过滤算法的基础上,提出了基于规则匹配的链接过滤算法;(2)从内容过滤上,主要从以下三个方面来研究:(a)提出了一种基于网站内容特征的网站类型辨识方法,(b)使用一种基于标签权重的网页文本特征词选择算法,在此基础上构建网页文本的空间向量模型,并将该向量模型跟已经设定好的主题向量模型进行相似度计算,从而形成基于向量空间模型的主题过滤算法,(c)在分析非结构化数据分类过程的基础上,使用了基于朴素贝叶斯分类器的主题类别过滤算法;最后设计并实现了一个过滤型网络爬虫系统,并详细介绍了系统的整体设计流程、系统结构以及系统几个关键模块和关键技术。更多还原

【Abstract】 Web crawler is a system which can automatically get web pages from Internet。It helps searching engine download web pages, so it is an important part of searching engine. Web crawler of normal searching engine starts working from some seeding links, and that web crawler of searching engine for special domain is able to identify links and content of web pages except functions of normal web crawler, so we call it focused web crawler. The main goals of focused web crawler are to get more web pages which are correlative with a certain topic and prepare data for users querying. The focused web crawler has been became a researching hotspot in technology domain of searching engine.We research the focused web crawler from another aspect—“filtrating technology”, so we call this web crawler as the filtrating web crawler. Firstly, we introduce the main function of the web crawler and the present condition of technology of web crawler; secondly, we research the technology of filtrating web crawler from two aspects: (1)from filtrating links, we give the concept of links’colony and classify links’colony as single pattern and multiple pattern; at the same time we give the filtrating links algorithm after analyzing the traditional algorithm;(2)from filtrating content of web pages, we research it from three aspects: (a)put forward a method to differentiate the style of website basing the characteristic of content,(b) use a method basing on calculating the weight of tag to select the characteristic words of web pages, and then we construct the VSM of web pages to calculate the similarity with the topic VSM which we have prepared,(c) basing on analyzing the process of classifying non-structural data, we use the native bayes classifier to differentiate the topic types of web pages; lastly, we design and implement a filtrating web crawler system, and introduce the main module and technology of this system.更多还原