【作者】 黄仲清；

【导师】 袁毅； 许鑫；

【作者基本信息】 华东师范大学 ， 情报学， 2010， 硕士

【摘要】 在信息时代的大背景下,互联网信息以超乎想象的速度迅猛增加,信息爆炸、信息过载使人们陷入了信息时代的新困境。如何在海量的互联网信息资源中快速有效地获取所需信息成为亟待解决的一个重要问题。与此同时,信息用户对信息的需求正逐渐呈现出主题化、领域化、专业化和个性化的趋势。如何满足这些主题化的信息需求也是当前的重要课题。正是基于这样的背景,本文首先研究和比较了当前适用于互联网主题信息定向采集的各类理论、技术和信息采集方案,包括通用引擎和垂直引擎策略、主题信息采集技术、中文自动切分词技术、大规模文本计算技术等支撑性技术；在此基础上提出了通用搜素引擎与垂直搜索引擎相结合的互联网主题定向采集策略,利用基于领域的主题词表生成和优化方法确定主题范围,采用文本相似度计算算法进行系统的文本处理。在确定了采集策略和底层技术后,即对互联网主题定向采集系统进行了框架设计。本文分析和改进了采集系统中的三个关键技术,包括提出了多种防屏蔽技术相结合的网络采集防屏蔽解决方案；改进了一种基于文本密度的网页正文抽取方法；采用了基于分词的向量空间模型和余弦夹角公式实现了基于内容的标题去重。文中以实例介绍了采用模拟浏览器技术自动登录网站的防屏蔽实现方法,改进的网页正文抽取方法适用于新闻类网页,是一种通用性较强、性能优越的正文抽取算法。在网页去重技术方面,本文主要介绍了网页URL比对去重技术和基于内容的去重策略,并采用基于分词的向量空间模型和余弦夹角公式实现了基于内容的标题去重,给出了其核心算法。最后,本文从互联网舆情研究出发,分析了舆情研究对网络信息采集和分析的需求,并针对网络舆情的分支——网络侨情,开发了互联网侨情采集系统。确定了侨情领域主题词表、种子网站,实现了从URL抓取、网页源文件抓取、标题和正文抽取、网页去重等一系列工作流程。为今后继续对网络舆情信息的分析和处理打下了基础。更多还原

【Abstract】 The amount of information on the Internet is dramatically increasing with the coming of the information age. Due to the information explosion and information overloading, human-beings are facing a lot of new difficulties. Furthermore, the requirements of information users are increasingly subject-oriented, specific, professional, and individual. The thesis proposed a strategy for subject acquisition in order to meet people’s diversified information requirements.Firstly this thesis has done comparison between different theories, technologies and information acquisition solutions which are related to subject oriented acquiring of Internet information. Secondly, an integrated information acquisition strategy was put forward based on the merging of the two strategies of general search engine and vertical search engine. Thirdly the framework of the directed subject acquisition system was designed after indentifying the acquisition strategies and underlying technologies.Three key technologies related to the system were analyzed and improved, including:anti-shielding solution integrated with different technologies to avoid shielding, web content extraction based on text density, eliminate duplication technology based on VSM and cosine angle formula.Lastly, a directed subject information acquisition system on overseas Chinese was developed. The process includes several steps from identifying the thesaurus of overseas Chinese Internet information and the seed sites, subject acquisition of URL and web source files, to the extraction of titles and text. The thesis will contribute to the research on Internet public opinion for the future.更多还原