基于适应概念漂移的垃圾邮件过滤系统设计与实现

【作者】 党建军；

【导师】 张凤荔；

【作者基本信息】 电子科技大学 ， 信息与通信工程， 2010， 硕士

【摘要】 电子邮件作为当今一种方便、快捷的互联网信息交流方式,受到越来越多人的青睐。但是垃圾邮件的出现,并且日益严峻,使这种便利的方式给人带来了烦恼。反垃圾邮件技术已成为互联网信息领域的一个研究热点,基于内容的反垃圾邮件过滤技术更是一种成熟而且有效的技术方案。基于朴素贝叶斯的垃圾邮件过滤方法是当前基于文本内容过滤方法的有效算法之一。随着时间的变化,垃圾邮件的特征也在不断的改变,然而传统的训练模型必须重新进行训练才能适应新的邮件特征的改变。因此,传统的朴素贝叶斯过滤方法必须与其它技术结合才能有效的适应新特征的变化。本文提出的实例选择-分类器加权集成算法,是采用数据挖掘领域的流问题解决方案来适应邮件流的问题的解决思路,成为当前的研究热点。本方法是在研究朴素贝叶斯的基本原理,分析其优缺点的基础上,基于传统分类器的静态特性,将概念漂移的思想应用到垃圾邮件过滤系统上,在中文的CCERT“2005-Jul”数据集上,取得了不错的效果,不仅在从精度上,更重要的适应性上,从不适应到适应,从精度低到精度高,完成了一个动态的适应过程。1)本文首先分析了中文词语的特点和常见的词典结构,解读了朴素贝叶斯算法的基本原理,概念漂移的基本思想,同时给出了通用分类算法评价标准。2)在第三章,描述了整个系统的总体目标,以及本模块的总体架构,并给予了模块概括性的描述。3)在第四章,阐释模块内部各个功能点的详细设计和实现,提供了伪代码级的说明了详述。4)在测试和分析章节,首先详述了中文和英文的语料集,并就该模块系统的参数和数据集选取给予了详细的说明,在概念漂移发生或未发生时,同传统分类器,在精度和适应性上的对比,并做出了详细的分析。综上所述,本系统提出对传统领域的垃圾邮件过滤模型的适应性研究是一个有实践价值、理论意义的尝试。更多还原

【Abstract】 Email is popular as one of convenient and economical ways of communication available by the internet.,however, spam appears, and even worse, becomes harassment for more and more persons and companies.Anti-spam technology has been hot pot in the realm of the Internet..the technology of filtering the spam based on the content is one of effective and efficiency methods.Naive bayes text classification technique has a dominant place in the area of spam filtering for its good categorization, high precision. As time goes, so goes the feature of mail, especially for spam.however, when the new feature appears, the traditional model of filtering the spam must be trained by the new mail which contains the changed features, therefore, the traditional models or methods of trained should be bound to be grafted on new methods or thinking to adjust to the constantly changing environment. The paper shows the methods who we call it combined instance selection-weighted of classifier algorithm ,from the domain of mining data streams,as a thinking for spam filtering. The method is prompted based on the basic principle of naive bayes,and the strong points and weaknesses; Based on static characteristic of the traditional models , the paper combines the idea of concept drift with the traditional models. The data set is“2005-Jul”provided by CCERT. The result is more efficient,not only on the precision, but also on the adaptation, the experiment reveals the process of dynamic adaptation.1) The paper analyzes the characteristic of Chinese character and the structure of dictionaries, then, gives a general overview of the basic principle of naive bayes, and the basic thought of concept drift, at the same time, the general criterions of classification.2)In chapter three,the paper gives the overall object of the system and the whole structure of the algorithm, describes the modules of the algorithm.3) In chapter four, the paper gives a specific description of Function Points, even pseudo codes.4) In the section of test and analysis chapter, we firstly induces the datasets of English and Chinese, explains the choice of datasets for the test, gives the results of experiment, including the diversification when concept drift takes place or not, not only on the precision,most of important, on the adaptation.At last ,the paper offers the analysis for the readers .To the conclusion, the paper offers one new attempt of practical merit and groping meaning to traditional trained model when the environment changes.更多还原