【摘要】 为探究水质分析与大数据技术结合的可行方案,以MySQL+Hive+SparkR为主体框架搭建一整套从数据输入、存储、调度到应用的SparkR水质分析平台。设置室内培养模拟人工湖藻类生长实验组及其重复实验组,监测各项指标数据,通过SparkR平台,在本地应用Adaptive-Lasso算法识别出对照组和苦草组藻类生长主要影响因子,并建立回归方程进行验证,在集群分布式部署GBTs藻类预测模型,经重复试验验证预测模型未来3天的相对误差均值分别为15.3%、14.8%。更多还原

【Abstract】 In order to explore the feasible scheme of combining water quality analysis with big data technology, a set of SparkR water quality analysis platform from data input, storage, dispatch to application is built with MySQL+Hive+SparkR as the main framework. Seting up experiment groups indoors to simulate algae growth of artificial lake and its repeated experimental groups, various indicators was monitored. Based on SparkR platform, the adaptive-Lasso algorithm was applied locally to identify the main influencing factors of algae growth in control group and validate the regression equation, and GBTs algae prediction model was deployed in the cluster, and repeated experiments showed that the relative error of GBRT algae prediction models in the next three days was 15.3% and 14.8% respectively.更多还原