摘 要
随着步入互联网时代，Web2.0和物联网技术飞速发展，全球每年产生越来越多的数据，如何从这些海量的数据中去帮助人们发现他们感兴趣的信息，同时为商家带来更大的收益，达到消费者与商家的共赢，极具现实意义。信息通信技术的发展，数据量爆炸式增长，大数据技术研究逐渐升温，备受关注。带来根本改变的并非海量数据自身，而是如何利用数据，从数据中挖掘潜在的价值，让数据更好的服务于用户。
本次设计是将豆瓣电影信息生成、豆瓣电影信息传送、豆瓣电影信息分析，最后落地并可视化展示。完成的业务需求是统计电影TOPN信息，按照地市统计电影TOPN信息，按照流量统计TOPN信息。
系统从需求分析、结构设计、数据库设计，最后到系统实现，分别实现了数据采集、数据收集集群、消息队列、大数据集群、spark数据的处理和落地、java web从数据库读取数据并可视化的功能。本文从系统描述、系统分析、系统设计、系统实现和系统测试几个方面对系统进行了描述和开发。系统使用了大数据的各部框架来辅助完成数据采集和分析功能。系统使用了hadoop集群和spark混用的模式，豆瓣电影采集使用了flume框架对豆瓣电影进行采集处理，消息队列使用了kafka框架来搭建，使用zookeeper进行集群容错性管理。最后Spark集群上使用了Spark SQL来对大数据进行离线批处理。在可视化的过程中使用了echarts开源框架等技术进行实现。
关键字：Spark SQL；离线批处理；豆瓣电影采集；MySQL数据库；Echarts
Abstract
Along with the rapid development of Web2.0 and Internet of things technology in the Internet era, more and more data are produced every year. How to help people discover the information they are interested in from these massive data, and at the same time merchants. To achieve a win-win consumer and business, very practical significance. With the development of information and communication technology, the amount of data increases explosively, and the research of big data technology is gradually heating up. What brings the fundamental change is not the massive data itself, but how to use the data, excavate the potential value from the data, and make the data better serve the user.
This design is the Douban film information generation, Douban film information transmission, Douban film information analysis, and finally landing and visual display. The completed business requirements are statistics of film TOPN information, according to the city statistics of film TOPN information, according to traffic statistics TOPN information.
From requirements analysis, structural design, database design, Finally, to the system implementation, The functions of data acquisition, data collection cluster, message queue, big data cluster, processing and landing of spark data, java web reading data from database and visualizing are realized respectively. This paper describes and develops the system from several aspects: system description, system analysis, system design, system implementation and system testing. The system uses the big data framework to assist in data acquisition and analysis. The system uses a mix of hadoop clusters and spark, Douban film collection uses a flume framework to collect and process Douban films, Message queues are built using kafka frameworks, Cluster fault tolerance management using zookeeper. finally Spark Spark SQL is used on the cluster for offline batch processing of big data. echarts open source framework is used in the process of visualization.
Keywords: Spark SQL; off-line batch processing; Douban movie collection; MySQL database; Echarts
