摘要

伴随着各种电子产品和网络的蓬勃发展，互联网上的数据正在以爆炸般的模式增长着，信息量逐渐从最小的字节增长成为TB乃至PB级的大数据。越来越多的信息量成为了一个难题，对于摒弃它们还是从中获取价值的这个选择题也逐渐有了显而易见的答案。但是面对如此严峻的形势，我们该如何从大量的数据中获取有价值的信息，并且对这些数据进行高效、精准的处理呢。就在此时，人们将目光移向了Hadoop，一个开源的能够对大量数据进行分布式处理的文件系统和并行计算编程模型。

本文将要介绍Hadoop的伪分布式集群具体的搭建过程和以HDFS为代表的分布式文件存储系统的数据存储平台的设计与实现。并且对于HDFS本身仍面临的一些问题进行性能优化和解决。

首先Hadoop适宜在Linux系统上进行，所以先介绍centos系统的安装和配置过程；然后在centos系统下进行Hadoop伪分布式集群的搭建和基本配置；之后进行一系列的文件、数据的存储和读取应用；最后对于HDFS目前仍具有的一些问题进行分析和假定并对其进行性能优化。

关键字：大数据；Hadoop;Access;centos;HDFS

ABSTRACT

With the rapid development of various electronic products and networks, the data on the Internet is increasing in an exploding mode, and the amount of information is gradually increasing from the smallest bytes to the TB and even the PB big data.More and more information is becoming a puzzle, and the answer to the multiple choice question of abandoning them and gaining value from them is becoming apparent.But in the face of such a severe situation, how can we obtain valuable information from a large number of data and process these data efficiently and accurately?At this point, people turned their eyes to Hadoop, an open source that could do a lot of data. Distributed file system and parallel computing programming model.

This paper will introduce the construction process of Hadoop pseudo-distributed cluster and the design and implementation of data storage platform of distributed file storage system represented by HDFS.And the HDFS itself is still faced with some problems for performance optimization and solution.

First, Hadoop is suitable for Linux system, so the installation and configuration process of centos system is introduced first, then the construction and basic configuration of Hadoop pseudo-distributed cluster under centos system, a series of files, data storage and reading applications are carried out.Finally, some problems still existing in HDFS are analyzed and hypothesized and its performance is optimized.

摘要

ABSTRACT

目录