發(fā)布時(shí)間：2018-07-18 08:20

【摘要】：隨著技術(shù)革新，從傳統(tǒng)互聯(lián)網(wǎng)廣泛應(yīng)用到最近幾年爆炸式增長的移動互聯(lián)網(wǎng)及物聯(lián)網(wǎng)的起步，依附在網(wǎng)絡(luò)上的數(shù)據(jù)越來越龐大，據(jù)國際數(shù)據(jù)公司（IDC），EMC公司等最新研究表明，在移動網(wǎng)絡(luò)設(shè)備（智能手機(jī)）及視頻監(jiān)控的推動下，目前全球的數(shù)據(jù)總量已經(jīng)到達(dá)4870億GB，而在2007年的相關(guān)報(bào)告表示當(dāng)年的數(shù)據(jù)總量才1610億GBN。這些數(shù)據(jù)中包括了大量的電話、郵件、照片、網(wǎng)絡(luò)社交、新聞以及視頻內(nèi)容。如何有效利用這些數(shù)據(jù)，給用戶提供優(yōu)質(zhì)的用戶體驗(yàn)，在科學(xué)研究上，隨著大量的帶有GPS的數(shù)據(jù)設(shè)備采集的數(shù)據(jù)的匯集和研究都是急切需要得到技術(shù)支持。 在今后，隨著物聯(lián)網(wǎng)的進(jìn)一步發(fā)展，有大量服務(wù)是基于位置的服務(wù)(Location Based Service，LBS)的數(shù)據(jù)產(chǎn)生，也將有大量請求服務(wù)基于LBS或個(gè)人偏愛。這也致使未來需要提供的服務(wù)應(yīng)該同現(xiàn)在統(tǒng)一服務(wù)相區(qū)分，針對不同的用戶屬性提供不同的服務(wù)資源，在浩若煙海的數(shù)據(jù)中如何最快、最正確地提供能滿足用戶需求、并且是在低成本狀態(tài)下完成變得尤為重要。本文的主要工作如下： 1.對現(xiàn)有的大數(shù)據(jù)存儲進(jìn)行分析研究，包括了GFS文件系統(tǒng)實(shí)現(xiàn)原理及Hadoop框架，為在大數(shù)據(jù)服務(wù)中的數(shù)據(jù)存儲做技術(shù)支撐。 2.針對數(shù)據(jù)特性進(jìn)行研究，并研究相關(guān)數(shù)據(jù)挖掘算法對非結(jié)構(gòu)化數(shù)據(jù)進(jìn)行歸納挖掘，同時(shí)針對結(jié)構(gòu)化數(shù)據(jù)進(jìn)行分析讀取，實(shí)現(xiàn)數(shù)據(jù)熱點(diǎn)權(quán)值的初始化。本文的主要成果如下： 1.在現(xiàn)有的Hadoop框架基礎(chǔ)上，，二次開發(fā)，修改Hadoop的存儲備份算法，通過資源權(quán)值列表來實(shí)現(xiàn)資源文件在修改后的框架中按照數(shù)據(jù)資源熱點(diǎn)權(quán)值存儲。 2.本文在Hadoop基礎(chǔ)上設(shè)計(jì)出的資源文件權(quán)值初始化算法，當(dāng)資源文件訪問量增加或通過外部導(dǎo)入設(shè)置熱點(diǎn)權(quán)值，實(shí)現(xiàn)對數(shù)據(jù)資源文件的存儲節(jié)點(diǎn)計(jì)算并分發(fā)。 3.實(shí)現(xiàn)了在庫資源文件的熱點(diǎn)不斷變化過程中的資源文件權(quán)值調(diào)整，并通該權(quán)值來實(shí)現(xiàn)數(shù)據(jù)資源文件的重新分發(fā)及調(diào)整，實(shí)現(xiàn)了服務(wù)資源的擴(kuò)增算法和服務(wù)資源收縮算法。
[Abstract]:With the technological innovation, from the traditional Internet of Internet to the explosive growth of the mobile Internet in recent years and the start of the Internet of things, the data attached to the network is becoming larger and larger. According to the latest research, such as International data Corporation (IDC) and EMC, Driven by mobile network devices (smartphones) and video surveillance, the global data total has reached 487 billion GB, compared with 161 billion GBN reported in 2007. This data includes a lot of phone, email, photos, social networking, news and video content. How to effectively use these data to provide users with a high-quality user experience, in scientific research, with a large number of data with GPS data collection and research are urgently needed technical support. In the future, with the further development of the Internet of things, there will be a large number of services based on location based LBS (location based Service LBS) data generation, there will be a large number of request services based on LBS or personal preference. This also leads to the need for future services to be differentiated from the current unified services, to provide different service resources for different user attributes, and how to provide the fastest and most correctly available services to meet the needs of users in the vast sea of data. And it is particularly important to do so in a low-cost state. The main work of this paper is as follows: 1. This paper analyzes and studies the existing big data storage, including the implementation principle of big data file system and the framework of Hadoop, which provides technical support for data storage in big data services. 2. Based on the research of data characteristics, the related data mining algorithms are studied to induce and mine the unstructured data. At the same time, the structured data is analyzed and read to realize the initialization of the hot data weights. The main results of this paper are as follows: 1. Based on the existing Hadoop framework, the storage backup algorithm of Hadoop is redeveloped, and the resource file is stored in the modified framework according to the hot spot weight value of the data resource by the list of resource weights. 2. A resource file weight initialization algorithm based on Hadoop is designed in this paper. When the access of resource file is increased or the hot spot weight is set by external import, the storage node of data resource file can be calculated and distributed. This paper realizes the adjustment of resource file weight value in the process of changing hot spot of database resource file, and realizes the redistribution and adjustment of data resource file by using this weight value, and realizes the expansion algorithm of service resource and the algorithm of service resource contraction.
【學(xué)位授予單位】：成都理工大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2013
【分類號】：TP333

【參考文獻(xiàn)】

相關(guān)期刊論文 前3條

1 林鴻飛,馬雅彬;基于聚類的文本過濾模型[J];大連理工大學(xué)學(xué)報(bào);2002年02期

2 劉越;;云計(jì)算綜述與移動云計(jì)算的應(yīng)用研究[J];信息通信技術(shù);2010年02期

3 孫廣中;肖鋒;熊曦;;MapReduce模型的調(diào)度及容錯(cuò)機(jī)制研究[J];微電子學(xué)與計(jì)算機(jī);2007年09期

相關(guān)碩士學(xué)位論文 前4條

1 霍樹民;基于Hadoop的海量影像數(shù)據(jù)管理關(guān)鍵技術(shù)研究[D];國防科學(xué)技術(shù)大學(xué);2010年

2 曹風(fēng)兵;基于Hadoop的云計(jì)算模型研究與應(yīng)用[D];重慶大學(xué);2011年

3 劉叢山;基于Hadoop的文本分類研究[D];上海交通大學(xué);2012年

4 邱榮太;基于Hadoop平臺的Map-Reduce應(yīng)用研究[D];河南理工大學(xué);2009年

本文編號：2131312