發(fā)布時(shí)間：2018-03-01 08:47

  本文關(guān)鍵詞： 實(shí)時(shí)數(shù)據(jù)注入 事務(wù) 分布式數(shù)據(jù)庫(kù) CLAIMS　出處：《華東師范大學(xué)》2017年碩士論文　論文類型：學(xué)位論文

【摘要】：隨著互聯(lián)網(wǎng)的發(fā)展,面對(duì)海量數(shù)據(jù)的存儲(chǔ)和管理問(wèn)題,金融等行業(yè)開(kāi)始采用分布式數(shù)據(jù)庫(kù)系統(tǒng)以提升系統(tǒng)的數(shù)據(jù)處理能力和可擴(kuò)展性。為減少因數(shù)據(jù)注入延時(shí)和分析延時(shí)帶來(lái)的數(shù)據(jù)價(jià)值下降,金融等行業(yè)要求其系統(tǒng)支持實(shí)時(shí)的數(shù)據(jù)注入和查詢;同時(shí)也需要事務(wù)性的注入和查詢以保證金融數(shù)據(jù)的正確性和一致性。而傳統(tǒng)分布式環(huán)境下的事務(wù)處理大多基于鎖和兩階段提交,無(wú)法實(shí)現(xiàn)高通量實(shí)時(shí)數(shù)據(jù)注入,另一方面NoSQL分布式系統(tǒng)不支持事務(wù)型數(shù)據(jù)注入。CLAIMS系統(tǒng)是開(kāi)源的分布式內(nèi)存OLAP系統(tǒng),支持實(shí)時(shí)數(shù)據(jù)查詢,但需要功能擴(kuò)展以實(shí)現(xiàn)事務(wù)性實(shí)時(shí)數(shù)據(jù)注入。本文針對(duì)CLAIMS系統(tǒng),設(shè)計(jì)基于元數(shù)據(jù)的事務(wù)并發(fā)控制機(jī)制,進(jìn)而提出分布式事務(wù)型實(shí)時(shí)數(shù)據(jù)注入框架,并通過(guò)大量實(shí)驗(yàn)證明其滿足金融等領(lǐng)域的需求。本文的主要貢獻(xiàn)如下:1.針對(duì)CLAIMS系統(tǒng)設(shè)計(jì)基于元數(shù)據(jù)的事務(wù)并發(fā)控制機(jī)制,通過(guò)邏輯與物理相分離來(lái)達(dá)到控制元數(shù)據(jù)即控制事務(wù)數(shù)據(jù)的目的,原子性并發(fā)控制元數(shù)據(jù)以隔離寫(xiě)事務(wù),利用事務(wù)快照以實(shí)現(xiàn)讀寫(xiě)分離,支持強(qiáng)一致性。2.在實(shí)現(xiàn)分布式事務(wù)管理器的基礎(chǔ)上,本文設(shè)計(jì)事務(wù)型分布式實(shí)時(shí)數(shù)據(jù)注入引擎,廣泛使用無(wú)鎖結(jié)構(gòu)提供高吞吐量和低延時(shí)的實(shí)時(shí)數(shù)據(jù)注入,通過(guò)事務(wù)保證注入過(guò)程的ACID特性,且具有較高的可擴(kuò)展性。3.在分布式OLAP系統(tǒng)上實(shí)現(xiàn)事務(wù)型實(shí)時(shí)數(shù)據(jù)注入框架,并通過(guò)一系列功能測(cè)試,與VoltDB進(jìn)行性能對(duì)比測(cè)試,證明本文設(shè)計(jì)的框架在滿足事務(wù)特性的同時(shí),能提供較高通量低時(shí)延的實(shí)時(shí)數(shù)據(jù)注入服務(wù)。本文提出的基于元數(shù)據(jù)的事務(wù)并發(fā)控制方式具有一定的創(chuàng)新性和學(xué)術(shù)價(jià)值,設(shè)計(jì)實(shí)現(xiàn)的實(shí)時(shí)數(shù)據(jù)注入基本滿足工業(yè)生產(chǎn)環(huán)境的需求,具有一定的實(shí)踐價(jià)值。本文的工作對(duì)我國(guó)金融行業(yè)的分布式數(shù)據(jù)實(shí)時(shí)注入具有借鑒意義。
[Abstract]:With the development of the Internet, facing the problem of massive data storage and management, In order to reduce the data value caused by data injection delay and analysis delay, finance and other industries began to adopt distributed database system to improve the data processing ability and extensibility of the system. Finance and other industries require their systems to support real-time data injection and query; At the same time, transactional injection and query are also needed to ensure the correctness and consistency of financial data. In the traditional distributed environment, most transactions are based on locks and two-phase commit, so high-throughput real-time data injection can not be realized. On the other hand, NoSQL distributed system does not support transactional data injection. CLAIMS is an open source distributed memory OLAP system, which supports real-time data query, but it needs function expansion to realize transactional real-time data injection. The transaction concurrency control mechanism based on metadata is designed, and a distributed transactional real-time data injection framework is proposed. The main contributions of this paper are as follows: 1. The transaction concurrency control mechanism based on metadata is designed for CLAIMS system. The purpose of controlling metadata, i.e. controlling transaction data, is achieved by separating logic from physics. Atomic concurrent control metadata is used to isolate write transactions, and transaction snapshots are used to realize the separation of reading and writing. Based on the implementation of distributed transaction manager, this paper designs a transactional distributed real-time data injection engine, which provides high throughput and low latency real-time data injection. The transactional real-time data injection framework is implemented on the distributed OLAP system, and a series of functional tests are carried out to compare the performance of the framework with that of VoltDB. It is proved that the proposed framework can provide real-time data injection services with high throughput and low delay while satisfying the transaction characteristics. The proposed transaction concurrency control method based on metadata has some innovative and academic value. The design and implementation of real-time data injection basically meet the needs of the industrial production environment and have certain practical value. The work of this paper has reference significance for the distributed data real-time injection in the financial industry of our country.
【學(xué)位授予單位】：華東師范大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2017
【分類號(hào)】：TP311.13

【參考文獻(xiàn)】

相關(guān)期刊論文 前2條

1 鄒先霞;賈維嘉;潘久輝;;基于數(shù)據(jù)庫(kù)日志的變化數(shù)據(jù)捕獲研究[J];小型微型計(jì)算機(jī)系統(tǒng);2012年03期

2 陸劍峰,張浩;數(shù)據(jù)倉(cāng)庫(kù)數(shù)據(jù)更新的研究及基于Oracle數(shù)據(jù)庫(kù)的開(kāi)發(fā)與應(yīng)用[J];計(jì)算機(jī)工程與應(yīng)用;2004年26期

相關(guān)碩士學(xué)位論文 前1條

1 張磊;Claims系統(tǒng)中的大表連接算法研究與實(shí)現(xiàn)[D];華東師范大學(xué);2015年

，

本文編號(hào)：1551219