【正文】 三七數(shù)據(jù)大數(shù)據(jù)技術(shù)解決方案北京三七數(shù)據(jù)技術(shù)有限公司2022 年 7 月 31 日目 錄1 概述 ..........................................................................................................................................62 面臨的挑戰(zhàn) ..............................................................................................................................6 數(shù)據(jù)采集 ...............................................................................................................................10 數(shù)據(jù)清洗 ...............................................................................................................................10 數(shù)據(jù)存儲(chǔ) ...............................................................................................................................12 數(shù)據(jù)并行處理 .......................................................................................................................12 數(shù)據(jù)分析 ...............................................................................................................................12 可視化 ...................................................................................................................................12 傳統(tǒng)解決方案的分析 ............................................................................................................123 相關(guān)技術(shù)的研究 .....................................................................................................................12 參考模型框架 .......................................................................................................................12 數(shù)據(jù)采集 ...............................................................................................................................12 結(jié)構(gòu)化數(shù)據(jù)的采集 .......................................................................................................12 半結(jié)構(gòu)化數(shù)據(jù)的采集 ...................................................................................................12 非結(jié)構(gòu)化文本數(shù)據(jù)中信息的抽取 ...............................................................................15 數(shù)據(jù)清洗和數(shù)據(jù)質(zhì)量的保證 ................................................................................................15 數(shù)據(jù)質(zhì)量的概念及分類 ...............................................................................................15 數(shù)據(jù)清洗的原理 ...........................................................................................................18 單數(shù)據(jù)源中的數(shù)據(jù)清洗 ...............................................................................................20 數(shù)據(jù)的集成和融合 ...............................................................................................................37 多數(shù)據(jù)源集成問題的分類 ...........................................................................................38 數(shù)據(jù)標(biāo)準(zhǔn)化的研究 .......................................................................................................40 數(shù)據(jù)集成的流程 ...........................................................................................................41 多數(shù)據(jù)源中重復(fù)實(shí)體的清理 .......................................................................................41 數(shù)據(jù)不一致性問題的研究 ...........................................................................................43 數(shù)據(jù)的存儲(chǔ)和處理 ...............................................................................................................43 并行和分布式處理理論 ...............................................................................................43 并行 RDBMS...............................................................................................................47 Hadoop.......................................................................................................................49 Hadoop 擴(kuò)展和優(yōu)化 ..................................................................................................53 NoSQL.........................................................................................................................58 查詢優(yōu)化 .....................................................................................................................110 大數(shù)據(jù)中的數(shù)據(jù)挖掘 ..........................................................................................................112 傳統(tǒng)數(shù)據(jù)挖掘概述 .....................................................................................................112 大數(shù)據(jù)時(shí)代數(shù)據(jù)挖掘發(fā)展新趨勢 .............................................................................120 WEB 數(shù)據(jù)挖掘 ...........................................................................................................124 超數(shù)據(jù)集成挖掘方法與技術(shù)研究 .............................................................................153 數(shù)據(jù)挖掘網(wǎng)格技術(shù) .....................................................................................................186 大規(guī)模機(jī)器學(xué)習(xí) .................................................................................................................208 機(jī)器學(xué)習(xí)概述 .............................................................................................................209 擴(kuò)展機(jī)器學(xué)習(xí)的必要性 .............................................................................................211 面臨的挑戰(zhàn) .................................................................................................................213 概率圖模型 .................................................................................................................214 集成學(xué)習(xí) .....................................................................................................................215 可視化和可視化分析 ..........................................................................................................225 概述 .............................................................................................................................225 可視化技術(shù) .................................................................................................................225 可視化分析 .................................................................................................................227 文本的可視化分析 ..............................................................