谷歌大規(guī)模排序?qū)嶒灥臍v史[翻譯](編輯修改稿)
2025-07-12 22:10 本頁面
【文章內(nèi)容簡介】 several days tuning the number of shards, various buffer sizes, prefetching/writeahead strategies, page cache usage, etc. Weblogged about the result here. The bottleneck ended up being writing the threeway replicated output GFS, which was the standard we used at Google at the time. Anything less would create a high risk of data loss.2008年我們第一次把注意力集中于調(diào)整。我們花費幾天的時間來調(diào)整分區(qū)的數(shù)量，緩沖區(qū)的大小，預(yù)取/預(yù)寫策略，頁面緩存使用等。我們曾經(jīng)在這個博客里記錄過結(jié)果。最終的瓶頸是寫三路復(fù)制的GFS輸出文件，這是當(dāng)時我們在谷歌使用的標(biāo)準(zhǔn)。任何事情的缺失都會造成數(shù)據(jù)丟失的高風(fēng)險。20102010(1PB, hours, TB/min, MB/s/worker)1PB, , TB/min, MB/s/workerFor this test, we used the new version of the GraySort benchmark that uses inpressible data. In the previous years, while we were reading/writing 1PB from/to GFS, the amount of data actually shuffled was only about 300TB, because the ASCII format used the previous years presses well.在這個測試中，我們使用了一種新的不可壓縮的GraySort基準(zhǔn)的數(shù)據(jù)版本。在前幾年，當(dāng)我們讀/寫1PB GFS文件時，實際上混排的數(shù)據(jù)只有300TB，因為前幾年的數(shù)據(jù)是用ASCII格式壓縮好的。This was also the year of Colossus, the next generation distributed storage system at Google, the successor to GFS. We no longer had the corruption issues we encountered before with GFS. We also used ReedSolomon encoding (a new Colossus feature) for output that allowed us to reduce the total amount of data written from 3 petabytes (for threeway replication) to about petabytes. For the first time, we also validated that the output was correct.這也是谷歌使用Colossus的一年，新一代的分布式存儲方式取代了GFS。我們不再有我們遇到過的GFS文件污染的問題。我們還使用了ReedSolomon編碼(Colossus新特征)作為輸出，這種編碼允許我們減少數(shù)據(jù)的總量。這也是第一次，我們驗證了輸出的結(jié)果是正確的。To reduce the impact of stragglers, we used a dynamic sharding technique called reduce subsharding. This is the precursor to fully dynamic sharding used inDataflow.為了減少人的影響，我們采用了一種叫做減少殘余碎片的動態(tài)分區(qū)技術(shù)。這也是數(shù)據(jù)流采用全動態(tài)分區(qū)的先兆。20112011(1PB, hours, TB/min, MB/s/worker)1PB, , TB/min, MB/s/workerThis year we enjoyed faster networking and started to pay more attention to p
法律信息相關(guān)推薦
鄂ICP備17016276號-1