基于hadoop的大規(guī)模數(shù)據(jù)排序算法(編輯修改稿)
2025-06-12 20:20 本頁(yè)面
【文章內(nèi)容簡(jiǎn)介】 waiting after reserving slots (ms)=0 11/11/06 04:38:37 INFO : Launched map tasks=2 11/11/06 04:38:37 INFO : SLOTS_MILLIS_REDUCES=0 11/11/06 04:38:37 INFO : File Input Format Counters 11/11/06 04:38:37 INFO : Bytes Read=0 11/11/06 04:38:37 INFO : File Output Format Counters 11/11/06 04:38:37 INFO : Bytes Written=1000 11/11/06 04:38:37 INFO : FileSystemCounters 11/11/06 04:38:37 INFO : HDFS_BYTES_READ=158 11/11/06 04:38:37 INFO : FILE_BYTES_WRITTEN=41776 11/11/06 04:38:37 INFO : HDFS_BYTES_WRITTEN=1000 11/11/06 04:38:37 INFO : MapReduce Framework 11/11/06 04:38:37 INFO : Map input records=10 11/11/06 04:38:37 INFO : Spilled Records=0 11/11/06 04:38:37 INFO : Map input bytes=10 11/11/06 04:38:37 INFO : Map output records=10 11/11/06 04:38:37 INFO : SPLIT_RAW_BYTES=158 如果 產(chǎn)生 1G 的數(shù)據(jù)，由于數(shù)據(jù)塊是 64 M 一塊，這會(huì)被分成 16 個(gè)數(shù)據(jù)塊，當(dāng)運(yùn)行 terasort時(shí)會(huì)有 64 個(gè) map task。 但是我們產(chǎn)生的是 100M 的數(shù)據(jù)，從下面的輸出中可以看到一些信息， Launched map tasks=2 apple@ubuntu:~/$ bin/hadoop jar teragen 1000000 terasort/100Minput Generating 1000000 using 2 maps with step of 500000 11/11/06 04:41:11 INFO : Running job: job_202111060257_0019 11/11/06 04:41:12 INFO : map 0% reduce 0% 11/11/06 04:41:43 INFO : map 10% reduce 0% 11/11/06 04:41:58 INFO : map 11% reduce 0% 11/11/06 04:42:04 INFO : map 50% reduce 0% 11/11/06 04:42:27 INFO : map 96% reduce 0% 6 11/11/06 04:42:33 INFO : map 100% reduce 0% 11/11/06 04:42:45 INFO : Job plete: job_202111060257_0019 11/11/06 04:42:45 INFO : Counters: 15 11/11/06 04:42:45 INFO : Job Counters 11/11/06 04:42:45 INFO : SLOTS_MILLIS_MAPS=121603 11/11/06 04:42:45 INFO : Total time spent by all reduces waiting after reserving slots (ms)=0 11/11/06 04:42:45 INFO : Total time spent by all maps waiting after reserving slots (ms)=0 11/11/06 04:42:45 INFO : Launched map tasks=2 11/11/06 04:42:45 INFO : SLOTS_MILLIS_REDUCES=0 11/11/06 04:42:45 INFO : File Input Format Counters 11/11/06 04:42:45 INFO : Bytes Read=0 11/11/06 04:42:45 INFO : File Output Format Counters 11/11/06 04:42:45 INFO : Bytes Written=100000000 11/11/06 04:42:45 INFO : FileSystemCounters 11/11/06 04:42:45 INFO : HDFS_BYTES_READ=167 11/11/06 04:42:45 INFO : FILE_BYTES_WRITTEN=41780 11/11/06 04:42:45 INFO : HDFS_BYTES_WRITTEN=100000000 11/11/06 04:42:45 INFO : MapReduce Framework 11/11/06 04:42:45 INFO : Map input records=1000000 11/11/06 04:42:45 INFO : Spilled Records=0 11/11/06 04:42:45 INFO : Map input bytes=1000000 11/11/06 04:42:45 INFO : Map output records=1000000 11/11/06 04:42:45 INFO : SPLIT_RAW_BYTES=167 執(zhí)行 terasort 程序，將會(huì)執(zhí)行 2 個(gè) MapTask，特別容易死在這兒。從下面輸出的時(shí)間可以看出，僅僅排序 100M 數(shù)據(jù)就從用來(lái) 15 分鐘。 apple@ubuntu:~/$ bin/hadoop jar terasort terasort/100Minput terasort/100Moutput 11/11/06 04:44:24 INFO : starting 11/11/06 04:44:26 INFO : Total input paths to process : 2 11/11/06 04:44:30 INFO : Loaded the nativehadoop library 11/11/06 04:44:30 INFO : Successfully loaded amp。 initialized nativezlib library 11/11/06 04:44:30 INFO : Got brandnew pressor Making 1 from 100000 records Step size is 11/11/06 04:44:32 INFO : Total input paths to process : 2 11/11/06 04:44:34 INFO : Running job: job_202111060257_0020 11/11/06 04:44:35 INFO : map 0% reduce 0% 11/11/06 04:49:00 INFO : map 1% reduce 0% 11/11/06 04:49:16 INFO : map 2% reduce 0% 11/11/06 04:49:19 INFO : map 4% reduce 0% 11
醫(yī)療健康相關(guān)推薦
鄂ICP備17016276號(hào)-1