【文章內(nèi)容簡介】 ed urls into crawl db.20090508 17:10:01,015 INFO JvmMetrics Cannot initialize JVM Metrics with processName=JobTracker, sessionId= already initialized20090508 17:10:15,953 WARN JobClient Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.20090508 17:10:16,156 WARN JobClient No job jar file set. User classes may not be found. See JobConf(Class) or JobConfsetJar(String).20090508 17:12:15,296 INFO FileInputFormat Total input paths to process : 120090508 17:13:40,296 INFO FileInputFormat Total input paths to process : 120090508 17:13:40,406 INFO MapTask numReduceTasks: 120090508 17:13:40,406 INFO MapTask = 10020090508 17:13:40,515 INFO MapTask data buffer = 79691776/9961472020090508 17:13:40,515 INFO MapTask record buffer = 262144/32768020090508 17:13:40,546 INFO MapTask Starting flush of map output20090508 17:13:40,765 INFO MapTask Finished spill 020090508 17:13:40,765 INFO TaskRunner Task:attempt_local_0002_m_000000_0 is done. And is in the process of miting20090508 17:13:40,765 INFO LocalJobRunner file:/tmp/hadoopAdministrator/mapred/temp/injecttemp474192304/part00000:0+14320090508 17:13:40,765 INFO TaskRunner Task 39。attempt_local_0002_m_000000_039。 done.20090508 17:13:40,796 INFO LocalJobRunner 20090508 17:13:40,796 INFO Merger Merging 1 sorted segments20090508 17:13:40,796 INFO Merger Down to the last mergepass, with 1 segments left of total size: 53 bytes20090508 17:13:40,796 INFO LocalJobRunner 20090508 17:13:40,906 WARN NativeCodeLoader Unable to load nativehadoop library for your platform... using builtinjava classes where applicable20090508 17:13:40,906 INFO CodecPool Got brandnew pressor20090508 17:13:40,906 INFO TaskRunner Task:attempt_local_0002_r_000000_0 is done. And is in the process of miting20090508 17:13:40,906 INFO LocalJobRunner 20090508 17:13:40,906 INFO TaskRunner Task attempt_local_0002_r_000000_0 is allowed to mit now20090508 17:13:40,921 INFO FileOutputCommitter Saved output of task 39。attempt_local_0002_r_000000_039。 to file:/D:/work/workspace/nutch_crawl/20090508/crawldb/189656774520090508 17:13:40,921 INFO LocalJobRunner reduce reduce20090508 17:13:40,937 INFO TaskRunner Task 39。attempt_local_0002_r_000000_039。 done.20090508 17:13:46,781 INFO JobClient Running job: job_local_000220090508 17:14:55,125 INFO JobClient Job plete: job_local_000220090508 17:14:59,328 INFO JobClient Counters: 1120090508 17:14:59,328 INFO JobClient File Systems20090508 17:14:59,328 INFO JobClient Local bytes read=10387520090508 17:14:59,328 INFO JobClient Local bytes written=20938520090508 17:14:59,328 INFO JobClient MapReduce Framework20090508 17:14:59,328 INFO JobClient Reduce input groups=120090508 17:14:59,328 INFO JobClient Combine output records=020090508 17:14:59,328 INFO JobClient Map input records=120090508 17:14:59,328 INFO JobClient Reduce output records=120090508 17:14:59,328 INFO JobClient Map output bytes=4920090508 17:14:59,328 INFO JobClient Map input bytes=5720090508 17:14:59,328 INFO JobClient Combine input records=020090508 17:14:59,328 INFO JobClient Map output records=120090508 17:14:59,328 INFO JobClient Reduce input records=120090508 17:17:30,984 INFO JvmMetrics Cannot initialize JVM Metrics with processName=JobTracker, sessionId= already initialized20090508 17:20:02,390 INFO Injector Injector: done generate方法描述：從爬取數(shù)據(jù)庫中生成新的segment,然后從中生成待下載任務(wù)列表(fetchlist).(fs, lock, force)。,猜測作用是防止crawldb的數(shù)據(jù)被修改,真實(shí)作用有待驗(yàn)證.接著執(zhí)行的過程和上邊大同小異,可參考上邊步驟,日志如下:20090508 17:37:18,218 INFO Generator Generator: Selecting bestscoring urls due for fetch.20090508 17:37:18,625 INFO Generator Generator: starting20090508 17:37:18,937 INFO Generator Generator: segment: 20090508/segments/2009050817313720090508 17:37:19,468 INFO Generator Generator: filtering: true20090508 17:37:22,312 INFO Generator Generator: topN: 5020090508 17:37:51,203 INFO Generator Generator: jobtracker is 39。local39。, generating exactly one partition.20090508 17:39:57,609 INFO JvmMetrics Cannot initialize JVM Metrics with processName=JobTracker, sessionId= already initialized20090508 17:40:05,234 WARN JobClient Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.20090508 17:40:05,406 WARN JobClient No job jar file set. User classes may not be found. See JobConf(Class) or JobConfsetJar(String).20090508 17:40:05,437 INFO FileInputFormat Total input paths to process : 120090508 17:40:06,062 INFO FileInputFormat Total input paths to process : 120090508 17:40:06,109 INFO MapTask numReduceTasks: 1省略插件加載日志……20090508 17:40:06,312 INFO Configuration found resource at file:/D:/work/workspace/nutch_crawl/bin/20090508 17:40:06,343 INFO FetchScheduleFactory Using FetchSchedule impl: 20090508 17:40:06,343 INFO AbstractFetchSchedule defaultInterval=259200020090508 17:40:06,343 INFO AbstractFetchSchedule maxInterval=777600020090508 17:40:06,343 INFO MapTask = 10020090508 17:40:06,437 INFO MapTask data buffer = 79691776/9961472020090508 17:40:06,437 INFO MapTask record buffer = 262144/32768020090508 17:40:06,453 WARN RegexURLNormalizer can39。t find rules for scope 39。partition39。, using default20090508 17:40:06,453 INFO MapTask Starting flush of map output20090508 17:40:06,625 INFO MapTask Finished spill 020090508 17:40:06,640 INFO TaskRunner Task:attempt_local_0003_m_000000_0 is done. And is in the process of miting20090508 17:40:06,640 INFO LocalJobRunner file:/D:/work/workspace/nutch_crawl/20090508/crawldb/current/