【正文】 120090508 17:40:06,109 INFO MapTask numReduceTasks: 1省略插件加載日志……20090508 17:40:06,312 INFO Configuration found resource at file:/D:/work/workspace/nutch_crawl/bin/20090508 17:40:06,343 INFO FetchScheduleFactory Using FetchSchedule impl: 20090508 17:40:06,343 INFO AbstractFetchSchedule defaultInterval=259200020090508 17:40:06,343 INFO AbstractFetchSchedule maxInterval=777600020090508 17:40:06,343 INFO MapTask = 10020090508 17:40:06,437 INFO MapTask data buffer = 79691776/9961472020090508 17:40:06,437 INFO MapTask record buffer = 262144/32768020090508 17:40:06,453 WARN RegexURLNormalizer can39。attempt_local_0002_r_000000_039。attempt_local_0002_m_000000_039。attempt_local_0001_r_000000_039。attempt_local_0001_m_000000_039。查看submitJob方法,首先獲得jobid,執(zhí)行configureCommandLineOptions方法后會(huì)在上邊的臨時(shí)文件夾生成一個(gè)system文件夾,日志如下:20090508 15:41:36,640 INFO Injector Injector: starting20090508 15:41:37,031 INFO Injector Injector: crawlDb: 20090508/crawldb20090508 15:41:37,781 INFO Injector Injector: urlDir: urls20090508 15:52:41,734 INFO Injector Injector: Converting injected urls to crawl db entries.20090508 15:56:22,203 INFO JvmMetrics Initializing JVM Metrics with processName=JobTracker, sessionId=20090508 16:08:20,796 WARN JobClient Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.20090508 16:08:20,984 WARN JobClient No job jar file set. User classes may not be found. See JobConf(Class) or JobConfsetJar(String).20090508 16:24:42,593 INFO FileInputFormat Total input paths to process : 120090508 16:38:29,437 INFO FileInputFormat Total input paths to process : 120090508 16:38:29,546 INFO MapTask numReduceTasks: 120090508 16:38:29,562 INFO MapTask = 10020090508 16:38:29,687 INFO MapTask data buffer = 79691776/9961472020090508 16:38:29,687 INFO MapTask record buffer = 262144/32768020090508 16:38:29,718 INFO PluginRepository Plugins: looking in: D:\work\workspace\nutch_crawl\bin\plugins20090508 16:38:29,921 INFO PluginRepository Plugin Autoactivation mode: [true]20090508 16:38:29,921 INFO PluginRepository Registered Plugins:20090508 16:38:29,921 INFO PluginRepository the nutch core extension points (nutchextensionpoints)20090508 16:38:29,921 INFO PluginRepository Basic Query Filter (querybasic)20090508 16:38:29,921 INFO PluginRepository Basic URL Normalizer (urlnormalizerbasic)20090508 16:38:29,921 INFO PluginRepository Basic Indexing Filter (indexbasic)20090508 16:38:29,921 INFO PluginRepository Html Parse Plugin (parsehtml)20090508 16:38:29,921 INFO PluginRepository Site Query Filter (querysite)20090508 16:38:29,921 INFO PluginRepository Basic Summarizer Plugin (summarybasic)20090508 16:38:29,921 INFO PluginRepository HTTP Framework (lib)20090508 16:38:29,921 INFO PluginRepository Text Parse Plugin (parsetext)20090508 16:38:29,921 INFO PluginRepository Passthrough URL Normalizer (urlnormalizerpass)20090508 16:38:29,921 INFO PluginRepository Regex URL Filter (urlfilterregex)20090508 16:38:29,921 INFO PluginRepository Http Protocol Plugin (protocol)20090508 16:38:29,921 INFO PluginRepository XML Response Writer Plugin (responsexml)20090508 16:38:29,921 INFO PluginRepository Regex URL Normalizer (urlnormalizerregex)20090508 16:38:29,921 INFO PluginRepository OPIC Scoring Plugin (scoringopic)20090508 16:38:29,921 INFO PluginRepository CyberNeko HTML Parser (libnekohtml)20090508 16:38:29,921 INFO PluginRepository Anchor Indexing Filter (indexanchor)20090508 16:38:29,921 INFO PluginRepository JavaScript Parser (parsejs)20090508 16:38:29,921 INFO PluginRepository URL Query Filter (queryurl)20090508 16:38:29,921 INFO PluginRepository Regex URL Filter Framework (libregexfilter)20090508 16:38:29,921 INFO PluginRepository JSON Response Writer Plugin (responsejson)20090508 16:38:29,921 INFO PluginRepository Registered ExtensionPoints:20090508 16:38:29,921 INFO PluginRepository Nutch Summarizer ()20090508 16:38:29,921 INFO PluginRepository Nutch Protocol ()20090508 16:38:29,921 INFO PluginRepository Nutch Analysis ()20090508 16:38:29,921 INFO PluginRepository Nutch Field Filter ()20090508 16:38:29,921 INFO PluginRepository HTML Parse Filter ()20090508 16:38:29,921 INFO PluginRepository Nutch Query Filter ()20090508 16:38:29,921 INFO PluginRepository Nutch Search Results Response Writer ()20090508 16:38:29,921 INFO PluginRepository Nutch URL Normalizer ()20090508 16:38:29,921 INFO PluginRepository Nutch URL Filter ()20090508 16:38:29,921 INFO PluginRepository Nutch Online Search Results Clustering Plugin ()20090508 16:38:29,921 INFO PluginRepository Nutch Indexing Filter ()20090508 16:38:29,921 INFO PluginRepository Nutch Content Parser ()20090508 16:38:29,921 INFO PluginRepository Nutch Scoring ()20090508 16:38:29,921 INFO PluginRepository Ontology Model Loader ()20090508 16:38:29,968 INFO Configuration found resource at file:/D:/work/workspace/nutch_crawl/bin/20090508 16:38:29,984 WARN RegexURLNormalizer can39。最終，各個(gè)獨(dú)立的segment索引被合并為一個(gè)最終的索引index（步驟10）。　　1. 創(chuàng)建一個(gè)新的WebDb (admin db create).　　2. 將抓取起始URLs寫(xiě)入WebDB中 (inject).　　3. 根據(jù)WebDB生成fetchlist并寫(xiě)入相應(yīng)的segment(generate).　　4. 根據(jù)fetchlist中的URL抓取網(wǎng)頁(yè) (fetch).　　5. 根據(jù)抓取網(wǎng)頁(yè)更新WebDb (updatedb).　　6. 循環(huán)進(jìn)行3－5步直至預(yù)先設(shè)定的抓取深度。另外Nutch遵守Robots Exclusion Protocol。l indexs：存放每次下載的獨(dú)立索引目錄l index：符合Lucene格式的索引目錄，是indexs里所有index合并后的完整索引 抓取過(guò)程概述引用到的類(lèi)主要有以下9個(gè)： 用來(lái)給抓取數(shù)據(jù)庫(kù)添加URL的插入器 用來(lái)生成待下載任務(wù)列表的生成器 完成抓取特定頁(yè)面的抓取器 負(fù)責(zé)內(nèi)容提取和對(duì)下級(jí)URL提取的內(nèi)容進(jìn)行解析的解析器 負(fù)責(zé)數(shù)據(jù)庫(kù)管理的數(shù)據(jù)庫(kù)管理工具 負(fù)責(zé)鏈接管理 負(fù)責(zé)創(chuàng)建索引的索引器 刪除重復(fù)數(shù)據(jù) 對(duì)當(dāng)前下載內(nèi)容局部索引和歷史索引進(jìn)行合并的索引合并器 抓取過(guò)程分析Cr