【正文】 I 摘 要 網(wǎng)絡(luò)爬蟲(chóng)（ Web Crawler），通常被稱為爬蟲(chóng)，是搜索引擎的重要組成部分。隨著信息技術(shù)的飛速進(jìn)步，作為搜索引擎的一個(gè)組成部分 ——網(wǎng)絡(luò)爬蟲(chóng)，一直是研究的熱點(diǎn)，它的好壞會(huì)直接決定搜索引擎的未來(lái)。目前，網(wǎng)絡(luò)爬蟲(chóng)的研究包括 Web 搜索策略研究的研究和網(wǎng)絡(luò)分析的算法，兩個(gè)方向，其中在 Web 爬蟲(chóng)網(wǎng)絡(luò)搜索主題是一個(gè)研究方向，根據(jù)一些網(wǎng)站的分析算法，過(guò)濾不相關(guān)的鏈接，連接到合格的網(wǎng)頁(yè)，并放置在一個(gè)隊(duì)列被抓取。 把互聯(lián)網(wǎng)比喻成一個(gè)蜘蛛網(wǎng)，那么 Spider 就是在網(wǎng)上爬來(lái)爬去的蜘蛛。網(wǎng)絡(luò)蜘蛛是通過(guò)網(wǎng)頁(yè)的鏈接地址來(lái)尋找 網(wǎng)頁(yè)，從網(wǎng)站某一個(gè)頁(yè)面（通常是首頁(yè)）開(kāi)始，讀取網(wǎng)頁(yè)的內(nèi)容，找到在網(wǎng)頁(yè)中的其它鏈接地址，然后通過(guò)這些鏈接地址尋找下一個(gè)網(wǎng)頁(yè)，這樣一直循環(huán)下去，直到把這個(gè)網(wǎng)站所有的網(wǎng)頁(yè)都抓取完為止。如果把整個(gè)互聯(lián)網(wǎng)當(dāng)成一個(gè)網(wǎng)站，那么網(wǎng)絡(luò)爬蟲(chóng)就可以用這個(gè)原理把互聯(lián)網(wǎng)上所有的網(wǎng)頁(yè)都抓取下來(lái)。 關(guān)鍵詞 ： 網(wǎng)絡(luò)爬蟲(chóng)； Linux Socket； C/C++。多線程；互斥鎖 II Abstract Web Crawler, usually called Crawler for short, is an important part of search engine. With the highspeed development of information, Web Crawler the search engine can not lack of which is a hot research topic those years. The quality of a search engine is mostly depended on the quality of a Web Crawler. Nowadays, the direction of researching Web Crawler mainly divides into two parts: one is the searching strategy to web pages。 the other is the algorithm of analysis URLs. Among them, the research of TopicFocused Web Crawler is the trend. It uses some webpage analysis strategy to filter topicless URLs and add fit URLs into URLWAIT queue. The metaphor of a spider web inter, then Spider spider is crawling around on the Inter. Web spider through web link address to find pages, starting from a one page website (usually home), read the contents of the page, find the address of the other links on the page, and then look for the next Web page addresses through these links, so has been the cycle continues, until all the pages of this site are crawled exhausted. If the entire Inter as a site, then you can use this Web crawler principle all the pages on the Inter are crawling down.. Keywords:Web crawler； Linux Socket； C/C++。 Multithreading。Mutex III 目 錄 摘 要 ............................................................................ I 第一章 概 述 ................................................................... 1 課題背景 ................................................................................................................................................. 1 網(wǎng)絡(luò)爬蟲(chóng)的歷史和分類 ......................................................................................................................... 1 網(wǎng)絡(luò)爬蟲(chóng)的歷史 .......................................................................................................................... 1 網(wǎng)絡(luò)爬蟲(chóng)的分類 .......................................................................................................................... 2 網(wǎng)絡(luò)爬蟲(chóng)的發(fā)展趨勢(shì) ............................................................................................................................. 3 系統(tǒng)開(kāi)發(fā)的必要性 ................................................................................................................................. 3 本文的組織結(jié)構(gòu) ..................................................................................................................................... 3 第二章 相關(guān)技術(shù)和工具綜述 ......................................................... 5 網(wǎng)絡(luò)爬蟲(chóng)的定義 ..................................................................................................................................... 5 網(wǎng)頁(yè)搜索策略介紹 ................................................................................................................................. 5 廣度優(yōu)先搜索策略 ...................................................................................................................... 5 相關(guān)工具介紹 ......................................................................................................................................... 6 操作系統(tǒng) ...................................................................................................................................... 6 軟件配置 ...................................................................................................................................... 6 第三章 網(wǎng)絡(luò)爬蟲(chóng)模型 的分析和概要設(shè)計(jì) ................................................ 8 網(wǎng)絡(luò)爬蟲(chóng)的模型分析 ............................................................................................................................. 8 網(wǎng)絡(luò)爬蟲(chóng)的搜索策略 ............................................................................................................................. 8 網(wǎng)絡(luò)爬蟲(chóng)的概要設(shè)計(jì) ........................................................................................................................... 10 第 四章 網(wǎng)絡(luò)爬蟲(chóng)模型的設(shè)計(jì)與實(shí)現(xiàn) ................................................... 12 網(wǎng)絡(luò)爬蟲(chóng)的總體設(shè)計(jì) ........................................................................................................................... 12 網(wǎng)絡(luò)爬蟲(chóng)的具體設(shè)計(jì) ........................................................................................................................... 12 URL 類設(shè)計(jì)及標(biāo)準(zhǔn)化 URL ....................................................................................................... 12 爬取網(wǎng)頁(yè) .................................................................................................................................... 13 網(wǎng)頁(yè)分析 .................................................................................................................................... 14 網(wǎng)頁(yè)存儲(chǔ) .................................................................................................................................... 14 Linux socket 通信 ....................................................................................................................... 16 EPOLL 模型及其使用 ............................................................................................................... 20 POSIX 多線程及其使用 ............................................................................................................ 22 第五章 程序運(yùn)行及結(jié)果分析 ........................................................ 25 Makefile 及編譯 ..................