【正文】 I 摘 要 網(wǎng)絡(luò)爬蟲（ Web Crawler），通常被稱為爬蟲，是搜索引擎的重要組成部分。隨著信息技術(shù)的飛速進(jìn)步，作為搜索引擎的一個組成部分 ——網(wǎng)絡(luò)爬蟲，一直是研究的熱點，它的好壞會直接決定搜索引擎的未來。目前，網(wǎng)絡(luò)爬蟲的研究包括 Web 搜索策略研究的研究和網(wǎng)絡(luò)分析的算法，兩個方向，其中在 Web 爬蟲網(wǎng)絡(luò)搜索主題是一個研究方向，根據(jù)一些網(wǎng)站的分析算法，過濾不相關(guān)的鏈接，連接到合格的網(wǎng)頁，并放置在一個隊列被抓取。 把互聯(lián)網(wǎng)比喻成一個蜘蛛網(wǎng)，那么 Spider 就是在網(wǎng)上爬來爬去的蜘蛛。網(wǎng)絡(luò)蜘蛛是通過網(wǎng)頁的鏈接地址來尋找 網(wǎng)頁，從網(wǎng)站某一個頁面（通常是首頁）開始，讀取網(wǎng)頁的內(nèi)容，找到在網(wǎng)頁中的其它鏈接地址，然后通過這些鏈接地址尋找下一個網(wǎng)頁，這樣一直循環(huán)下去，直到把這個網(wǎng)站所有的網(wǎng)頁都抓取完為止。如果把整個互聯(lián)網(wǎng)當(dāng)成一個網(wǎng)站，那么網(wǎng)絡(luò)爬蟲就可以用這個原理把互聯(lián)網(wǎng)上所有的網(wǎng)頁都抓取下來。 關(guān)鍵詞 ： 網(wǎng)絡(luò)爬蟲； Linux Socket； C/C++。多線程；互斥鎖 II Abstract Web Crawler, usually called Crawler for short, is an important part of search engine. With the highspeed development of information, Web Crawler the search engine can not lack of which is a hot research topic those years. The quality of a search engine is mostly depended on the quality of a Web Crawler. Nowadays, the direction of researching Web Crawler mainly divides into two parts: one is the searching strategy to web pages。 the other is the algorithm of analysis URLs. Among them, the research of TopicFocused Web Crawler is the trend. It uses some webpage analysis strategy to filter topicless URLs and add fit URLs into URLWAIT queue. The metaphor of a spider web inter, then Spider spider is crawling around on the Inter. Web spider through web link address to find pages, starting from a one page website (usually home), read the contents of the page, find the address of the other links on the page, and then look for the next Web page addresses through these links, so has been the cycle continues, until all the pages of this site are crawled exhausted. If the entire Inter as a site, then you can use this Web crawler principle all the pages on the Inter are crawling down.. Keywords:Web crawler； Linux Socket； C/C++。 Multithreading。Mutex III 目 錄 摘 要 ............................................................................ I 第一章 概 述 ................................................................... 1 課題背景 ................................................................................................................................................. 1 網(wǎng)絡(luò)爬蟲的歷史和分類 ......................................................................................................................... 1 網(wǎng)絡(luò)爬蟲的歷史 .......................................................................................................................... 1 網(wǎng)絡(luò)爬蟲的分類 .......................................................................................................................... 2 網(wǎng)絡(luò)爬蟲的發(fā)展趨勢 ............................................................................................................................. 3 系統(tǒng)開發(fā)的必要性 ................................................................................................................................. 3 本文的組織結(jié)構(gòu) ..................................................................................................................................... 3 第二章 相關(guān)技術(shù)和工具綜述 ......................................................... 5 網(wǎng)絡(luò)爬蟲的定義 ..................................................................................................................................... 5 網(wǎng)頁搜索策略介紹 ................................................................................................................................. 5 廣度優(yōu)先搜索策略 ...................................................................................................................... 5 相關(guān)工具介紹 ......................................................................................................................................... 6 操作系統(tǒng) ...................................................................................................................................... 6 軟件配置 ...................................................................................................................................... 6 第三章 網(wǎng)絡(luò)爬蟲模型 的分析和概要設(shè)計 ................................................ 8 網(wǎng)絡(luò)爬蟲的模型分析 ............................................................................................................................. 8 網(wǎng)絡(luò)爬蟲的搜索策略 ............................................................................................................................. 8 網(wǎng)絡(luò)爬蟲的概要設(shè)計 ........................................................................................................................... 10 第 四章 網(wǎng)絡(luò)爬蟲模型的設(shè)計與實現(xiàn) ................................................... 12 網(wǎng)絡(luò)爬蟲的總體設(shè)計 ........................................................................................................................... 12 網(wǎng)絡(luò)爬蟲的具體設(shè)計 ........................................................................................................................... 12 URL 類設(shè)計及標(biāo)準(zhǔn)化 URL ....................................................................................................... 12 爬取網(wǎng)頁 .................................................................................................................................... 13 網(wǎng)頁分析 .................................................................................................................................... 14 網(wǎng)頁存儲 .................................................................................................................................... 14 Linux socket 通信 ....................................................................................................................... 16 EPOLL 模型及其使用 ............................................................................................................... 20 POSIX 多線程及其使用 ............................................................................................................ 22 第五章 程序運行及結(jié)果分析 ........................................................ 25 Makefile 及編譯 ..................