【導(dǎo)讀】蜘蛛的功能和設(shè)計(jì)要求。在對(duì)網(wǎng)絡(luò)蜘蛛系統(tǒng)結(jié)構(gòu)和工作原理所作分析的基礎(chǔ)上，研究了頁面。爬取、解析等策略和算法，并使用Java實(shí)現(xiàn)了一個(gè)網(wǎng)絡(luò)蜘蛛的程序，對(duì)其運(yùn)行結(jié)果做了分析。

　　

【正文】 // (bb)。 (a href=+url++bb+/a)。 } } } return true。 } //執(zhí)行實(shí)際的搜索操作 public ArrayList String crawl(String startUrl, int maxUrls, String searchString,boolean limithost,boolean caseSensitive ) { (searchString=+searchString)。//搜索字符串 HashSet String crawledList = new HashSet String()。 LinkedHashSet String toCrawlList = new LinkedHashSet String()。 if (maxUrls 1) { (Invalid Max URLs value.)。 (Invalid Max URLs value.)。 } if (() 1) { (Missing Search String.)。 (Missing search String)。 } if (() 0) { (err!!!)。 return errorList。 } // 從開始 URL 中移出 startUrl = removeWwwFromUrl(startUrl)。 (startUrl)。 while (() 0) { if (maxUrls != 1) { if (() == maxUrls) { break。 } } // Get URL at bottom of the list. String url = ().next()。 // Remove URL from the to crawl list. (url)。 // Convert string url to URL object. URL verifiedUrl = verifyUrl(url)。 // Skip URL if robots are not allowed to access it. //if (!isRobotAllowed(verifiedUrl)) { // continue。 // } // 增加已處理的 URL 到 crawledList (url)。 //(提示搜索過的 :+verifiedUrl)。//提示搜索過的 url String pageContents = downloadPage(verifiedUrl)。 if (pageContents != null amp。amp。 () 0){ // 從頁面中獲取有效的鏈接 //ArrayList String links =retrieveLinks(verifiedUrl, pageContents, crawledList,limitHost)。 HtmlParser parser=new HtmlParser(pageContents)。 ArrayList String links=()。 //(test message!)。 // for(int j=0。j()。j++){ // ( (j))。///測試是否取出連接 // } (links)。//添加新取得的連接 if (searchStringMatches(url,pageContents, searchString,caseSensitive)) { //(url)。 (該字段存在于： +url)。//輸出找到的地址 } } } return result。 } // 主函數(shù) public static void main(String[] args) { if(!=3){ (Usage:java SearchCrawler startUrl maxUrl searchString)。 return。 } int max=(args[1])。 myspider crawler = new myspider(args[0],max,args[2])。 Thread search=new Thread(crawler)。 (Start searching...)。 (result:)。 ()。 /**/ } } 五、系統(tǒng)測試 搜索測試以默認(rèn)開始網(wǎng)頁作為起始頁面，輸入搜索字符串： 百度 ，如下圖所示： 點(diǎn)擊搜索，開始執(zhí)行。執(zhí)行完畢，出現(xiàn)結(jié)果： 搜索成功。 六、結(jié)論 本系統(tǒng)開發(fā)過程中用到了許多學(xué)過的知識(shí)，具體說來有數(shù)據(jù)結(jié)構(gòu)、 java 語言程序設(shè)計(jì)、軟件工程、優(yōu)化理論等等。在編程中發(fā)現(xiàn)這些學(xué)科相互聯(lián)系，相輔相成，在以后更加復(fù)雜的系統(tǒng)中肯定會(huì)涉及到更多、更復(fù)雜的學(xué)科，需要我們認(rèn)真學(xué)習(xí)和掌握的東西實(shí)在是太多了。 本軟件只是對(duì)搜索引擎功能的基本實(shí)現(xiàn)，在技術(shù)方面還存在許多不足之處。當(dāng)然在這突飛猛進(jìn)的信息時(shí)代，技術(shù)的更新更是日新月異，所以其中有的思想不可能完全適應(yīng)于各種實(shí)際情況。由于本人學(xué)習(xí)軟件工程的時(shí)間不長，程序的設(shè)計(jì)方面不夠規(guī)范，有些簡單的想法卻用了很長的代碼來實(shí)現(xiàn)造成了代碼冗余 ，還有部分想法沒有實(shí)現(xiàn)。我將在今后的學(xué)習(xí)中不斷完善自己。 致謝 當(dāng)這篇論文最終完成的時(shí)候，我要向給予過我熱情幫助和悉心指導(dǎo)的老師和師兄們致以真誠的謝意。 首先，我要感謝我的導(dǎo)師，感謝他帶給我來學(xué)習(xí)的機(jī)會(huì)，感謝他對(duì)我學(xué)術(shù)上的悉心指導(dǎo)，感謝他對(duì)我生活上的關(guān)懷和體貼。導(dǎo)師是不僅是我學(xué)業(yè)上的導(dǎo)師，更是生活中讓我敬佩的學(xué)者、長者。給我留下深刻印象的，是他知識(shí)的淵博、治學(xué)態(tài)度的嚴(yán)謹(jǐn)、誨人不倦的學(xué)者風(fēng)范，是他謙遜待人、處處關(guān)心別人的長者風(fēng)度，是他勤奮忘我的工作態(tài)度、精益求精的治學(xué)作風(fēng)。特別是老師做大事的風(fēng)范和氣度，尤其 讓我欽佩。這里我要再次感謝老師。 在本文的最后，我要再次感謝我的導(dǎo)師，同時(shí)也向與老師一樣辛勤育人，無私付出的各位導(dǎo)師、師長致以深深的謝意。 參考文獻(xiàn) [1] 李曉明，悶宏飛，王繼民．搜索引擎 — — 原理、技術(shù)與系統(tǒng) [M]．北京：科學(xué)出版社，2020． [2] Heaton J．網(wǎng)絡(luò)機(jī)器人 Java 編程指南 [M]．北京：電子工業(yè)出版社， 2020． [3] 崔澤永，常曉燕．搜索引擎的 Web Robot 的技術(shù)與優(yōu)化 [J]．微機(jī)發(fā)展， 2020， 14(4)： 100— 102． [4] Shafer C．?dāng)?shù)據(jù)結(jié)構(gòu)與算法分析 (JAVA 版 )[M]．北京：電子工業(yè)出版社， 2020． [5]賈年．基于移動(dòng) Agent 搜索引擎的研究與實(shí)現(xiàn) [D]．成都：電子科技大學(xué)， 2020． [6]賈年．移動(dòng) Agent 研究 [J]．四川工業(yè)學(xué)院學(xué)報(bào)， 2020， 23(3)： 51— 54． [7]S． Charkabarti． Mimng the Web’ s Link structure[J]． IEEE Computer， 2020， 32(8)： 60— 67． [8]徐寶文，張衛(wèi)豐．搜索引擎與信息獲取技術(shù)【 M】．北京：清華大學(xué)出版社， 2020 168。 Itˇ s a wonder Retana wouldnˇ t give us enough light to see the horses by,〃 one picador said. 168。 He knows weˇ ll be happier if we donˇ t get too good a look at these skins,〃 another pic answered. 168。 This thing Iˇ m on barely keeps me off the ground,〃 the first picador said. 168。 Well, theyˇ re horses.〃 168。 Sure, theyˇ re horses.〃 They talked, sitting their gaunt horses in the dark. Zurito said nothing. He had the only steady horse of the lot. He had tried him, wheeling him in the corrals, and he responded to the b it and the spurs. He had ta ken the bandage off his right eye and cut the strings where they had tied his ears tight shut at the base. He was a good, solid horse, solid on his legs. That was all he needed. He intended to ride him all through the corrida. He had already, since he had mounted, sitting in the halfdark in the big, quilted saddle, waiting for the paseo, piced through the whole corrida in his mind. The other picadors went on talking on both sides of him. He did not hear them. The tw o matadors stood together in front of their three peones, their capes furled over their left arms in the same fashion. Manuel was thinki ng about the three lads in back of him. They were all three Madrile?os, like Hernandez, boys about nieen. One of them, a gypsy, serious, aloof, and darkfaced, he liked the look of. He turned. 168。 Whatˇ s your name, kid?〃 he asked the gypsy. 168。 Fuentes,〃 the gypsy said. What he said. Am I sick? TH had settled it that Mr. Darcy would bring his sister to visit her the very day after her reaching Pemberley。 and was conse quently resolved not to be out of sight of the inn the whole of that morning. But her conclusion was false。 for on the very morning after their own arrival at Lambton, these visitors came. They had been walking about the place with some of their new friends, and were just returned to the inn to dress themselves for dining with the same family, when the sound of a carriage drew them to a window, and they saw a gentleman and lady in a curricle, driving up the street. Elizabeth, immediately recognising the livery, guessed wha t it meant, and im parted no small degree of surprise to her relations by acquainting them with the honour which she expected. Her uncle and aunt were all amazement。 and the embarrassment of her manner