【導(dǎo)讀】的功能和設(shè)計要求。在對網(wǎng)絡(luò)蜘蛛系統(tǒng)結(jié)構(gòu)和工作原理所作分析的基礎(chǔ)上，研究了頁面爬取、解。析等策略和算法，并使用Java實現(xiàn)了一個網(wǎng)絡(luò)蜘蛛的程序，對其運行結(jié)果做了分析。

　　

【正文】 // (bb)。 (a href=+url++bb+/a)。 } } } return true。 } //執(zhí)行實際的搜索操作 public ArrayList String crawl(String startUrl, int maxUrls, String searchString,boolean limithost,boolean caseSensitive ) { (searchString=+searchString)。//搜索字符串 HashSet String crawledList = new HashSet String()。 LinkedHashSet String toCrawlList = new LinkedHashSet String()。 if (maxUrls 1) { (Invalid Max URLs value.)。 (Invalid Max URLs value.)。 } if (() 1) { (Missing Search String.)。 (Missing search String)。 } if (() 0) { (err!!!)。 return errorList。 } // 從開始 URL 中移出 startUrl = removeWwwFromUrl(startUrl)。 (startUrl)。 while (() 0) { if (maxUrls != 1) { if (() == maxUrls) { break。 } } // Get URL at bottom of the list. String url = ().next()。 // Remove URL from the to crawl list. (url)。 // Convert string url to URL object. URL verifiedUrl = verifyUrl(url)。 // Skip URL if robots are not allowed to access it. //if (!isRobotAllowed(verifiedUrl)) { // continue。 // } // 增加已處理的 URL 到 crawledList (url)。 //(提示搜索過的 :+verifiedUrl)。//提示搜索過的 url String pageContents = downloadPage(verifiedUrl)。 if (pageContents != null amp。amp。 () 0){ // 從頁面中獲取有效的鏈接 //ArrayList String links =retrieveLinks(verifiedUrl, pageContents, crawledList,limitHost)。 HtmlParser parser=new HtmlParser(pageContents)。 ArrayList String links=()。 //(test message!)。 // for(int j=0。j()。j++){ // ( (j))。///測試是否取出連接 // } (links)。//添加新取得的連接 if (searchStringMatches(url,pageContents, searchString,caseSensitive)) { //(url)。 (該字段存在于： +url)。//輸出找到的地址 } } } return result。 } // 主函數(shù) public static void main(String[] args) { if(!=3){ (Usage:java SearchCrawler startUrl maxUrl searchString)。 return。 } int max=(args[1])。 myspider crawler = new myspider(args[0],max,args[2])。 Thread search=new Thread(crawler)。 (Start searching...)。 (result:)。 ()。 /**/ } } 五、系統(tǒng)測試 搜索測試以默認開始網(wǎng)頁作為起始頁面，輸入搜索字符串： 百度 ，如下圖所示： 點擊搜索，開始執(zhí)行。執(zhí)行完畢，出現(xiàn)結(jié)果： 搜索成功。 六、結(jié)論 本系統(tǒng)開發(fā)過程中用到了許多學(xué)過的知識，具體說來有數(shù)據(jù)結(jié)構(gòu)、 java 語言程序設(shè)計、軟件工程、優(yōu)化理論等等。在編程中發(fā)現(xiàn)這些學(xué)科相互聯(lián)系，相輔相成，在以后更加復(fù)雜的系統(tǒng)中肯定會涉及到更多、更復(fù)雜的學(xué)科，需要我們認真學(xué)習(xí)和掌握的東西實在是太多了。 本軟件只是對搜索引擎功能的基本實現(xiàn)，在技術(shù)方面還存在許多不足之處。當(dāng)然在這突飛猛進的信息時代，技術(shù)的更新更是日新月異，所以其中有的思想不可能完全適應(yīng)于各種實際情況。由于本人學(xué)習(xí)軟件工程的時間不長，程序的設(shè)計方面不夠規(guī)范，有些簡單的想法卻用了很長的代碼來實現(xiàn)造成了代碼冗余 ，還有部分想法沒有實現(xiàn)。我將在今后的學(xué)習(xí)中不斷完善自己。 致謝 當(dāng)這篇論文最終完成的時候，我要向給予過我熱情幫助和悉心指導(dǎo)的老師和師兄們致以真誠的謝意。 首先，我要感謝我的導(dǎo)師，感謝他帶給我來學(xué)習(xí)的機會，感謝他對我學(xué)術(shù)上的悉心指導(dǎo)，感謝他對我生活上的關(guān)懷和體貼。導(dǎo)師是不僅是我學(xué)業(yè)上的導(dǎo)師，更是生活中讓我敬佩的學(xué)者、長者。給我留下深刻印象的，是他知識的淵博、治學(xué)態(tài)度的嚴謹、誨人不倦的學(xué)者風(fēng)范，是他謙遜待人、處處關(guān)心別人的長者風(fēng)度，是他勤奮忘我的工作態(tài)度、精益求精的治學(xué)作風(fēng)。特別是老師做大事的風(fēng)范和氣度，尤其 讓我欽佩。這里我要再次感謝老師。 在本文的最后，我要再次感謝我的導(dǎo)師，同時也向與老師一樣辛勤育人，無私付出的各位導(dǎo)師、師長致以深深的謝意。 參考文獻 [1] 李曉明，悶宏飛，王繼民．搜索引擎 — — 原理、技術(shù)與系統(tǒng) [M]．北京：科學(xué)出版社， 2020． [2] Heaton J．網(wǎng)絡(luò)機器人 Java 編程指南 [M]．北京：電子工業(yè)出版社， 2020． [3] 崔澤永，常曉燕．搜索引擎的 Web Robot 的技術(shù)與優(yōu)化 [J]．微機發(fā)展， 2020， 14(4)： 100—102． [4] Shafer C．?dāng)?shù)據(jù)結(jié)構(gòu)與算法分析 (JAVA版 )[M]．北京：電子工業(yè)出版社， 2020． [5]賈年．基于移動 Agent 搜索引擎的研究與實現(xiàn) [D]．成都：電子科技大學(xué)， 2020． [6]賈年．移動 Agent 研究 [J]．四川工業(yè)學(xué)院學(xué)報， 2020， 23(3)： 51— 54． [7]S． Charkabarti． Mimng the Web’ s Link structure[J]． IEEE Computer， 2020， 32(8)： 60— 67． [8]徐寶文，張衛(wèi)豐．搜索引擎與信息獲取技術(shù)【 M】．北京：清華大學(xué)出版社， 2020 t was not u ntil midafternoon that the inspector ambled up on his pony. My father pulled himself together, and went out to receive him。 the effor t to be even formally polite nearly strangled him. Even then the inspector was not brisk. He dis mounted in a leisurely fashion, and strolled into the house, chatting about the weather. Father, red in the face, handed him over to Mary who took him along to mother39。s room. Then followed the worst wait of all. Mary said afterwards that he hummed and ha39。d for an unconscionable time while he examined the baby in minutest detail. At last, however, he emerged, with an expressionless face. In the littleused sittingroom he sat down at the table and fussed for a while about getting a good point on his quill. At last he took a form f rom his pouch, and in a slow, deliberate hand wrote that he officially found the child to be a true female human being, free from any detectable form of deviation. He regarded that thoughtfully for some moments, as though not perfectly satisfied. He let his hand hesitate before he actually dated and signed it, then he sanded it carefully, and handed it to my enraged father, still with a faint air of uncertainty. He had, of course, no real doubt in his mind, or he would have called for another opinion。 my father was perfectly well aware of that, too. At last Petra39。s existence could be admitted. I was formally told that I had a new sister, and presently I was taken to see he r where she lay in a crib beside my mother39。s bed. She looked so pink and wrinkled to me that I did not see how the inspector could have been quite sure about her. However, there was nothing obviously wrong with her, so she had got her certificate. Nobody could blame the inspector for that。 she did appear to be as normal as a newborn baby ever looks. ... While we were taking turns to look at her somebody started to ring the stable bell in the customary way. Everyone on the farm stopped work, and very soon we were all assembled in the kitchen for prayers of thanksgiving. Two, or it may have been three, days after Petra was born I happened upon a piece of my family39。s history that I would prefer not to have known. I was sitting quietly in the room next to my parents39。 bedroom where my mother still lay in bed. It was a matter of chance, and strategy, too. It was the latest place that I