【正文】 e system modules, including the acquisition task configuration, management and acquisition. The user fills the task of collecting configuration according to their needs, and the spider automatically and efficiently extracts information and downloads page in accordance with the access rules and extraction rules. Keywords: Information collection。 必備條件： 系統(tǒng)配置： Windows 2020/XP 開(kāi)發(fā)環(huán)境：不限 開(kāi)發(fā)語(yǔ)言： Java/PHP/Ruby/Python 不限 基本要求： 三大子系統(tǒng)：任務(wù)管理、網(wǎng)頁(yè)采集、信息抽取 1）能夠?qū)Σ杉蝿?wù)進(jìn)行配置、管理 2）能夠根據(jù)規(guī)則批量采集網(wǎng)頁(yè) 3）能夠根據(jù)規(guī)則批量提取網(wǎng)頁(yè)信息，并轉(zhuǎn)化成結(jié)構(gòu)化數(shù)據(jù) 較高要求： 1）能夠?qū)κ褂?Cookies 的網(wǎng)站進(jìn)行采集 2）能夠?qū)κ褂?Post 方式發(fā)送請(qǐng)求的網(wǎng)站進(jìn)行采集 3） 使用“正則表達(dá)式”或“ XPath”描述采集、提取規(guī)則 關(guān)鍵技術(shù)或創(chuàng)新點(diǎn)： 網(wǎng)頁(yè)采集技術(shù)、網(wǎng)頁(yè)信息抽取技術(shù) 指導(dǎo)教師簽字 年 月 日 負(fù)責(zé)老師簽字 年 月 日 IV 目 錄 1 引言 ................................................................................................................................... 1 項(xiàng)目背景 ..................................................................................................................... 1 論文的目的及設(shè)計(jì)要求 ............................................................................................ 1 論文目的及設(shè)計(jì)要求 .......................................................................................... 1 主要貢獻(xiàn) .............................................................................................................. 2 2 研究現(xiàn)狀及設(shè)計(jì)目標(biāo) ....................................................................................................... 3 相近研究課題研究現(xiàn)狀及優(yōu)缺點(diǎn) ............................................................................. 3 Wget 功能及特點(diǎn) .................................................................................................. 3 網(wǎng)絡(luò) 信息采集專家功能及特點(diǎn) .......................................................................... 3 設(shè)計(jì)目標(biāo) ..................................................................................................................... 3 經(jīng)濟(jì)效益分析 ............................................................................................................. 4 3 關(guān)鍵問(wèn)題及分析 ............................................................................................................... 5 關(guān)鍵問(wèn)題 ..................................................................................................................... 5 怎樣實(shí)現(xiàn)網(wǎng)絡(luò)爬蟲(chóng)網(wǎng)頁(yè)采集 .............................................................................. 5 提取有效鏈接 ...................................................................................................... 5 關(guān)鍵技術(shù)及復(fù)雜性分析 ............................................................................................. 6 網(wǎng)絡(luò)爬蟲(chóng) .............................................................................................................. 6 Ruby on Rails ........................................................................................................ 7 4 需求分析 ........................................................................................................................... 8 系統(tǒng)用例圖 ................................................................................................................. 8 系統(tǒng)主要業(yè)務(wù)流程分析 ............................................................................................. 8 功能需求分析 ............................................................................................................. 9 功能介紹 .............................................................................................................. 9 登錄流程圖 ........................................................................................................ 10 任務(wù)采集流程圖 ................................................................................................ 12 5 系統(tǒng)設(shè)計(jì) ......................................................................................................................... 13 設(shè)計(jì)實(shí)現(xiàn)的策略 ....................................................................................................... 13 V 編程模型 ................................................................................................................... 14 數(shù)據(jù)模型分析 ........................................................................................................... 15 數(shù)據(jù)庫(kù)設(shè)計(jì) ............................................................................................................... 16 數(shù)據(jù)庫(kù)分析 ........................................................................................................ 16 數(shù)據(jù)庫(kù)表結(jié)構(gòu)設(shè)計(jì) ............................................................................................ 16 6 系統(tǒng)實(shí)現(xiàn)及測(cè)試 ............................................................................................................. 19 各模塊具體實(shí)現(xiàn)方法 .............................................................................................. 19 建立工程 ............................................................................................................ 19 創(chuàng)建數(shù)據(jù)庫(kù)表 .................................................................................................... 21 生成手腳架 ........................................................................................................ 23 登錄模塊 ............................................................................................................ 24 用戶管理模塊 .................................................................................................... 25 任務(wù)組管理模塊 ................................................................................................ 27 任務(wù)管理模塊 ....................