【導讀】軟件工程專業(yè)畢業(yè)論文--面向webservice的網絡爬蟲。設計與實現(xiàn)+任務書+開題報告+外文翻譯。原始依據(jù)包括設計或論文的工作基礎研究條件應用環(huán)。WebServices技術是一種基于XML語言和標準的網絡協(xié)議實現(xiàn)的分布式計。算技術WebServices技術是可以改變軟件產業(yè)面貌和軟件應用模式的技術之一。WebServices技術可以將Inter上的各種信息資源和服務功能整合在一。起在任何時間任何地點向用戶提供優(yōu)質的服務WebServices技術還有助于改變。企業(yè)對IT系統(tǒng)的應用方式和依賴程度所以WebServices技術成為了網絡上的主。流技術之一隨之也來的是大量的開發(fā)人員開始開發(fā)Web服務隨著時間的推移網。課題的任務就是研究如何把網絡上分散的Web服務搜集在一起。李盛韜余智華程學旗Web信息采集研究進展J計算機科學2021. 把搜索到的Web服務相關信息存入數(shù)據(jù)庫中。為了達到以上目的本系統(tǒng)將對Heritrix開源框架進行擴展并針對服務網站。審題小組組長簽字。稱軟件工程學生姓名指導教師課題來源及意義。收集互聯(lián)網上的信息的系統(tǒng)被稱為網絡爬蟲又被稱為網頁蜘蛛網絡機

　　

【正文】 perceived in the discovery of Web pages will bee tangible and the searching for specific business applications or Web services bees challenging and time consuming particularly as the number of UDDI Business Registries UBRs begins to multiply In addition decentralizing UBRs adds another level of plexity on how to effectively find Web services within these distributed registries Decentralization of UBRs is being tangible as new operating systems applications and APIs are already equipped with builtin functionalities and tools that enable anizations or businesses to publish their own internal UBRs for intra and extra use such as the Enterprise UDDI Services in Windows Server 2021 WebShpere Application Server Systi Business Registry jUDDI to name a few Enabling businesses or anizations to selfoperate and mange their own UBRs will imize the likelihood of having a significant increase in the number of business registries and therefore clients will soon face the challenge of finding Web services across hundreds if not thousands of UBRs At the heart of the Service Oriented Architecture SOA is a service registry which connects and mediates service providers with clients as shown in Figure 1 Service registries extend the concept of an applicationcentric Web by allowing clients or conceivably applications to access a wide range of Web services that match specific search criteria in an autonomous manner Without publishing Web services through registries clients will not be able to locate services in an efficient manner and service providers will have to devote extra efforts in advertising their services through other channels There are several panies that offer Webbased Web service directories such as WebServiceList [1] RemoteMethods [2] WSIndex [3] and XMethods [4] However due to the fact that these Webbased service directories fail to adhere to Web services standards such as UDDI it is likely that they bee vulnerable to being unreliable sources forfinding relevant Web services and may bee disconnected from the Web services environment as in the cases of BindingPoint and SalCentral which closed their Webbased Web service directories after many years of exposure Apart from having Webbased service directories there have been numerous efforts that attempted to improve the discovery of Web services [56921] however many of them have failed to address the issue of handling discovery operations across multiple UBRs Due to the fact that UBRs are hosted on Web servers they are dependent on work traffic and performance and therefore clients that are looking for appropriate Web services are susceptible to performance issues when carrying out multiple UBR search requests To address the abovementioned issues this work introduces a framework that serves as the heart of our Web Services Repository Builder WSRB architecture [7] by enhancing the discovery of Web services without having any modifications to exiting standards In this paper we propose the Web Service Crawler Engine WSCE which actively crawls accessible UBRs and collects business and Web service information Our architecture enables businesses and anizations to maintain autonomous control over their UBRs while allowing clients to perform search queries adapted to largescale discovery of Web services Our solution has been tested and results present high performance rates when pared with other existing models The remainder of this paper is anized as follows Section two discusses related work Section three discusses some of the limitations with existing UBRs Section four discusses the motivations for WSCE Section five presents our Web service crawler engines architecture Experiments and results are discussed in Section six and finally conclusion and future work are discussed in Section seven 2 Related Work Discovery of Web services is a fundamental area of research in ubiquitous puting Many researchers have focused on discovering Web services through a centralized UDDI registry [8910] Although centralized registries can provide effective methods for the discovery of Web services they suffer from problems associated with having centralized systems such as single point of failure and bottlenecks In addition other issues relating to the scalability of data replication providing notifications to all subscribers when performing any system upgrades and handling versioning of services from the same provider have driven researchers to find other alternatives Other approaches focused on having multiple publicprivate registries grouped into registry federations [612] such as METEORS for enhancing the discovery process METEORS provides a discovery mechanism for publishing Web services over federated registries but this solution does not provide the means for articulating advanced search techniques which are essential for locating appropriate business applications In addition having federated registry environments can potentially provide inconsistent policies to be employed which will have a significant impact on the practicability of conducting inquiries across them Furthermore federated registry environments will have increased configuration overhead additional processing time and poor performance in terms of execution time when performing service discovery operations A desirable solution would be a Web services crawler engine such as WSCE that can facilitate the aggregation of Web service references resources and description documents and can provide clients with a standard universal access point for discovering Web services distributed across multiple registries Several approaches focused on applying traditional Information Retrieval IR techniques or using keywordbased matching [1314] which primarily depend on analyzin