【正文】 第四章 機(jī)器學(xué)習(xí)算法模型 聚類分析 聚類分析[5](Cluster analysis)是數(shù)據(jù)挖掘及機(jī)器學(xué)習(xí)領(lǐng)域內(nèi)的重點(diǎn)問題之一，在許多領(lǐng)域受到廣泛應(yīng)用，包括機(jī)器學(xué)習(xí)、數(shù)據(jù)挖掘、模式識(shí)別、決策支持、圖像分析以及生物信息，是最重要的數(shù)據(jù)分析方法之一。它與處理混合正態(tài)分布的最大期望算法很相似，因?yàn)樗麄兌荚噲D找到數(shù)據(jù)中自然聚類的中心。2． 任意產(chǎn)生k個(gè)聚類，然后確定聚類中心，或者直接生成k個(gè)中心。通常kn。 5. 對(duì)于噪聲和孤立點(diǎn)數(shù)據(jù)敏感，少量的該類數(shù)據(jù)能夠?qū)ζ骄诞a(chǎn)生極大影響。 $ grep 打印出QOS文檔中的內(nèi)容至大屏幕。 ，5個(gè)代表聚類中心的點(diǎn)。4. 只要我們假設(shè)的類簇的數(shù)目等于或者高于真實(shí)的類簇的數(shù)目時(shí)，該指標(biāo)上升會(huì)很緩慢，而一旦試圖得到少于真實(shí)數(shù)目的類簇時(shí)，該指標(biāo)會(huì)急劇上升。非拖動(dòng)緩沖次數(shù)的上升，則可能與用戶的網(wǎng)絡(luò)、視頻所在的服務(wù)器、地理位置等相關(guān)。 　 Spark是一個(gè)很強(qiáng)大的數(shù)據(jù)挖掘工具，需要進(jìn)行更深層的了解。正是導(dǎo)師的悉心指導(dǎo)和學(xué)長(zhǎng)的幫助，才使我能夠很好的完成畢業(yè)設(shè)計(jì)任務(wù)。參考文獻(xiàn)[1] [2] AMP:[3] Powered By Spark: +By+Spark[4] Spark quickstart:[5] Cluster analysis:[6]MacQueen, J. B. .Some Methods for classification and Analysis of Multivariate Observations. Proceedings of 5th Berkeley Symposium on Mathematical Statistics and Probability 1. University of California Press. pp. 281–297. Retrieved 2009.[7] Kmeans clustering:[8]Florin Dobrian,Asad Awan,Dilip Joseph,Aditya Ganjam,Jibin Zhan,Vyas Sekar,Ion Stoica,Hui the Impact of Video Quality on User 2011 . [9][10] Athula Balachandran,Vyas Sekar,Aditya Akella,Srinivasan Seshan,Ion Stoica,Hui Quest for an Internet Video QualityofExperience 2012. [11] Florin Dobrian,Asad Awan,Dilip Joseph,Aditya Ganjam,Jibin Zhan,Vyas Sekar,Ion Stoica,Hui the Impact of Video Quality on User 2011 . [12] Ahahzad Ali,Anket Mathur,Hui of Commercial PeertoPeer Live Video in Recent Advances in PeertoPeer ,2006. [13] Phillipa Gill,Martin Arlitt,Zongpeng Li,Anirban Traffic Characterization: A View From the Edge. In Proc. IMC, 2007.附錄A Quest for an Internet Video QualityofExperience MetricAthula Balachandran Vyas SekarAditya AkellaSrinivasan Seshan Ion StoicaHui ZhangCarnegie Mellon UniversityUniversity of Wisconsin MadisonStony Brook UniversityUniversity of California BerkeleyABSTRACTAn imminent challenge that content providers, CDNs, thirdparty analytics and optimization services, and video player designers in the Internet video ecosystem face is the lack of a single “gold standard” to evaluate different peting solutions. Existing techniques that describe the quality of the encoded signal or controlled studies to measure opinion scores do not translate directly into user experience at scale. Recent work shows that measurable performance metrics such as buffering, startup time, bitrate, and number of bitrate switches impact user experience. However, converting these observations into a quantitative qualityofexperience metric turns out to be challenging since these metrics are interrelated in plex and sometimes counterintuitive ways,and their relationship to user experience can be further plicate things, many confounding factors are introduced by the nature of the content itself (., user interest, genre). We believe that the issue of interdependency can be addressed by casting this as a machine learning problem to build a suitable predictive model from empirical observations. We also show that setting up the problem based on domainspecific and measurementdriven insights can minimize the impact of the various confounding factors to improve the prediction performance.Categories and Subject Descriptors [Performance of Systems]: measurement techniques,performance attributesGeneral TermsHuman Factors, Measurement, Performance1. INTRODUCTIONWith the decreasing cost of content delivery and the growing success of subscription and adbased business models(., [2]), video traffic over the Internet is predicted to increase in the years to e, possibly even surpassing television based viewership in the future [3]. An imminent challenge that all players in the Internet video ecosystem—content providers, content delivery networks, analytics services, video player designers, and users—face is the lack of a standardized approach to measure the QualityofExperience (QoE) that different solutions provide. With the “ing of age” of this technology and the establishment of industry standard groups (., [13]), such a measure will bee a fundamental requirement to promote further innovation by allowing us to objectively pare different peting designs [11,17].The notion of QoE appears to many forms of media and has a rich history in the multimedia munity (., [9, 10,14, 15]). However, Internet video introduces new effects interms of measuring both quality and experience:Measuring quality: Internet video is delivered using HTTPbased modity technology over a largely unreliable network via existing CDN infrastructures. Consequently, the traditional encodingrelated measures of quality like Peak SignaltoNoise Ratio are replaced by a suite of quality metrics that capture several effects introduced by the delivery mechanism—buffering, bitrate delivered, frame rendering rate, bitrate switching, and startup delay [6, 33].Measuring experience: In the context of advertismentand subscriptionsupported services, the perceptual opinion of a user in a controlled study does not necessarily translate into objective measures of engagement that impact providers’ business objectives. Typical measures of engagement used today to approximate these business objectives are inthewild measurements of user behavior。產(chǎn)品經(jīng)理可以研究用戶的體驗(yàn)，企業(yè)可以據(jù)此做出調(diào)整得到利益效益的最大化。[8] 　從本次實(shí)驗(yàn)的結(jié)果來看緩沖次數(shù)基本不會(huì)隨著用戶的觀看時(shí)間、拖動(dòng)次數(shù)，非拖動(dòng)緩沖次數(shù)而改變。 K取值從2到9時(shí)的類簇指標(biāo)的變化曲線，選擇類簇指標(biāo)是K個(gè)類簇的平均質(zhì)心距離的加權(quán)平均值。 所以K個(gè)初始類簇點(diǎn)的選取還有兩種方法：1)選擇彼此距離盡可能遠(yuǎn)的K個(gè)點(diǎn) 2)先對(duì)數(shù)據(jù)用層次聚類算法或者Canopy算法進(jìn)行聚類，得到K個(gè)簇之后，從每個(gè)類簇中選擇一個(gè)點(diǎn)，該點(diǎn)可以是該類簇的中心點(diǎn)，或者是距離類簇中心點(diǎn)最近的那個(gè)點(diǎn)。 首先通過下面的命令安裝pip，pip是Python的一個(gè)安裝和管理擴(kuò)展庫(kù)的工具。此待待測(cè)量數(shù)據(jù)文檔命名為qos，含有10140條，由用戶uid、ip地址，觀看視頻時(shí)間、緩沖次數(shù)、拖動(dòng)次數(shù)，非拖動(dòng)緩沖次數(shù)這六個(gè)字段組成。 3． 算法嘗試找出使平方誤差函數(shù)值最小的k個(gè)劃分。即對(duì)每個(gè)點(diǎn)確定其聚類中心點(diǎn) 再計(jì)算其聚類新中心。對(duì)剩余的每個(gè)對(duì)象根據(jù)其與各個(gè)簇中心的距離，將它賦給最近的簇，然后重新計(jì)算每個(gè)簇的平均值。由聚類所生成的簇是一組數(shù)據(jù)對(duì)象的集合，這些對(duì)象與同一個(gè)簇中的對(duì)象彼此相似，與其他簇中的對(duì)象彼此相異。 解壓編譯 　 : 　 $ tar zxvf 　 運(yùn)行sbt進(jìn)行編譯： 　 $ cd ～/ 　 $ sbt assembly 這個(gè)步驟會(huì)下載很多庫(kù)，然后進(jìn)行編譯，編譯時(shí)間大概會(huì)在1個(gè)小時(shí)左右?！　? jps 　檢查各進(jìn)程是否運(yùn)行，這時(shí)，應(yīng)該看到有6個(gè)java虛擬機(jī)的進(jìn)程，