【導(dǎo)讀】在這一章節(jié)我們討論一些方法,為了解決LS-SVM的大數(shù)據(jù)設(shè)定中的方法設(shè)計(jì)和歸類(lèi)問(wèn)題.

　　

【正文】 general differentiable optimization problems. Individual optimizers correspond then to individual LSSVMs which are interacting through state synchronization constraints, can optimize in a cooperative way a group cost function for the ensemble and in this way realize collective intelligence. Multilayer works of LSSVMs In a mittee work one linearly bines the individual LSSVM models. A natural extension could be to bine the works in a nonlinear fashion. This might be done in several ways. One potion is to take an MLP which has the outputs of the individual LSSVMs as input to the work. The LSSVMs are first trained on subsets of data D1, D2, … , Dm and then serve in fact as a first nonlinear preprocessing layer after which an output layer is trained that is represented by an MLP work. Instead of an MLP one may take any other static nonlinear model, hence also LSSVMs could in principle be taken as a second layer. However, towards large data sets is will be more advantageous to use a parametric modeling approach for the second layer. A parametric approach to the training of kernel based models was also taken . by Tipping [253] in the relevance vector machine. A multilayer work of LSSVMs where the bination of the models is determined by solving a parametric optimization problem (. by Bayesian learning), can be considered as a nonlinear extension of this. The relevance of the LSSVM submodels is determined by the estimation of the second layer. When taking an MLP in the second layer, the model is described by () with () where M denotes the number of individual LSSVM models whose outputs Zi are the input to a MLP with output weight vector WERnh, hidden layer matrix VERn*m and bias vector dErnh where Nh denotes the number of hidden units. On can take multiple outputs as well. The coefficients aki, bki for i = 1, … , m are the solutions to a number of m linear systems for each of the individual LSSVMs trained on data sets Di(). In and the approach is illustrated on the estimation of a noisy sinc function, both tor m= 4 and m = 40 bined LSSVMs trained with RBF kernel Q = 5 and regularization constant R = 1. the outputs of these LSSVMs are taken as input to an MLP with one hidden layer and 6 hidden units. The MLP is trained by Bayesian learning. The results are pared with taking () which corresponds also to one single neuron with linear activation function. Both are trained by the function trained in Matlab’s neural works toolbox. In the zero mean random Gaussian noise with standard deviation was imposed on the true sinc function for generating the training data ,while in a standard deviation of 3 was taken. Nonlinear binations of trained LSSVM models. First LSSVM models are trained on subsets of data D1,… , Dm. Then a nonlinear bination of the LSSVM work outputs is taken. This leads to a multilayer work where the nonlinear function G(.) can for example be represented by a multilayer perceptron or by a second LSSVM layer which is trained then with the outes of the LSSVM models as input data. Alternatively, one may also view the LSSVM models as a linear or nonlinear preprocessing of the input data. Linear and nonlinear binations of LSSVMs with (Left) m = 4 LSSVMs and (Right) m = 40 LSSVMs for a noisy sinc function estimation problem (standard dev. Of Gaussian noise is ): (Top) 4000 given training data points, the intervals with vertical lines indicate the training data used for the individual LSSVMs。 (Middle) linear bination of LSSVMs with single neuron trained by Bayesian learning。 (Bottom) nonlinear bination of LSSVMs with MLP trained by Bauesian learning. Similar results as for the previous Figure but with standard dev. of Gaussian noise equal to 3. 第六章 大規(guī)模問(wèn)題 在這一章節(jié) , 為了解決大規(guī)模數(shù)據(jù)設(shè)定中的功能設(shè)計(jì)和歸類(lèi)問(wèn)題 ,我們討論一些方法。基于高斯過(guò)程和不完整的 Cholesky 分解，我們解釋一下 Nystrom 方法對(duì)于低跌逼近的建議。然后提出在固定規(guī)模 LSSVM 方面的一個(gè)新技術(shù) ——最小二乘支持向量機(jī) 。在這個(gè) 固定 規(guī)模 LSSVM 方法的 最小二乘支持向量機(jī) 中，單一取代了對(duì)偶，解決了存在的主要問(wèn)題。在估計(jì)的映射到特征空間 B 后，形式上獲得了由核主成分分析得到的特征函數(shù)。這部分內(nèi)容將在下一章節(jié)進(jìn)行更詳細(xì)的介紹。這種方法提供了函數(shù)估計(jì)和密度估計(jì)之間的明確聯(lián) 系，利用原始的雙配方，解決如何用主動(dòng)選擇適當(dāng)?shù)闹蜗蛄縼?lái)取代原有的隨機(jī)選擇的問(wèn)題。下一步，我們旨在解釋在特征空間中構(gòu)造一個(gè)合適的基本成分。此外，方法結(jié)合子是討論諸如委員會(huì)網(wǎng)絡(luò)、非線性以及多層擴(kuò)展的方法。 低階近似法 Nystrom 方法 圖 線性支持向量機(jī)對(duì)偶問(wèn)題既是如何解決大型數(shù)據(jù)集的問(wèn)題，但是大維輸入空間問(wèn)題是主要的問(wèn)題。然而，對(duì)于非線性 SVMs 中沒(méi)有 的表達(dá)式的研究，其結(jié)果只能解決相關(guān)的內(nèi)核函數(shù)的雙重問(wèn)題。在固定規(guī)模尺寸方法 LSSVM 中 Nystrom 方法用于估計(jì) 本征函數(shù)。獲得估計(jì)為 和連接原始雙配方后，計(jì)算的 w， b 是在原始空間。 假設(shè)存在一個(gè)線性?xún)?nèi)核。前幾章我們已經(jīng)提到過(guò)的對(duì)偶問(wèn)題，其實(shí)可以同樣很好地解決。事實(shí)上存在的主要問(wèn)題是對(duì)大型數(shù)據(jù)設(shè)置更有利詭計(jì)的雙重問(wèn)題，以及找到更適合的大維輸入空間的問(wèn)題。因?yàn)槲粗獢?shù)分別為 和 ，其中 n 表示輸入空間的維以及 N 表示給定的訓(xùn)練數(shù)據(jù)點(diǎn)。例如在線性函數(shù)估計(jì)的情況下，有消除錯(cuò)誤變量 ，其中一個(gè)可以立即解決。在這種情況下映射 變?yōu)?，沒(méi)有需要解決的支持值 a 的雙重問(wèn)題，所以這組數(shù)據(jù)肯定不是大型數(shù)據(jù)集。 （ ） 另一方面是在非線性的情況下，此情況要復(fù)雜得多。對(duì)于多選擇的內(nèi)核，可能會(huì)變成無(wú)窮維數(shù)，因此也將 W 作為載體。然而，在這種情況下，仍然可以嘗試找到有意義的估計(jì)值 。 一個(gè)要找到這樣估算的過(guò)程是運(yùn)用了 Nystrom方法，這就是眾所周知的面積積分方程 [14； 63]，并且此方程已成功地應(yīng)用在高斯過(guò)程的 Williams 和 Seeger下，在 [294]頁(yè)。此方法是用低秩逼近給定的核矩陣尋找隨機(jī)選擇 M 行 /列的核矩陣的方法。讓我們用表示內(nèi)核大矩陣的 ，和表示小核矩陣基于隨機(jī)子樣本的 ，其中 M N（在實(shí)踐中常常要求 ），討論小核矩陣 （ ） 其中 包含對(duì)特征值和 對(duì)應(yīng)的特征向量的特征值的分解。這是關(guān)系到特征函數(shù) 和特征值 的積分方程 （ ） 如下： （ ） 和 分別是 和 的積分方程 ， 并且 是表示 項(xiàng)的矩陣 。這是可以理解的形式采樣 M 點(diǎn) 的積分。對(duì)于大內(nèi)核矩陣的特征值分解為 （ ） 此外，正如在 [294]頁(yè)： （ ）然后可以顯示 () 其中 是取自 的分塊矩陣 。 這些不帶偏見(jiàn)的見(jiàn)解是通過(guò)高斯過(guò)程回歸問(wèn)題審議的，然后使用近似意義上的求解線性系統(tǒng)得到長(zhǎng)期模型： （ ） 然后運(yùn)用謝爾曼 莫里森 Woodbury 公式 [98]獲得 [294]： （ ） 其中 是經(jīng)過(guò)公式（ ）計(jì)算后得到的， 來(lái)源于小核矩陣。人們通常認(rèn)為 LSSVM 的分類(lèi)和回歸導(dǎo)致圍繞核矩陣偏置項(xiàng)。以 Nystrom 方法的應(yīng)用為中心的核矩陣的特征值分解方法是被采用的。最后，內(nèi)核矩陣逼近誤差的進(jìn)一步證實(shí)了調(diào)查結(jié)果 [206； 226]頁(yè)。 Nystrom方法已被應(yīng)用于 LSSVM 的貝葉斯推理的第二個(gè)層次的框架，而解決的共軛梯度法 [273]頁(yè)的第一個(gè)解決方法卻無(wú)法解決 Nystrom方法的逼近問(wèn)題。數(shù)據(jù)如表 ，這說(shuō)明了三個(gè)數(shù)據(jù)集 CRA（ leptograpsus 蟹）， RSY（里普利合成數(shù)據(jù)）， HEA（心臟疾?。┦歉鶕?jù) [273]頁(yè)的數(shù)據(jù)得到的。在 [273]中也說(shuō)明，更大的數(shù)據(jù)設(shè)置，如在 UCI 成人組數(shù)據(jù)中，可以由一個(gè)子樣本 100 個(gè)數(shù)據(jù)點(diǎn)，在第二個(gè)層次的推理，以確定 由 Nystrom方法的成功逼近而不是整個(gè)訓(xùn)練數(shù)據(jù)集規(guī)模 33000。 表（ ）