Catálogo de publicaciones - libros

The automatic relevance determination (ARD) shows good performance in many applications. Recently, it has been applied to brain current estimation with the variational method. Although people who use the ARD tend to pay attention to one benefit of the ARD, sparsity, we, in this paper, focus on another benefit, generalization. In this paper, we clarify the generalization error of the ARD in the case that a class of prior distributions is used, and show that good generalization is caused by singularities of the ARD. Sparsity is not observed in that case, however, the mechanism that the singularities provide good generalization implies the mechanism that they also provide sparsity.

A lot of learning machines which have the hidden variables or the hierarchical structures are the singular statistical models. They have a different learning performance from the regular statistical models. In this paper, we show that the learning coefficient is easily computed by weighted blow up, in contrast, and that there is the case that the learning coefficient cannot be correctly computed by blowing up at the origin only.

Echo state neural networks, which are a special case of recurrent neural networks, are studied from the viewpoint of their learning ability, with a goal to achieve their greater prediction ability. A standard training of these neural networks uses pseudoinverse matrix for one-step learning of weights from hidden to output neurons. This regular adaptation of Echo State neural networks was optimized by updating the weights of the dynamic reservoir with Anti-Oja’s learning. Echo State neural networks use dynamics of this massive and randomly initialized dynamic reservoir to extract interesting properties of incoming sequences. This approach was tested in laser fluctuations and Mackey-Glass time series prediction. The prediction error achieved by this approach was substantially smaller in comparison with prediction error achieved by a standard algorithm.

Belief propagation (BP) is the calculation method which enables us to obtain the marginal probabilities with a tractable computational cost. BP is known to provide true marginal probabilities when the graph describing the target distribution has a tree structure, while do approximate marginal probabilities when the graph has loops. The accuracy of loopy belief propagation (LBP) has been studied. In this paper, we focus on applying LBP to a multi-dimensional Gaussian distribution and analytically show how accurate LBP is for some cases.

The reduction of input dimensionality is an important subject in modelling, knowledge discovery and data mining. Indeed, an appropriate combination of inputs is desirable in order to obtain better generalisation capabilities with the models. There are several approaches to perform input selection. In this work we will deal with techniques guided by measures of input relevance or input sensitivity. Six strategies to assess input relevance were tested over four benchmark datasets using a backward selection wrapper. The results show that a group of techniques produces input combinations with better generalisation capabilities even if the implemented wrapper does not compute any measure of generalisation performance.

Although encouraging results have been reported, existing Bayesian network (BN) learning algorithms have some troubles on limited data. A statistical or information theoretical measure or a score function may be unreliable on limited datasets, which affects learning accuracy. To alleviate the above problem, we propose a novel BN learning algorithm MRMRG, Max Relevance and Min Redundancy Greedy algorithm. MRMRG algorithm applies Max Relevance and Min Redundancy feature selection technique and proposes Local Bayesian Increment (LBI) function according to the Bayesian Information Criterion (BIC) formula and the likelihood property of overfitting. Experimental results show that MRMRG algorithm has much better accuracy than most of existing BN learning algorithms when learning BNs from limited datasets.

An incremental one-class learning algorithm is proposed for the purpose of outlier detection. Outliers are identified by estimating - and thresholding - the probability distribution of the training data. In the early stages of training a non-parametric estimate of the training data distribution is obtained using kernel density estimation. Once the number of training examples reaches the maximum computationally feasible limit for kernel density estimation, we treat the kernel density estimate as a maximally-complex Gaussian mixture model, and keep the model complexity constant by merging a pair of components for each new kernel added. This method is shown to outperform a current state-of-the-art incremental one-class learning algorithm (Incremental SVDD [5]) on a variety of datasets, while requiring only an upper limit on model complexity to be specified.

Estimating the size of neural networks for achieving high classification accuracy is a hard problem. Existing studies provide theoretical upper bounds on the size of neural networks that are unrealistic to implement. This work provides a computational study for estimating the size of neural networks using as an estimation parameter the size of available training data. We will also show that the size of a neural network is problem dependent and that one only needs the number of available training data to determine the size of the required network for achieving high classification rate. We use for our experiments a threshold neural network that combines the perceptron algorithm with simulated annealing and we tested our results on datasets from the UCI Machine Learning Repository. Based on our experimental results, we propose a formula to estimate the number of perceptrons that have to be trained in order to achieve a high classification accuracy.

This paper is to identify the clustering structure and the relevant features automatically and simultaneously in the context of Gaussian mixture model. We perform this task by introducing two sets of weight functions under the recently proposed (MWL) learning framework. One set is to reward the significance of each component in the mixture, and the other one is to discriminate the relevance of each feature to the cluster structure. The experiments on both the synthetic and real-world data show the efficacy of the proposed algorithm.

Learning machines such as neural networks, Gaussian mixtures, Bayes networks, hidden Markov models, and Boltzmann machines are called singular learning machines, which have been applied to many real problems such as pattern recognition, time-series prediction, and system control. However, these learning machines have singular points which are attributable to their hierarchical structures or symmetry properties. Hence, the maximum likelihood estimators do not have asymptotic normality, and conventional asymptotic theory for statistical regular models can not be applied. Therefore, theoretical optimal model selections or designs involve algebraic geometrical analysis. The algebraic geometrical analysis requires blowing up, which is to obtain maximum poles of zeta functions in learning theory, however, it is hard for complex learning machines. In this paper, a new method which obtains the maximum poles of zeta functions in learning theory by numerical computations is proposed, and its effectiveness is shown by experimental results.