Principal Tensor Embedding for Unsupervised Tensor Learning

Tensors and multiway analysis aim to explore the relationships between the variables used to represent the data and find a summarization of the data with models of reduced dimensionality. However, although in this context a great attention was devoted to this problem, dimension reduction of high-order tensors remains a challenge. The aim of this article is to provide a nonlinear dimensionality reduction approach, named principal tensor embedding (PTE), for unsupervised tensor learning, that is able to derive an explicit nonlinear model of data. As in the standard manifold learning (ML) technique, it assumes multidimensional data lie close to a low-dimensional manifold embedded in a high-dimensional space. On the basis of this assumption a local parametrization of data that accurately captures its local geometry is derived. From this mathematical framework a nonlinear stochastic model of data that depends on a reduced set of latent variables is obtained. In this way the initial problem of unsupervised learning is reduced to the regression of a nonlinear input-output function, i.e. a supervised learning problem. Extensive experiments on several tensor datasets demonstrate that the proposed ML approach gives competitive performance when compared with other techniques used for data reconstruction and classification.

SECTION I. Introduction

Tensors, also referred to as multiway arrays, are high-order generalizations of vectors and matrices and have been adopted in diverse branches of data analysis, to represent a wide range of real-world data. Examples of tensor data are grayscale and color video sequences [1]–​[4], gene expression [5], genome-scale signals [6], magnetic resonance imaging [7], to cite just a few.

Data modeling and classification of these data are important problems in several applications, such as human action and gesture recognition [8], tumor classifications [9], spatio-temporal analysis in climatology, geology and sociology [10], neuroimaging data analysis [11], big data representation [12], completion of big data [13], and so on. To address these problems most previous works represent a tensor by a vector in high-dimensional space and apply ordinary learning methods for vectorial data. Representative techniques in this context include feature extraction and selection [14], [15], linear discriminant analysis (LDA) [16], and support vector machine (SVM) [17]. Unfortunately this approach needs to arrange the tensor data into long vectors causing two main problems, i) loss of structural information of tensors, ii) vectors with very high dimensionality. To face these problems specific tensor learning algorithms that retain the original structure of tensor data have been recently developed [18]–​[26]. However, in all these methods the problem of high dimensionality of data, also known as the curse of dimensionality, remains. To deal with such an issue the classical dimensionality reduction method, known as principal component analysis (PCA) [27], [28] was generalized to the second-order case (2-DPCA) [29], low-rank matrices (GLRAM) [30], and high-order cases (MPCA) [31], [32]. Likewise, the linear discriminant analysis (LDA) [33] technique was extended to a 2D case (2-DLDA) [34], [35] and multilinear discriminant analysis (MDA) [36]. Nevertheless to overcome the limitations of methods based on classical approaches, the decomposition of tensors into low-rank components, using two popular models, namely the Tucker decomposition (TD) [37] and the CANDECOMP/PARAFAC (CP) decomposition [38], has been one of the main concerns in tensor analysis to reduce the dimensionality of data [39], [40]. In all the above methods, data are considered as points in a multidimensional space, thus using the global structure information of the dataset alone. Instead many studies have shown that some classes of real world high-dimensional data exist in which they lie on a low-dimensional manifold (a parametrized surface), thus showing a local geometric structure.

Manifold learning (ML) is a nonlinear dimensionality reduction (NLDR) technique that, assuming the existence of an intrinsic structure, the manifold, has proven to be very effective in modeling data with reduced dimensionality [41], [42]. ML, also classified as embedding method, is based on the assumption that high-dimensional data are embedded in a nonlinear manifold of lower dimension [43]–​[47], [48]–​[50]. In this context several algorithms have been proposed, such as locally linear embedding (LLE) [51], local tangent space alignment (LTSA) [52], locally multidimensional scaling (LMDS) [53], and ISOMAP [54].

To deal with the high-order tensor data, some of these methods were extended by using multiway data analysis [55] and in particular higher order tensor decomposition [56], [57]. For example, Lai et al. [58] proposed a robust tensor learning method called sparse tensor alignment (STA) for unsupervised tensor feature extraction. Ju et al. [59] introduced a new tensor dimension reduction model based on the Bayesian theory. The proposed method assumes that each observation can be represented as a linear combination of some tensor bases, thus CP decomposition and variational EM algorithm are used to solve this model. He et al. [60] proposed tensor subspace analysis (TSA) for second-order learning. In the method suggested by Jiang et al. [61], given image tensor data, a $k$ -nearest neighbour graph to encode the geometrical structure of data is constructed. Liu et al. [62] proposed a non-linear dimensionality reduction algorithm based on locally linear embedding called supervised locally linear embedding in tensor space (SLLE/T). SLLE/T preserves local manifold structure within each class based on locally linear embedding (LLE) and enforces separability of data points belonging to different classes. Chen et al. [63], assuming that data lie in a nonlinear manifold, attempted to discover the intrinsic structure of this manifold with a two-stage algorithm named tensor-based Riemannian manifold distance-approximating projection (TRIMAP). Jia and Fu [64] suggested a low-rank tensor subspace learning for RGB-D action recognition, in which the tensor samples are factorized to obtain three projection matrices by Tucker Decomposition (TD).

The central objective in ML algorithms is to determine an effective parametrization of data. This is a key issue in order to accurately capture the local geometry of the low-dimensional manifold and the following aspects are relevant to this end: i) nonlinearity, ii) explicit modeling, iii) intrinsic dimension (ID) estimation.

With regard to nonlinearity data generally have a nonlinear geometric structure, thus using the tangent space at each data point to locally describe its neighbour, as assumed in some of the previous techniques, is a strong limitation.

With reference to the second aspect, a main drawback of most ML methods is that no explicit mapping representing the local manifold parametrization can be obtained after the training process, as they learn high-dimensional data implicitly.

Regarding ID estimation, the intrinsic dimension (ID) may be interpreted as the minimum number of parameters required to describe the data [65], thus to derive a low-dimensional model, the dataset ID has to be discovered first.

At present, none of the methods suggested so far are able to take into account all the key requirements of nonlinearity, explicit modeling, and ID estimation for low-dimensional tensor modeling.

The aim of this article is to develop a manifold learning-based approach, named principal tensor embedding (PTE), for unsupervised tensor learning, that is able to address the first two of the aforementioned key points, while adopting the most relevant state-of-the-art methods for the ID estimation. This result represents an advancement with respect to the state-of-the-art, as the standard tensor learning techniques are not able to combine all the relevant aspects previously mentioned. The method has been derived by considering a tensor as an element of the finite-dimensional linear space of tensors, in which the inner product defines a metric for the space. Once a basis is computed using the Gram-Schmidt procedure, the coefficient vector in this basis, establishes an isomorphism between a vector space of rank-one and the space of tensors. In this way the problem of dimensionality reduction in tensor space reduces to the dimensionality reduction in vector space. To this end an effective manifold algorithm recently proposed, can be used for the parametrization of data, to accurately capture the local geometry of the low-dimensional manifold that represents the data. In such a way a nonlinear model of data with reduced dimensionality is obtained.

The model establishes an explicit one-to-one correspondence between a tensor, a point on the manifold embedded in the high dimensional space, and a vector, a point in the low-dimensional Euclidean space. Additionally a relationship for the geodesic distance of all pairs of points on the manifold, as a nonlinear function of the Euclidean distance between points in the low-dimensional space, is given.

The rest of the paper is organized as follows. Section II reviews related work on tensor learning. Section III summarizes our method highlighting the most relevant aspects. Section IV introduces some general concepts on finite dimensional linear space of tensors. Section V gives a representation of a tensor in terms of a basis derived by the tensor version of the Gram-Schmidt procedure. In Section VI a nonlinear dimensionality reduction approach, named principal tensor embedding (PTE), is developed, and the estimation of nonlinearity in such a model is treated in Section VII using a nonparametric kernel regression (NPKR) technique. Experimental results are presented in Section VIII, in which the proposed tensor learning approach is used and compared with some other techniques for data reconstruction and classification problems.

SECTION II. Related Work

Tensor learning techniques have been more widely applied to 2-order and 3-order tensors. In the following we summarize the most relevant approaches in these two main fields.

A. Tensor Learning of 2-Order Tensors

Principal component analysis (PCA) is one of the most common techniques for unsupervised subspace learning, however when applied to tensor objects it requires their reshaping into vectors with high-dimensionality (vectorization). As a result this implies high processing costs in terms of computational and memory demand.

Multilinear principal component analysis (MPCA) [31] is the multilinear extension of the classical PCA that have been used both for two-order and three-order tensors. Recently a method called graph-Laplacian PCA (gLPCA) that combines a manifold learning method, i.e. the graph-Laplacian, with PCA has been proposed [66].

Tucker decomposition (TD) is a technique that reduces a given tensor to a low-rank tensor [37]. It solves an optimization problem, by minimizing the Frobenius distance between the given tensor and a tensor of lower dimensionality.

To account for the geometrical (manifold) structure of image tensor data, a technique called graph-Laplacian Tucker tensor decomposition (GLTD), that combines graph-Laplacian with a regularized version of TD, has recently been proposed in [61].

B. Tensor Learning of 3-Order Tensors

Tensor neighborhood preserving embedded (TNPE) and tensor locality preserving projection (TLPP) [67] are tensor embedding techniques. They extend neighborhood preserving embedding (NPE) and locality preserving projection (LPP), which can only work with vectorized representation, to be used with more general tensor representations. More specifically, given a set of data points $\{ x_{i}, \, i\,\,= 1, \ldots, N \}$ in higher-dimensional space, NPE and LPE seek a transformation matrix that maps each data point $x_{i}$ to a corresponding lower-dimensional data point $y_{i}$ . Similarly TNPE and TLPP find a set of transformation matrices for defining the embedded tensor subspaces that together give an optimal approximation to the tensor manifold, preserving some local geometric properties.

Orthogonal tensor neighborhood preserving embedded (OTNPE) [68] is a generalized tensor subspace model similar to TNPE. However, while TNPE cannot ensure the obtained transformation matrices have orthogonal column vectors, OTNPE aims to derive orthonormal basis tensor for TNPE.

Sparse tensor alignment (STA) [58] is a sparse representation incorporated into tensor alignment (TA) framework, a technique that unifies the tensor learning methods. Since a tensor $\mathcal {X}_{i}$ can be unfolded into a large size matrix $X_{i}$ , to $k$ nearest neighbours tensors correspond $k$ large size matrices $X_{i}^{(k)}$ . The alignment techniques aims to obtain the projection matrices $U_{k}$ that map the unfolding tensors $X_{i}^{(k)}$ into a low-dimensional unfolding tensors $Y_{i}^{(k)}, \, U_{k}: \, X_{i}^{(k)} \rightarrow Y_{i}^{(k)}$ .

Unfortunately none of the aforementioned techniques is able to satisfy the key requirements in order to accurately capture the local geometry of tensors embedded in a manifold, i.e. nonlinearity, explicit modeling and ID estimation.

SECTION III. Our Method

In this article we address the problem of unsupervised learning of tensors. In this case one has a set of $N$ observations, the data $\Omega = \{ \mathcal {X}^{(1) }, \ldots, \mathcal {X}^{(N)} \}$ , of a random $M$ -order tensor $\mathcal {X}$ . The goal is to derive a model that depends on a reduced set of parameters, the latent variables, that is able to reconstruct the data. To be effective the dimension $d$ of parameters in the model must be less than the dimension $L$ of the tensor linear space.

Mathematically this problem is equivalent to determine a $d$ -dimensional manifold $\mathcal {M}$ embedded in $\mathbb {R}^{I_{1} \times I_{2} \times \ldots \times I_{M}} \; \left({d \ll L = \prod _{j=1}^{M} I_{j}}\right)$ characterized by a nonlinear map

$$\begin{equation*} \mathcal {X} = \gamma (\beta ''), \; \beta '' \in U \subset \mathbb {R}^{d}, \; \mathcal {X} = \mathbb {R}^{I_{1} \times I_{2} \times \ldots \times I_{M}}\tag{1}\end{equation*}$$ from low-dimensional space $U \subset \mathbb {R}^{d}$ to high-dimensional space $\mathbb {R}^{I_{1} \times I_{2} \times \ldots \times I_{M}}$ .

The main steps of our approach are:

• Given the data set $\Omega = \{ \mathcal {X}^{(1) }, \mathcal {X}^{(2) }, \ldots, \mathcal {X}^{(N)} \}$ the Gram-Schmidt procedure is applied to $L$ observations $\{ \hat {\mathcal {X}}^{(1) }, \hat {\mathcal {X}}^{(2) }, \ldots, \hat {\mathcal {X}}^{(L)} \}$ so that an orthonormal basis $\{ \mathcal {U}^{(1) }, \mathcal {U}^{(2) }, \ldots, \mathcal {U}^{(L)} \}$ is obtained.

• The generic tensor $\mathcal {X}$ of the set $\Omega$ can be represented as the summation

  $$\begin{equation*} \mathcal {X} = \sum _{i=1}^{L} \alpha _{i} \mathcal {U}^{(i)}, \; L = \prod _{j=1}^{M} I_{j}\tag{2}\end{equation*}$$ where $\alpha = (\alpha _{1}, \alpha _{2}, \ldots, \alpha _{L})^{T}$ is the coefficient vector. Thus to the dataset $\Omega = \{ \mathcal {X}^{(1) }, \mathcal {X}^{(2) }, \ldots, \mathcal {X}^{(N)} \}$ corresponds a set of $N$ vectors $\{ \alpha ^{(1) }, \alpha ^{(2) }, \ldots, \alpha ^{(N)} \}$ in $\mathbb {R}^{L}$ .

• We assume these vectors are points sampled from a manifold $\mathcal {M}$ of dimension $d$ embedded in the $L$ -dimensional observation space, so that a local parametrization

  $$\begin{equation*} \alpha = F(\theta), \; \theta \in \mathbb {R}^{d}, \; d \in \mathbb {R}^{L},\tag{3}\end{equation*}$$ exists, with $d < L$ , where $\theta$ is the vector of latent variables and $d$ is the so-called intrinsic dimension.

• Since $\theta$ is hidden, i.e. not known, it will be shown that a local parametrization $\gamma$ of the manifold, through a partition $\alpha = \varphi (\alpha '') = \left ({\alpha '', G(\alpha '') }\right)^{T}$ of the vector $\alpha$ can be derived.

  This is an explicit modeling of the manifold depending on known variables $\alpha '' \in \mathbb {R}^{d}$ with $d$ = ID. The model is completely defined once $\alpha$ and $G(\cdot)$ have been estimated.

• By estimating the variance vector $\sigma _{\alpha } = (\sigma _{\alpha _{1}}, \ldots, \sigma _{\alpha _{L}})^{T}$ of $\alpha$ , their elements are put in decreasing order and the corresponding terms $\{ \hat {\mathcal {U}}^{(1) }, \hat {\mathcal {U}}^{(2) }, \ldots, \hat {\mathcal {U}}^{(L)} \}$ are ordered accordingly. In this way a new representation of the tensor $\mathcal {X}$ is obtained

  $$\begin{equation*} \mathcal {X} = \sum _{i=1}^{L} \beta _{i} \hat {\mathcal {U}}^{(i)} = \sum _{i=1}^{d} \beta ''_{i} \hat {\mathcal {U}}^{(i)} + \sum _{i>d} G_{i}(\beta '') \hat {\mathcal {U}}^{(i)}\tag{4}\end{equation*}$$ in terms of the new vector $\beta = (\beta _{1}, \beta _{2}, \ldots, \beta _{L})^{T}$ such that $\sigma _{\beta _{1}} \geq \sigma _{\beta _{2}} \geq \ldots \geq \sigma _{\beta _{L}}$ . It is worth to notice that as the values $\sigma _{\beta _{i}}$ are in decreasing order, then the higher the index $i$ , the lower the importance of the corresponding component. An overview diagram that explains the main steps of the approach is reported in Fig. 1.

• On the basis of this result, a low-dimensional representation of tensor $\mathcal {X}$ can be simply obtained by truncating the summation in (4) to the first $r$ terms

  $$\begin{equation*} \mathcal {X} \cong \sum _{i=1}^{r} \beta _{i} \, \hat {\mathcal {U}}^{(i)},\tag{5}\end{equation*}$$ thus obtaining a truncation error $$\begin{equation*} \mathcal {E}_{r} = \sum _{i>r} \sigma _{\beta _{i}}\tag{6}\end{equation*}$$ that decreases as $r$ increases. Following this property the proposed technique has been called principal tensor embedding (PTE).

• Assuming the ID has been determined with one of the methods known in literature, to estimate the function $G(\cdot)$ an effective method for nonparametric input-output nonlinear function regression in tensor space, called nonparametric regression kernel (NPKR), will be used.

FIGURE 1.

Overview diagram of the proposed method.

It is worth to notice that the proposed approach satisfy nonlinearity, explicit modeling and ID estimation, i.e. all the key aspects of ML, thus representing a real advancement to previous techniques.

SECTION IV. The Finite Dimensional Linear Space of Tensors

Let us refer to tensors, regarded as multidimensional arrays and denoted by Euler script calligraphic letters, e.g. $\mathcal {X} \in \mathbb {R}^{I_{1} \times I_{2} \times \ldots \times I_{M}}$ , where $\times$ represents the Cartesian product. The number of dimensions $M$ , also known as modes, of a tensor denotes the order of a tensor. The elements of an $M$ -order tensor $\mathcal {X}$ will be represented by

$$\begin{equation*} x_{i_{1}, i_{2}, \ldots, i_{M}}, \;\; i_{l} = 1,2, \ldots, I_{l}, \;\; l = 1,2, \ldots, M.\tag{7}\end{equation*}$$

The inner product of two tensors of the same size $\mathcal {X}, \, \mathcal {Y} \in \mathbb {R}^{I_{1} \times I_{2} \times \ldots I_{M}}$ is defined as

$$\begin{equation*} \left \langle{ \mathcal {X}, \mathcal {Y} }\right \rangle = \sum _{i_{1} = 1}^{I_{1}} \ldots \sum _{i_{M} = 1}^{I_{M}} x_{i_{1} i_{2} \ldots i_{M}} y_{i_{1} i_{2} \ldots i_{M}}.\tag{8}\end{equation*}$$ From this definition it follows that the norm of a tensor is given by $$\begin{equation*} \lVert \mathcal {X} \rVert = \sqrt {\left \langle{ \mathcal {X}, \mathcal {X} }\right \rangle }.\tag{9}\end{equation*}$$

The outer product of tensor $\mathcal {X} = x_{i_{1}\,\,i_{2} \ldots i_{M}} \in \mathbb {R}^{I_{1} \times I_{2} \times \ldots \times I_{M}}$ with tensor $\mathcal {Y} = y_{j_{1}\,\,j_{2} \ldots j_{L}} \in \mathbb {R}^{J_{1} \times J_{2} \times \ldots \times J_{L}}$ is the $(M + L)$ -order tensor $\mathcal {Z}$ defined as

$$\begin{equation*} \mathcal {Z} = \mathcal {X} \circ \mathcal {Y}\tag{10}\end{equation*}$$ where the generic element of $\mathcal {Z}$ is given by $$\begin{equation*} z_{i_{1} i_{2} \ldots i_{M} j_{1} j_{2} \ldots j_{L}} = x_{i_{1} i_{2} \ldots i_{M}} \cdot y_{j_{1} j_{2} \ldots j_{L}}\tag{11}\end{equation*}$$ In particular for two vectors $x$ and $y$ the generic element of outer product $\mathcal {Z} = x \circ y$ is the matrix $$\begin{equation*} z_{ij} = x_{i} \, y_{j}.\tag{12}\end{equation*}$$

With reference to the canonical basis $e_{1} = (1, 0, \ldots, 0), \ldots, e_{I_{l}} = (0,0,\ldots,1)$ for $\mathbb {R}^{I_{l}}$ , an $M$ order tensor $\mathcal {X}$ can be decomposed as

$$\begin{equation*} \mathcal {X} = \sum _{i_{1} = 1}^{I_{1}} \ldots \sum _{i_{M} = 1}^{I_{M}} x_{i_{1} i_{2} \ldots i_{M}} \; e_{i_{1}} e_{i_{2}} \ldots e_{i_{M}}\tag{13}\end{equation*}$$ where the outer product $e_{i_{1}} e_{i_{2}} \ldots e_{i_{M}}$ is the $M$ -order canonical basis tensor. As this basis is of size $\prod _{j=1}^{M} I_{j}$ , thus the set of $M$ -order tensors form a linear space of dimensionality $L = \prod _{j=1}^{M} I_{j}$ . As an example for $M=2$ and $I_{1} = I_{2} = M$ we have $$\begin{align*} e_{1}~e_{1}=&\left ({\begin{array}{ccc} 1 &\quad 0 & \quad 0 \\ 0 &\quad 0 &\quad 0 \\ 0 &\quad 0 &\quad 0 \end{array} }\right), \; \\ e_{1}~e_{2}=&\left ({\begin{array}{ccc} 0 &\quad 1 &\quad 0 \\ 0 &\quad 0 &\quad 0 \\ 0 &\quad 0 &\quad 0 \end{array} }\right), \; \\ e_{1}~e_{3}=&\left ({\begin{array}{ccc} 0 &\quad 0 &\quad 1 \\ 0 &\quad 0 &\quad 0 \\ 0 &\quad 0 &\quad 0 \end{array} }\right), \; \ldots\tag{14}\end{align*}$$

For easy of reference Table 1 reports some notations, that will be frequently used in the following.

TABLE 1 Basic Notation

SECTION V. A Basis from Data

One of the main problems in representing elements of a linear space, is to find a basis. For vectors $v \in \mathbb {R}^{I_{1}}$ the problem can be solved estimating the covariance function

$$\begin{equation*} R_{vv} = E \{ v v^{T} \}\tag{15}\end{equation*}$$ from data. Assuming a set $\{ v^{(1) }, v^{(2) }, \ldots,v^{(N)} \}$ of observations is collected, then (15) can be approximated as $$\begin{equation*} R_{vv} \cong \frac {1}{N} V V^{T}\tag{16}\end{equation*}$$ where $V = [v^{(1) }, v^{(2) }, \ldots, v^{(N)}] \in \mathbb {R}^{I_{1} \times N}$ is the data matrix. Thus, once an estimation of $R_{vv}$ is derived, the problem reduces to the decomposition of $R_{vv}$ $$\begin{equation*} R_{vv} = U \Sigma U^{T}\tag{17}\end{equation*}$$ By noting that $v v^{T} = v \circ v$ , (15) can be generalized to tensor space by simply substituting $v$ with $\mathcal {X}$ , thus obtaining $$\begin{equation*} R_{\chi \chi } = E \{ \mathcal {X} \circ \mathcal {X} \}\tag{18}\end{equation*}$$ where $R_{\chi \chi }$ is a tensor of $2M$ order. Even though some techniques for the decomposition of a tensor are known in literature, the dimension of tensor $R_{\chi \chi }$ in (18) can be very large, so that these approaches cannot be used in practice.

A more effective approach for this purpose can be derived using the well known Gram-Schmidt procedure. Extracting from a given dataset $\Omega = \{ \mathcal {X}^{(1) }, \mathcal {X}^{(2) }, \ldots, \mathcal {X}^{(N)} \}$ , $L = \prod _{j=1}^{M} I_{j}$ observations $\Lambda = \{ \hat {\mathcal {X}}^{(1) }, \hat {\mathcal {X}}^{(2) }, \ldots, \hat {\mathcal {X}}^{(L)} \}$ and assuming they are independent each other, the tensor version of Gram-Schmidt procedure is as follows.

$$\begin{align*} 1.\quad \mathcal {Y}^{(1) }=&\hat {\mathcal {X}}^{(1) }, \;\; \mathcal {U}^{(1) } = \frac {\mathcal {Y}^{(1) }}{\lVert \mathcal {Y}^{(1) } \rVert } \\ 2.\quad \mathcal {Y}^{(2) }=&\hat {\mathcal {X}}^{(2) } - \left \langle{ \hat {\mathcal {X}}^{(2) }, \mathcal {U}^{(1) } }\right \rangle \mathcal {U}^{(1) }, \;\; \mathcal {U}^{(2) } = \frac {\mathcal {Y}^{(2) }}{\lVert \mathcal {Y}^{(2) } \rVert } \\ k.\quad \mathcal {Y}^{(k)}=&\hat {\mathcal {X}}^{(k)} - \sum _{i=1}^{k-1} \left \langle{ \hat {\mathcal {X}}^{(k)}, \mathcal {U}^{(i)} }\right \rangle \mathcal {U}^{(i)}, \;\; \mathcal {U}^{(k)} = \frac {\mathcal {Y}^{(k)}}{\lVert \mathcal {Y}^{(k)} \rVert }. \\\tag{19}\end{align*}$$ It is straightforward to show that the tensors so obtained $\mathcal {U}^{(1) }, \mathcal {U}^{(2) }, \ldots, \mathcal {U}^{(L)}$ are orthonormal, meaning that $$\begin{align*} \left \langle{ \mathcal {U}^{(i)}, \mathcal {U}^{(j)} }\right \rangle = \begin{cases} 1, \, i = j \\ 0, \, i \neq j \\ \end{cases}.\tag{20}\end{align*}$$ Having derived a basis $\mathcal {U} = \{ \mathcal {U}^{(1) }, \mathcal {U}^{(2) }, \ldots, \mathcal {U}^{(L)} \}$ , then the generic tensor $\mathcal {X} \in \mathbb {R}^{I_{1} \times I_{2} \times \ldots \times I_{M}}$ can be represented as the summation $$\begin{equation*} \mathcal {X} = \sum _{i=1}^{L} \alpha _{i} \mathcal {U}^{(i)}, \;\; L = I_{1} \cdot I_{2} \cdot \ldots \cdot I_{M}\tag{21}\end{equation*}$$ where $\alpha _{i} = \left \langle{ \mathcal {X}, \mathcal {U}^{(i)} }\right \rangle$ is the i-th coordinate with respect to the element $\mathcal {U}^{(i)}$ of the basis. Due to the randomness of $\mathcal {X}$ thus $\alpha = (\alpha _{1}, \alpha _{2}, \ldots, \alpha _{L})^{T}$ is a realization of a random vector $\boldsymbol {\alpha }$ . Here a bold face character is used for random variables. In such a way the correspondence $\xi$ $$\begin{equation*} \xi: \; \alpha \in \mathbb {R}^{L} \Longleftrightarrow \mathcal {X} = \sum _{i=1}^{L} \boldsymbol {\alpha _{i}} \mathcal {U}^{(i)}\tag{22}\end{equation*}$$ is defined, or that is the same $$\begin{equation*} \xi: \; \alpha \in \mathbb {R}^{L} \Longleftrightarrow \boldsymbol {\alpha } \in \boldsymbol {H}\tag{23}\end{equation*}$$ where $\boldsymbol {H}$ is the space of random variables. For every observation $\mathcal {X}^{(k)}$ of dataset $\Omega$ , let us determine the vector $$\begin{equation*} \alpha ^{(k)} = (\alpha ^{(k)}_{1}, \ldots, \alpha ^{(k)}_{L})^{T}, \; k = 1, \ldots, N\tag{24}\end{equation*}$$ where $$\begin{equation*} \alpha ^{(k)}_{i} = \left \langle{ \mathcal {U}^{(i)}, \mathcal {X}^{(k)} }\right \rangle, \; i = 1, \ldots, L, \; k = 1, \ldots, N \;.\tag{25}\end{equation*}$$ Thus the following correspondence $$\begin{equation*} A = [\alpha ^{(1) }, \ldots, \alpha ^{(N)}] \Longleftrightarrow \Omega = \{ \mathcal {X}^{(1) }, \ldots, \mathcal {X}^{(N)} \}\tag{26}\end{equation*}$$ holds, where $A \in \mathbb {R}^{L \times N}$ is the data matrix of the coefficient random vector $\boldsymbol {\alpha }$ . To represent this correspondence in a more compact way we use the concept of cell array in which the elements, the cells, are containers that can hold arrays of different sizes. Using this concept (26) becomes $$\begin{equation*} \{ A | \Omega \} = \{ \alpha ^{(k)} | \, \mathcal {X}^{(k)}, \; k = 1, \ldots, N\}.\tag{27}\end{equation*}$$

SECTION VI. Local Parametrization of Data

On the basis of previous results for a generic tensor $\mathcal {X}$ we can write

$$\begin{equation*} \mathcal {X} = \sum _{i=1}^{L} \alpha _{i} \, \mathcal {U}^{(i)}\tag{28}\end{equation*}$$ which establishes a one-to-one correspondence (an isomorphism) between $\mathbb {R}^{L}$ and the space of tensors.

To reduce the dimensionality of the above representation, a Manifold Learning (ML) approach will be used. In this context the problem can be formalized as follows.

To the dataset $\Omega = \{\mathcal {X}^{(1) }, \ldots, \mathcal {X}^{(N)} \} \subset \mathbb {R}^{I_{1} \times I_{2} \times \ldots I_{M}}$ in tensor space corresponds a set of $N$ data points $A = \{ \alpha ^{(1) }, \ldots, \alpha ^{(N)}\} \subset \mathbb {R}^{L}$ , which are assumed to be sampled from a manifold $\mathcal {M}$ of dimension $d$ embedded in the $L$ -dimensional observation space. We further assume that the data points do not contain either noise or outliers. This implies that a local parametrization

$$\begin{equation*} \boldsymbol {\alpha } = F(\boldsymbol {\theta }), \; \boldsymbol {\theta } \in \mathbb {R}^{d}, \; \boldsymbol {\alpha } \in \mathbb {R}^{L}\tag{29}\end{equation*}$$ exists, with $d < L$ . The dimension $d$ represents the so-called intrinsic dimension (ID), which may be interpreted as the minimum number of parameters required to represent the data [65].

From differential geometry [69], [70] it can be shown that assuming the values of $\alpha$ lie on a manifold $\mathcal {M}$ of dimension $d$ , then a local parametrization $\varphi$ exists represented by the graph of a function $G(\cdot)$ such that

$$\begin{align*}&\hspace {-.5pc} \boldsymbol {\alpha } = \varphi (\boldsymbol {\alpha ''}) = (\boldsymbol {\alpha ''}, G(\boldsymbol {\alpha ''}))^{T} = (\boldsymbol {\alpha ''}, \boldsymbol {\alpha '})^{T}, \\&\boldsymbol {\alpha ''} \in \mathbb {R}^{d}, \; \boldsymbol {\alpha '} \in \mathbb {R}^{m}, \; m = L-d\tag{30}\end{align*}$$ where $(\boldsymbol {\alpha ''}, \boldsymbol {\alpha '})^{T}$ is a partition of $\boldsymbol {\alpha }$ and $\boldsymbol {\alpha ''}, \boldsymbol {\alpha '}$ are row vectors. A proof of this result is reported in the Appendix IX for ease of reference.

Comparing (29) and (30) it clearly results $\boldsymbol {\theta } = \boldsymbol {\alpha ''}$ and $F(\cdot) = \varphi (\cdot)$ . In this way the data matrix $A$ can be partitioned accordingly

$$\begin{align*} A = \left [{ \begin{array}{c} A'' \\ A' \\ \end{array} }\right], \; A'' \in \mathbb {R}^{d \times N}, \; A' \in \mathbb {R}^{m \times N}\tag{31}\end{align*}$$ where $A''$ and $A'$ are the data matrices of $\boldsymbol {\alpha ''}$ and $\boldsymbol {\alpha '} = G(\boldsymbol {\alpha ''})$ , respectively.

A. Principal Tensor Embedding

Being $\boldsymbol {\alpha }$ a random variable, the mean vector $\mu _{\alpha } = (\mu _{\alpha _{1}}, \ldots, \mu _{\alpha _{L}})^{T}$

$$\begin{equation*} \mu _{\alpha } \triangleq E \{ \boldsymbol {\alpha } \} = \frac {1}{N} \sum _{k=1}^{N} \alpha ^{(k)} \in \mathbb {R}^{L \times 1}\tag{32}\end{equation*}$$ and the variance vector $\sigma _{\alpha } = (\sigma _{\alpha _{1}}, \ldots, \sigma _{\alpha _{L}})^{T}$ $$\begin{align*} \sigma _{\alpha }\triangleq&\textit {diag} \, E \{ (\boldsymbol {\alpha } - \mu _{\alpha }) (\boldsymbol {\alpha } - \mu _{\alpha })^{T} \} \\=&\textit {diag} \, \left [{ \frac {1}{N} \sum _{i=1}^{N} (\alpha ^{(k)} - \mu _{\alpha }) (\alpha ^{(k)} - \mu _{\alpha })^{T} }\right] \in \mathbb {R}^{L \times 1}\tag{33}\end{align*}$$ can be computed from the set $A$ , so that the following cell array $$\begin{equation*} \{ \sigma _{\alpha } | \, \mathcal {U} \} = \{ \sigma _{\alpha _{i}} | \, \mathcal {U}^{(i)}, \; i = 1, \ldots, L\}\tag{34}\end{equation*}$$ can be defined. If the elements $(\sigma _{\alpha _{1}}, \ldots, \sigma _{\alpha _{L}})$ of $\sigma _{\alpha }$ are put in decreasing order and the corresponding terms $\{ \mathcal {U}^{(1) }, \ldots, \mathcal {U}^{(L)} \}$ are ordered accordingly, then a new cell array is derived $$\begin{equation*} \{ \sigma _{\beta } | \, \hat {\mathcal {U}} \}\tag{35}\end{equation*}$$ where $\boldsymbol {\beta } = (\boldsymbol {\beta _{1}}, \ldots, \boldsymbol {\beta _{L}})^{T}$ is a random vector such that $\sigma _{\beta _{1}} \geq \sigma _{\beta _{2}} \geq \ldots \geq \sigma _{\beta _{L}}$ and a bold face character is used, following the convention previously adopted for random variables. In this way for every $\mathcal {X}^{(k)}$ it results $$\begin{equation*} \mathcal {X}^{(k)} = \sum _{i=1}^{L} \beta ^{(k)}_{i} \, \hat {\mathcal {U}}^{(i)}, \; k = 1, \ldots, N\tag{36}\end{equation*}$$ where the terms $$\begin{equation*} \beta ^{(k)}_{i} = \left \langle{ \hat {\mathcal {U}}^{(i)}, \mathcal {X}^{(k)} }\right \rangle, \quad i = 1, \ldots,L, \; k = 1, \ldots, N\tag{37}\end{equation*}$$ correspond to the cell arrays $$\begin{equation*} \{ \beta ^{(k)} | \, \hat {\mathcal {U}} \}, \quad k = 1, \ldots, N.\tag{38}\end{equation*}$$ It is worth to notice that when $\beta$ is used in normal face it represents an observation or realization of the random variable $\boldsymbol {\beta }$ . Having reordered the basis elements, the correspondence (27) becomes $$\begin{equation*} \{ B | \Omega \} = \{ \beta ^{(k)} | \, \mathcal {X}^{(k)}, \; k = 1, \ldots, N \}\tag{39}\end{equation*}$$ where $B = \left [{ \beta ^{(1) }, \ldots, \beta ^{(N)} }\right] \in \mathbb {R}^{L \times N}$ is the new data matrix of coefficients in the basis $\hat {\mathcal {U}} = \{ \hat {\mathcal {U}}^{(1) }, \ldots, \hat {\mathcal {U}}^{(L)} \}$ . As a main result of this reordering, to the dataset $B$ corresponds the new random vector $$\begin{equation*} \boldsymbol {\beta } = (\boldsymbol {\beta ''}, \boldsymbol {\beta '})^{T}\tag{40}\end{equation*}$$ where the components of $\boldsymbol {\beta }$ are such that $\sigma _{\beta _{1}} \geq \ldots \geq \sigma _{\beta _{L}}$ . Among the possible choices the vector $\boldsymbol {\alpha }$ can be partitioned, (40) has the following useful property. By choosing a generic index $r < L$ , (28) can be rewritten as $$\begin{equation*} \mathcal {X} = \sum _{i=1}^{r} \alpha _{i} \, \mathcal {U}^{(i)} + \sum _{i>r} \alpha _{i} \, \mathcal {U}^{(i)} = \mathcal {X}_{r} + \mathcal {N}\tag{41}\end{equation*}$$ where $\mathcal {X}_{r} = \sum _{i=1}^{r} \alpha _{i} \, \mathcal {U}^{(i)}$ . Assuming $\mathcal {X}_{r}$ is a good approximation of $\mathcal {X}$ and $\mu _{\alpha } = 0$ for the sake of notation simplicity, then the truncation error in approximating $\mathcal {X}$ with the first $r$ components is given by the norm of residual $\mathcal {N} = \mathcal {X} - \mathcal {X}_{r}$ , i.e., $$\begin{align*} \mathcal {E}_{r}=&E \{ \langle \mathcal {N}, \mathcal {N} \rangle \} = E \{ \langle \mathcal {X} - \mathcal {X}_{r}, \mathcal {X} - \mathcal {X}_{r} \rangle \} \\=&E \left\{{ \sum _{i>r} \boldsymbol {\alpha ^{2}_{i}} }\right\} = \sum _{i>r} \sigma _{\alpha _{i}}.\tag{42}\end{align*}$$ As the values $\sigma _{\beta _{i}}$ are in decreasing order, thus the partition (41) ensures the truncation error (42) is minimum when $\sigma _{\alpha _{i}} = \sigma _{\beta _{i}}$ .

With reference to the new partition (40), the local parametrization of corresponding data $\{\beta ^{(1) }, \ldots, \beta ^{(N)} \}$ can be written as

$$\begin{equation*} \boldsymbol {\beta } = \varphi (\boldsymbol {\beta ''}) = (\boldsymbol {\beta ''}, G(\boldsymbol {\beta ''})^{T} = (\boldsymbol {\beta ''}, \boldsymbol {\beta '})^{T}, \; \boldsymbol {\beta ''} \in \mathbb {R}^{d}\tag{43}\end{equation*}$$ and the generic observation of dataset $\Omega$ can be modeled by $$\begin{align*} \mathcal {X}^{(k)}=&\sum _{i=1}^{d} \beta ''^{(k)}_{i} \hat {\mathcal {U}}^{(i)} + \sum _{i>d} G_{i}(\beta ''^{(k)}) \hat {\mathcal {U}}^{(i)}, \; k = 1, \ldots, N \\ \beta ''^{(k)}_{i}=&\langle \hat {\mathcal {U}}^{(i)}, \mathcal {X}^{(k)} \rangle, \; i = 1, \ldots, d, \; k = 1, \ldots, N.\tag{44}\end{align*}$$ The data matrix $B$ of $\boldsymbol {\beta }$ can be partitioned accordingly $$\begin{align*} B = \left [{ \begin{array}{c} B'' \\ B' \\ \end{array} }\right], \; B'' \in \mathbb {R}^{d \times N}, \; B' \in \mathbb {R}^{m \times N}\tag{45}\end{align*}$$ where $B''$ and $B'$ are the data matrices of $\boldsymbol {\beta ''}$ and $\boldsymbol {\beta '} = G(\boldsymbol {\beta ''})$ respectively. As you can see from (44) the tensor dataset $\Omega$ of dimension $L \times N$ is represented by the data matrix $B''$ of dimension $d \times N$ , thus a reduction of complexity in representing data from $L \times N$ to $d \times N$ is obtained with the model (44). Formally this model is equivalent to the following correspondence $$\begin{equation*} B'' = [\beta ''^{(1) }, \ldots, \beta ''^{(N)}] \Longleftrightarrow \Omega = \{ \mathcal {X}^{(1) }, \ldots, \mathcal {X}^{(N)} \}\tag{46}\end{equation*}$$ which can be rewritten in a more compact form as $$\begin{equation*} \{ B'' | \, \Omega \} = \{\beta ''^{(k)} | \, \mathcal {X}^{(k)}, \; k = 1, \ldots, N \}.\tag{47}\end{equation*}$$ The values $\mathcal {X}^{(k)}$ in (44) can be interpreted as observations of a random tensor $\mathcal {X}$ , thus the following stochastic model $$\begin{align*} \mathcal {X} = \gamma (\boldsymbol {\beta ''}) = \sum _{i=1}^{d} \boldsymbol {\beta ''_{i}} \hat {\mathcal {U}}^{(i)} + \sum _{i>d} \boldsymbol {\beta '_{i}}(\boldsymbol {\beta ''}) \hat {\mathcal {U}}^{(i)}, \; \boldsymbol {\beta '} = G(\boldsymbol {\beta ''}) \\\tag{48}\end{align*}$$ that established a one-to-one correspondence between the random tensor $\mathcal {X}$ and the random vector $\boldsymbol {\beta ''}$ , holds. In the context of random models for tensors, $\boldsymbol {\beta ''}$ represents the vector of latent variables, that is a smaller set of variables that cannot be observed directly, and $\gamma (\boldsymbol {\beta ''})$ is a local parametrization of $\mathcal {X}$ . Once the new basis $\hat {\mathcal {U}}$ is obtained from $\mathcal {U}$ by the reordering procedure previously described, the hyperparameter vector $\beta ''^{(k)}$ for a generic observation $\mathcal {X}^{(k)}$ , can be easily derived by the inner product (44).

On the basis of previous results, a low-dimensional representation of the tensor $\mathcal {X}$ can be obtained by truncating the summation in (48) to the first $r$ terms

$$\begin{equation*} \mathcal {X} \cong \sum _{i=1}^{r} \beta _{i} \, \hat {\mathcal {U}}^{(i)} \quad,\tag{49}\end{equation*}$$ thus giving a truncation error $$\begin{equation*} \mathcal {E}_{r} \cong \sum _{i>r} \sigma _{\beta _{i}}\tag{50}\end{equation*}$$ that decreased as $r$ increases.

As the terms in (49) correspond to the most important components in the representation of the tensor $\mathcal {X}$ , this approach for tensor learning can be called principal embedded tensor (PTE) technique.

B. The Metric of the Manifold $\mathcal{M}$

A relationship between the metric on the manifold $\mathcal {M}$ , that is the geodesic distance of all pairs of points on the manifold, and the Euclidean metric of the corresponding points in $\mathbb {R}^{d}$ , can be derived as follows. By taking advantage of the one-to-one property of the correspondence (48), the geodesic distance between two points $\mathcal {X}^{(i)}$ and $\mathcal {X}^{(j)}$ on the manifold can be defined as

$$\begin{align*} d(\mathcal {X}^{(i)}, \mathcal {X}^{(j)}) \!= \!\lVert \beta ''^{(i)} - \beta ''^{(j)} \rVert \!=\! \lVert \gamma ^{-1} (\mathcal {X}^{(i)}) - \gamma ^{-1} (\mathcal {X}^{(j)}) \rVert \\\tag{51}\end{align*}$$ where $\beta ''^{(i)}, \, \beta ''^{(j)}$ are the corresponding points in low-dimensional space $\mathbb {R}^{d}$ and $\gamma ^{-1}$ is the inverse of $\gamma$ . Besides a relationship for the Euclidean distance between two points $\mathcal {X}^{(i)}$ and $\mathcal {X}^{(j)}$ on the manifold is given by $$\begin{equation*} \lVert \mathcal {X}^{(i)} - \mathcal {X}^{(j)} \rVert = \lVert \gamma (\beta ''^{(i)}) - \gamma (\beta ''^{(j)}) \rVert.\tag{52}\end{equation*}$$ Using the property of differentiability of local parametrization we can apply the first-order Taylor expansion at $\beta ''_{0}$ to represent a generic point on the manifold at $\beta ''$ $$\begin{align*} \mathcal {X} = \gamma (\beta '') = \gamma (\beta ''_{0}) + J(\gamma) (\beta '' - \beta ''_{0})^{T} + o(\lVert \beta '' - \beta _{0}'' \rVert) \\\tag{53}\end{align*}$$ where $J(\gamma)$ is the Jacobian of $\gamma$ . Choosing $\beta ''^{(j)} = \beta ''_{0}$ and $\beta ''^{(i)} = \beta ''$ and substituting (53) into (52) we have $$\begin{equation*} \lVert \mathcal {X}^{(i)} - \mathcal {X}^{(j)} \rVert \!=\! \lVert J(\gamma) (\beta '' - \beta _{0})^{T} \!+\! o(\lVert \beta '' - \beta ''_{0}) \rVert) \rVert.\tag{54}\end{equation*}$$ This relationship clearly shows that the geodesic distance between two points $\mathcal {X}^{(i)}, \mathcal {X}^{(j)}$ , defined by (51), is different from their Euclidean distance.

SECTION VII. Nonparametric Kernel Regression (NPKR)

Assuming the ID of dataset $B$ has been determined with one of the methods known in literature [71]–​[75], the unsupervised learning of the stochastic process (s.p.) $\mathcal {X}$ that generates the data $\Omega$ , reduces to the estimation of the input-output function $G(\cdot)$ in (43). In this way the initial problem of unsupervised learning reduces to a supervised learning problem as the input data $B''$ of $G(\cdot)$ are known. The function $G(\cdot)$ represents a mapping (in general nonlinear) from data $B''$ in the low-dimensional feature space to high-dimensional data $B'$ . The estimation of this function is a regression problem that can be solved using several different approaches.

In this context an effective method for non parametric input-output nonlinear function regression in tensor space has been recently proposed [76]. The method can be summarized as follows. Given any continuous and bounded function $f(x)$ of the $n$ -dimensional variable $x = (x^{1}, \ldots, x^{n})$ , defined in a compact subset $I \in \mathbb {R}^{n}$ , then some sequences $k_{m}(x)$ , named kernel functions, exists such that the convolution

$$\begin{equation*} f_{m}(x) = k_{m} * f(x) = \int _{I} f(t) k_{m}(x-t) dt\tag{55}\end{equation*}$$ converges uniformly to $f(x)$ on $I$ , as $m \rightarrow \infty$ . Examples of these functions are:

1. the polynomial kernel defined as

   $$\begin{align*} k_{m}(x) = \begin{cases} \dfrac {(1 - \lVert x \rVert ^{2})^{m}}{C_{m}}, & \lVert x \rVert \leq 1 \\ 0, & \lVert x \rVert > 1 \end{cases}\tag{56}\end{align*}$$ where $\lVert x \rVert = (x^{T} x)^{1/2}$ is the norm of $X$ and $C_{m}$ is a normalized factor given by $$\begin{equation*} C_{m} = \int _{I} (1 - \lVert t \rVert ^{2})^{m} dt\tag{57}\end{equation*}$$

2. the Gaussian kernel defined as

   $$\begin{equation*} k_{m}(x) = \frac {m^{n}}{(2 \pi)^{n/2}} \exp \left ({- \frac {1}{2} m^{2} \lVert x \rVert ^{2} }\right).\tag{58}\end{equation*}$$

   As a consequence of property (55) we have

   $$\begin{equation*} f(x) \cong f_{m}(x), \; \text {for} \;\; m \gg 1\tag{59}\end{equation*}$$ which can be considered as a universal approximating relationship. The main issue of (59) is that it requires calculating the integral on the right hand side of (55). To overcome this problem, suppose we want to compute the mean value of the $n$ -dimensional function $g(t), t=(t^{1},\ldots,t^{n})$ , in the interval $I$ $$\begin{equation*} E(g(t)) = \int _{I} g(t) p(t) dt\tag{60}\end{equation*}$$ where $t$ is a realization of the random variable (r.v.) $\boldsymbol {t = (t^{1}, \ldots, t^{n})}$ , with probability density function (pdf) $p(t)$ , and $E(\cdot)$ denotes the expected value. To numerically solve the integral in (60) a Monte Carlo integration technique [77], [78] can be derived as follows. Let us select at random $N$ points $(t^{1}, \ldots, t^{N})$ sampled from pdf $p(t)$ , then the Monte Carlo approximation of (60) is $$\begin{equation*} E(g(t)) \cong \frac {1}{N} \sum _{i=1}^{N} g(t_{i}).\tag{61}\end{equation*}$$ By applying this approach to the convolution $g_{m}(x) = k_{m} * g(x)$ , where $g(x) = f(x) p(x)$ , we have $$\begin{align*} g_{m}(x)=&k_{m} * g(x) = E(f(t) k_{m}(x-t)) \\\cong&\frac {1}{N} \sum _{i=1}^{N} f(t_{i}) k_{m}(x - t_{i})\tag{62}\end{align*}$$ and in particular for $f(x) = 1(x) = \{ 1 | x \in I\}$ $$\begin{align*} 1_{m}(x) = k_{m} * p(x) = E(k_{m}(x-t)) \cong \frac {1}{N} \sum _{i=1}^{N} k_{m} (x-t_{i}). \\\tag{63}\end{align*}$$ Combining (62) and (63) we finally get $$\begin{equation*} f(x) \cong f_{m}(x) = \frac {\sum _{i=1}^{N} f(t_{i}) k_{m}(x - t_{i})}{\sum _{i=1}^{N} k_{m}(x - t_{i})}.\tag{64}\end{equation*}$$ The function approximation (64) is a non-parametric model of function $f(x)$ as it only depends on the observations $f(t_{i})$ and not on parameters to be estimated. The method described above, named nonparametric kernel regression (NPKR), can be used for the approximation of the function $G(\cdot)$ in (48), thus giving the following relationship $$\begin{equation*} G(\beta '') \cong G_{m}(\beta '') = \frac {\sum _{k=1}^{N} \beta ^{'(k)} \, k_{m} (\beta '' - \beta ''^{(k)})}{\sum _{k=1}^{N} k_{m} (\beta '' - \beta ''^{(k)})}\tag{65}\end{equation*}$$ where $k_{m}(\cdot)$ is a given kernel function, $\beta ''^{(k)}$ and $\beta '^{(k)}$ are the input and output training points respectively and $\beta ''$ is a testing point, chosen from data matrix $B'' = [\beta ''^{(1) }, \ldots, \beta ''^{(N)}]$ . It is worth to notice that the previous relationship has the same form as the well-known Nadaraya-Watson nonparametric kernel estimator [79]–​[81] that was proposed for the estimation of the regression function of data $(x_{1}, y_{1}), \ldots, (x_{n}, y_{n})$ sampled from a population having a density $f(x,y)$ , thus giving a link between the two theories.

SECTION VIII. Experiments

The experiments for the validation of the proposed tensor learning approach address two different problems, namely data reconstruction and classification.

Data reconstruction aims at reconstructing the original dataset $\Omega = \{ \mathcal {X}^{(1) }, \ldots, \mathcal {X}^{(N)} \}$ by the embedded vectors $B'' = \{ \beta ^{''(k)}, \; k = 1, \ldots, N \}$ using the model given by (48) and (65).

A pseudo-code of the algorithm used for the estimation of the model from the dataset $\Omega$ is reported in Algorithm 1.

SECTION Algorithm 1 PTE

INPUT:

dataset

$\Omega = \{ \mathcal {X}^{(1) }, \ldots, \mathcal {X}^{(N)} \}, \, \mathcal {X}^{(i)} \in \mathbb {R}^{I_{1} \times \ldots \times I_{M}}, \, L = \prod _{j=1}^{M} I_{j}$

1.

Extract $L$ observations at random

$\Lambda = \{\hat {\mathcal {X}}^{(1)}, \ldots, \hat {\mathcal {X}}^{(L)} \}$

2.

Compute a basis by Gram-Schmidt procedure

$\mathcal {U} = \{ \mathcal {U}^{(1) }, \ldots, \mathcal {U}^{(L)} \}$

3.

Compute the data matrix $A = [\alpha ^{(1) }, \ldots, \alpha ^{(N)}]$ such that

$\alpha _{i}^{(k)} = \left \langle{ \mathcal {U}^{(i)}, \mathcal {X}^{(k)} }\right \rangle, \, i\,\,= 1, \ldots, L, \; k = 1, \ldots, N$

4.

Determine the reordered basis $\hat {\mathcal {U}} = \{ \hat {\mathcal {U}}^{(1) }, \ldots, \hat {\mathcal {U}}^{(L)} \}$ such that $\sigma _{\beta _{1}} \geq \sigma _{\beta _{2}} \geq \ldots \geq \sigma _{\beta _{L}}$ ,

$\beta _{i}^{(k)} = \left \langle{ \hat {\mathcal {U}}^{(i)}, \mathcal {X}^{(k)} }\right \rangle, \, i\,\,= 1, \ldots, L, \; k = 1, \ldots, N$ 5. Estimate the intrinsic dimension $d$ of data matrix $B = [\beta ^{(1) }, \ldots, \beta ^{(N)}]$ and extract $B''$ such that

$\begin{aligned} B = \left [{ \begin{array}{c} B''\\ B' \end{array} }\right], \, B'' \in \mathbb {R}^{d \times N} \end{aligned}$

6.

Approximate the function $\beta ' = G(\beta '')$ by NPKR,

with $G_{m}(\beta '')$ given by (65)

OUTPUT:

the model of dataset $\Omega$

$\mathcal {X} = \gamma (\beta '') = \sum _{i=1}^{d} \beta ''_{i} \hat {\mathcal {U}}^{(i)} + \sum _{i>d} \beta '_{i}(\beta '') \hat {\mathcal {U}}^{(i)}, \, \beta ' = G(\beta '')$

As far as the extraction of $L$ observations is concerned, the following considerations are useful. A complete basis can be derived provided the number $N$ of observations is larger than the dimensionality $L$ of tensor $\mathcal {X}$ . In this case to obtain a set of non-zero orthonormal tensors the selected $L$ data are required to be independent. In general this assumption is naturally satisfied since each element in the database is obtained independently from each other. Nevertheless independence can be easily proven by checking that the elements obtained by the Gram-Schmidt procedure are non-zero, as they are a linear combination of dataset. In case we have less data, i.e. $N < L$ , a complete basis cannot be obtained but the method for manifold nonlinear dimensionality reduction can still be applied provided the condition $d \ll N$ is satisfied. However, in this case, the main consequence of having a reduced dataset is a large error in the estimation of data ID, as it will be discussed in the experiment VIII-B for classification.

In order to visually assess the quality of Algorithm 1, a simple experiment on a synthetic dataset was preliminary performed. To this end data $\mathcal {X} \in \mathbb {R}^{3}$ in a low dimensional space were generated by the following parametrized function

$$\begin{equation*} \mathcal {X} = \alpha ''_{1} \phi ^{(1) } + \alpha ''_{2} \phi ^{(2) } + G(\alpha ''_{1}, \alpha ''_{2}) \phi ^{(3) }\tag{66}\end{equation*}$$ where $$\begin{equation*} \alpha ' = G(\alpha ''_{1}, \alpha ''_{2}) = (\alpha ''_{1})^{3} - 3 \alpha ''_{1} (\alpha ''_{2})^{2}\tag{67}\end{equation*}$$ and the vectors $$\begin{align*} \phi ^{(1) }=&\left({\begin{array}{lll} 1 &\quad 0 &\quad 0 \end{array} }\right)^{T}, \\ \phi ^{(2) }=&\left({\begin{array}{lll} 0 &\quad 1 &\quad 0 \end{array} }\right)^{T}, \, \phi ^{(3) } = \left({\begin{array}{lll} 0 &\quad 0 &\quad 1 \end{array} }\right)^{T}\tag{68}\end{align*}$$ form the canonical basis $\phi = \left [{ \phi ^{(1) }, \phi ^{(2) }, \phi ^{(3) } }\right] \in \mathbb {R}^{3\times 3}$ . (66) represents a parametrized surface in $\mathbb {R}^{3}$ known as Monkey Saddle whose behaviour is plotted in Fig. 2. From (66) a dataset $\Omega = \{ \mathcal {X}^{(1) }, \ldots, \mathcal {X}^{(N)} \}$ was achieved by $N$ values of $\alpha '' = (\alpha ''_{1}, \alpha ''_{2})$ randomly chosen in the interval $[-1.5, \, 1.5] \times [-1.5, \, 1.5]$ , thus obtaining the data matrix $A = [\alpha ^{(1) }, \ldots, \alpha ^{(N)}] \in \mathbb {R}^{3 \times N}$ , where $\alpha = (\alpha '', \alpha ')^{T}$ . Using $\Omega$ as the input dataset in Algorithm 1, the model $\mathcal {X} = \gamma (\beta '')$ has been derived. Fig. 3 depicts the surface achieved with the points sampled from the model. As you can see the model is able to reconstruct the manifold embedded in the high-dimensional space of data.

FIGURE 2.

Monkey Saddle surface.

FIGURE 3.

Monkey Saddle surface reconstructed.

In all the experiments a Gaussian kernel was used for the regression of function $G(\cdot)$ in (48), using the NPKR method.

Classification aims at classifying the data $\Omega$ by the kNN algorithm, using the embedded vectors $B''$ as low-dimensional features.

A. Data Reconstruction

The capability of the proposed method to model data with a reduced dimensionality, has been validated by three experiments conducted on different datasets, namely CIFAR-10, RGB-D Object Dataset, AT&T Faces Dataset.

1) Experiment on CIFAR-10 (3D-Tensor)

CIFAR-10 dataset [82], [83] consists of 60000, $32 \times 32$ colour images divided in 10 classes, with 6000 images per class, of which 50000 are for training and 10000 for testing.

In this experiment, we use the 50000 RGB images of the training set for image reconstruction by the model (48)–​(65). For this purpose, the data has been organized in a 3D-tensor dataset $\Omega = \{ \mathcal {X}^{(1) }, \ldots, \mathcal {X}^{(N)} \}$ , with $N = 50000, \; \mathcal {X} \in \mathbb {R}^{32 \times 32 \times 3}$ , so that the dimension of the basis is $L = 32 \cdot 32 \cdot 3 = 3072$ . Once a basis $\mathcal {U}$ is derived with the Gram-Schmidt procedure, to the set $\Omega$ corresponds the data matrix $A = \left [{ \alpha ^{(1) }, \ldots, \alpha ^{(N)} }\right]$ where the columns can be considered as realizations of the random vector $\alpha$ . For the set chosen in this experiment the estimated value of the vectors $\mu _{\alpha }$ and $\sigma _{\alpha }$ are reported in Fig. 4.

FIGURE 4.

Vectors $\mu _{\alpha }$ and $\sigma _{\alpha }$ for Experiment VIII-A1 (x-axis truncated to value 500).

To estimate the intrinsic dimension $d$ of data matrix $B$ we used the following relevant state-of-the-art intrinsic dimension (ID) estimators: Dimensionality from Angle and Norm Concentration (DANCo) and its faster variant (FastDANCo) [84], [85], Minimum Neighbor Distance - Maximum Likelihood (MiND_ML) and Minimum Neighbor Distance - Kullback Leibler (MiND_KL) [86], Maximum Likelihood Estimation (MLE) [87], Intrinsic Dimensionality Estimation of Submanifolds in $\mathcal {R}^{d}$ (Hein) [88]. Table 2 reports the values of intrinsic dimension as obtained with the author’s Matlab implementation¹ of the above mentioned methods for ID estimation. As you can see, although the value of ID so obtained show a large spread, they are all of the same order and significantly reduce the dimensionality of tensor data, which is two-orders higher ($L = 3072$ ). To stress the model and prove the dimensionality reducing capability of the approach we chose the minimum value of ID.

TABLE 2 ID of the Data Matrix $B$ Achieved From CIFAR-10 Dataset, as Estimated With the Methods DANCo, FastDANCo, MiNDML, MiNDKL, MLE and Hein

Fig. 5 shows a set of original images and the corresponding images reconstructed with the model (48)–​(65) and $d=10$ . Table 3 reports the root mean squared error (RMSE) for each processed images, computed using the NPKR method for regression of function $G(\cdot)$ in (48), (here a Gaussian kernel and $m=2$ are used in (58)), and compared with two well-known regression methods, namely Support Vector Machine (SVM) (with different kernels) and Regression Tree. Table 3 shows that the NPKR method for the regression of the function $G(\cdot)$ gives the better results, as it is able to reconstruct the data with the minimum error.

TABLE 3 RMSE for the Reconstruction of Random Images Extracted From CIFAR-10 Dataset

FIGURE 5.

Comparison of the original images extracted from the CIFAR-10 dataset and the corresponding reconstructed images with the proposed approach.

In order to study how robust is the NPKR method with respect to hyperparameters, the sensitivity of RMSE and PSNR (peak signal-to-noise ratio) to the dimension $d$ of $\beta ''$ is reported in Fig. 6 and Fig. 7 for different values of $m$ . The mathematical representation of the PSNR is as follows:

$$\begin{equation*} PSNR = 20 \log _{10} \left ({\frac {max(f)}{\sqrt {MSE}} }\right)\tag{69}\end{equation*}$$ where $f$ represents the original image and MSE the mean squared error. PSNR and MSE are used to compare the squared error between the original image and the reconstructed image. There is an inverse relationship between PSNR and MSE. So a higher PSNR value indicates a higher quality of the image.

FIGURE 6.

Sensitivity of RMSE to the dimension $d$ of $\beta ''$ for different values of $m$ for Experiment VIII-A1.

FIGURE 7.

Sensitivity of PSNR to the dimension $d$ of $\beta ''$ for different values of $m$ for Experiment VIII-A1.

2) Experiment on RGB-D Object Dataset (4D-Tensor)

The RGB-D Object Dataset [89] contains 300 objects divided in 51 categories. For each object the dataset provides a number of images ranging from a minimum of 506 to a maximum of 852 for a total of 207920 frames. In this experiment we used the subset Cropped RGB and depth images with object segmentation masks [90] that contains the cropped RGB-D frames and tightly include the object as it is spun around on a turntable. We used all the 207920 images, resized into $32 \times 32$ pixel box. In particular, we divided each class in 5 frames of $32 \times 32 \times 3$ RGB images to obtain a 4D-tensor dataset $\Omega = \{ \mathcal {X}^{(1) }, \ldots, \mathcal {X}^{(N)} \}$ , with $N = 207920, \; \mathcal {X} \in \mathbb {R}^{32 \times 32 \times 3 \times 5}$ , resulting in a dimension $L = 15360$ of the basis. Once a basis $\mathcal {U}$ is derived with the Gram-Schmidt procedure, to the set $\Omega$ corresponds the data matrix $A = \left [{ \alpha ^{(1) }, \ldots, \alpha ^{(N)} }\right]$ where the columns can be considered as realizations of the random vector $\alpha$ . For the set chosen in this experiment the estimated value of the vectors $\mu _{\alpha }$ and $\sigma _{\alpha }$ are reported in Fig. 8.

FIGURE 8.

Vectors $\mu _{\alpha }$ and $\sigma _{\alpha }$ for Experiment VIII-A2 (x-axis truncated to value 500).

Table 4 reports the results obtained with the same ID estimators used in Experiment VIII-A1. Choosing an intrinsic dimension $d=5$ for data matrix $B''$ , thus a reduction of dimensionality from $L=15360$ to $d=5$ is obtained with the model (48). Then we applied the proposed reconstruction method to different 4D-tensors, that is $32 \times 32$ RGB videos of 5 frames each.

TABLE 4 ID of the Data Matrix $B$ Achieved From the RGB-D Object Dataset, as Estimated With the Methods DANCo, FastDANCo, MiNDML, MiNDKL, MLE and Hein

Figs. 9–​13 show the 5 frames that composed the original videos and the corresponding reconstructed frames obtained with the proposed approach, demonstrating the validity of this approach.

FIGURE 9.

Comparison of the 5 frames (extracted from the RGB-D Object Dataset) that composed the original video V1 and the corresponding reconstructed frames obtained with the proposed approach.

FIGURE 10.

Comparison of the 5 frames (extracted from the RGB-D Object Dataset) that composed the original video V2 and the corresponding reconstructed frames obtained with the proposed approach.

FIGURE 11.

Comparison of the 5 frames (extracted from the RGB-D Object Dataset) that composed the original video V3 and the corresponding reconstructed frames obtained with the proposed approach.

FIGURE 12.

Comparison of the 5 frames (extracted from the RGB-D Object Dataset) that composed the original video V4 and the corresponding reconstructed frames obtained with the proposed approach.

FIGURE 13.

Comparison of the 5 frames (extracted from the RGB-D Object Dataset) that composed the original video V5 and the corresponding reconstructed frames obtained with the proposed approach.

Table 5 reports the RMSE for each processed 4D-tensors, obtained using NPKR method with Gaussian kernel and $m=5$ in (58), showing that the method is able to reconstruct the data with a very low error.

TABLE 5 RMSE for the Reconstruction of Random Sequences of Images Extracted From RGB-D Object Dataset

In this case, as well, to assess the robustness of the PTE method with respect to hyperparameters, the sensitivity of RMSE and PSNR to the dimension $d$ of $\beta ''$ is reported in Fig. 14 and Fig. 15 for different values of $m$ .

FIGURE 14.

Sensitivity of RMSE to the dimension $d$ of $\beta ''$ for different values of $m$ for Experiment VIII-A2.

FIGURE 15.

Sensitivity of PSNR to the dimension $d$ of $\beta ''$ for different values of $m$ for Experiment VIII-A2.

3) Experiment on AT&T Faces Dataset (2D-Tensor)

The AT&T Faces Dataset [91], [92] contains 10 different images for each of 40 distinct peoples. The size of each image is $92 \times 112$ , with 256 gray levels per pixel.

In this experiment, we applied the proposed approach on the original and occluded images achieved from AT&T Faces Dataset, with the same procedure described in [61]. Here, 20 distinct persons are selected and each face image is resized into $56 \times 46$ format. In addition to original face data, we also test our method on the partially occluded face data. Here, 20% images were selected randomly and corrupted manually for each person class, and the size of corruption is $11 \times 10$ . To apply the proposed method, the data has been organized in a 2D-tensor dataset $\Omega = \{ \mathcal {X}^{(1) }, \ldots, \mathcal {X}^{(N)} \}$ , with $N = 200, \; \mathcal {X} \in \mathbb {R}^{56 \times 46}$ resulting in a dimension of the basis of $L = 2576$ . Once a basis $\mathcal {U}$ is derived with the Gram-Schmidt procedure, to the set $\Omega$ corresponds the data matrix $A = \left [{ \alpha ^{(1) }, \ldots, \alpha ^{(N)} }\right]$ where the columns can be considered as realizations of the random vector $\alpha$ . For the set chosen in this experiment the estimated value of the vectors $\mu _{\alpha }$ and $\sigma _{\alpha }$ are reported in Fig. 16.

FIGURE 16.

Vectors $\mu _{\alpha }$ and $\sigma _{\alpha }$ for Experiment VIII-A3 (x-axis truncated to value 200).

On the basis of the estimated ID values reported in Table 6, we select an optimal intrinsic dimension $d=6$ for the reconstruction of this dataset.

TABLE 6 ID of the Data Matrix $B$ Achieved From the AT&T Dataset, as Estimated With the Methods DANCo, FastDANCo, MiNDML, MiNDKL, MLE and Hein

Also in this case to assess the robustness of the PTE method with respect to hyperparameters, the sensitivity of RMSE and PSNR to the dimension $d$ of $\beta ''$ is reported in Fig. 17 and Fig. 18 for different values of $m$ .

FIGURE 17.

Sensitivity of RMSE to the dimension $d$ of $\beta ''$ for different values of $m$ for Experiment VIII-A3.

FIGURE 18.

Sensitivity of PSNR to the dimension $d$ of $\beta ''$ for different values of $m$ for Experiment VIII-A3.

Fig. 19 and Fig. 20 show a set of original images and the corresponding reconstructed images with the model (48)–​(65), demonstrating the validity of this approach. Furthermore, Fig. 21 and Fig. 22 report a set of original partially occluded images and the corresponding reconstructed images showing the good noise tolerance of the method. Table 7 reports the reconstruction residuals for different reconstruction methods. The methods used to evaluate the PTE method are: PCA-based methods (PCA [27], gLPCA [66]) and tensor decomposition methods (TD [37], GLTD [61]). In this table the average residual for reconstructed tensor $\text {Res}(\mathcal {X})$ has been defined as:

$$\begin{equation*} \text {Res}(\mathcal {X}) = \sqrt { \frac {1}{N} \sum _{k=1}^{N} \lVert \mathcal {X}_{0}^{(k)} - \mathcal {X}^{(k)} \rVert ^{2} },\tag{70}\end{equation*}$$ where $\mathcal {X}_{0} = (\mathcal {X}_{0}^{(1) }, \cdots, \mathcal {X}_{0}^{(N)})$ represents the original $N$ images and $\mathcal {X} = (\mathcal {X}^{(1) }, \cdots, \mathcal {X}^{(N)})$ the reconstructed images. Meanwhile, for the occluded image data, the same equation (70) defines the average noise-free residual (NF-Res), being $\mathcal {X}_{0}$ in this case the set of original non occluded signals.

TABLE 7 Residual of Different Methods on Both Original AT&T Face and AT&T Noisy Face Datasets, Respectively

FIGURE 19.

Comparison of the original images extracted from the AT&T dataset and the corresponding reconstructed images with the proposed approach, on subject 8.

FIGURE 20.

Comparison of the original images extracted from the AT&T dataset and the corresponding reconstructed images with the proposed approach, on subject 12.

FIGURE 21.

Comparison of the partially occluded original images extracted from the AT&T dataset and the corresponding reconstructed images with the proposed approach, on subject 8.

FIGURE 22.

Comparison of the partially occluded original images extracted from the AT&T dataset and the corresponding reconstructed images with the proposed approach, on subject 12.

B. Classification

1) Experiment on Classification of 2-Order Tensors

This experiment was addressed to the classification of data collected from the following datasets.

• AT&T Faces dataset: as described in the experiment of Section VIII-A3.

• MNIST dataset: is consisted of 8-bit gray-scale images of digits from ”0” to “9”. There are about 6000 examples for each class [93]. Each image is centered on a $28 \times 28$ grid. In our experiments, we randomly choose 50 images for each class.

• COIL-20 dataset: contains 20 objects [94]. Each object has 72 images. The size of each image is $32 \times 32$ pixels, with 256 gray levels per pixel. We use the first 32 images for each object in our experiments.

It is worth to notice that for all such datasets the number of images in each class is far less the dimension of the corresponding basis. The main consequence of this mismatch is a large error in the estimation of intrinsic dimension of data, so that all the methods for ID estimation fails. However, although the intrinsic dimension of data cannot be determined with a certain degree of reliability, the model given by (48)–​(65) continuous to be valid with an uncertainty on the dimension $d$ . As a consequence, in all the experiments the dimension $d$ was empirically determined, to obtain a good modeling of data, instead of using the method for ID estimation.

We perform semisupervised learning on different datasets, by training the classifier on the labeled data (20% of dataset) and use the rest as unlabeled data (80% of dataset). In particular 20% of data points for each class were randomly selected as labeled data, and the rest was used as unlabeled data. The classifier was trained on the labeled data and the class labels were predicted on the unlabeled data.

We performed classification using k-nearest neighbor (kNN) algorithm [28], and compared the results obtained with features extracted by our model (matrix $B''$ ) and several other tensor learning methods, including PCA-based methods (PCA [27], gLPCA [66]), tensor decomposition methods (TD [37], GLTD [61]). Although other valuable methods for classification exist, we choose to use kNN algorithm since it is common in the literature of manifold learning [60], [62], thus making comparison with other tensor learning techniques easier.

Table 8 reports the results for the classification experiment as achieved by the methods used in [61] and PTE method. Also in this case the intrinsic dimension $d$ was empirically determined. As you can see, our method outperforms all the other methods.

TABLE 8 Comparison of Classification Results of 2-Order Tensors Datasets Using kNN Classification Algorithm (Train Set = 20%, Test Set = 80%). The Best Results are Marked by Bold Font

2) Experiment on Classification of 3-Order Tensors

In order to compare the proposed method with some other tensor-based learning methods, a last experiment was performed on the following datasets.

• Weizmann Action Database: an high-order dataset commonly used for human action recognition. The database includes 90 videos coming from 10 categories of actions - a) bending (bend), b) jacking (jack), c) jumping (jump), d) jumping in places (pjump), e) running (run), f) galloping sideways (side), g) skipping (skip), h) walking (walk), i) single-hand waving (wave1), g) both-hands waving (wave2) - which were performed by nine subjects [95], [96]. A tensor samples of size $32 \times 24 \times 10$ , represented in a spatio-temporal domain, is formed by 10 successive frames of each action, each of which was normalized to the size $32 \times 24$ pixels.

• Cambridge Hand Gesture Database: consists of 900 image sequences of nine gesture classes, which are defined by three primitive hand shapes and three primitive motions [97], [98]. Each class contains 100 image sequences (5 different illuminations $\times10$ arbitrary motions $\times {2}$ subjects). The procedure to format data is the same as in the Weizmann action database.

In these experiments, we randomly selected six action tensors of each category for training and the remaining tensors were used for testing. The experiments were independently performed 10 times using the kNN algorithm with Euclidean distance for classification, following the procedure in [58].

The methods used to evaluate the proposed PTE are: the baseline method (nearest classifier on the original data), multilinear PCA (MPCA) [31], tensor locality preserving projection (TLPP) and tensor neighborhood preserving embedded (TNPE) [67], orthogonal tensor neighborhood preserving embedded (OTNPE) [68], sparse tensor alignment (STA) [58].

Table 9 reports the results for the classification experiment as achieved by the aforementioned methods and PTE method. Also in this case the intrinsic dimension $d$ was empirically determined. As you can see the proposed PTE outperforms the other methods in terms of recognition rate.

TABLE 9 Comparison of Classification Results of 3-Order Tensors Datasets Using kNN Classification Algorithm. The Best Results are Marked by Bold Font

SECTION IX. Conclusion

In this article a nonlinear, explicit model of tensor data that depends on a reduced set of latent variables is derived. The main steps required for the estimation of the model from data are:

1. compute a basis by a Gram-Schmidt procedure;

2. reorder the basis in such a way the variances of coefficients are in decreasing order;

3. estimate the intrinsic dimension $d$ of data;

4. define a data parametrization of dimension $d$ ;

5. approximate the nonlinearity in the parametrization by a regression model.

The capability of the proposed approach for data reconstruction has been validated by performing several experiments on datasets with tensors of different orders. In these experiments several methods for regression, i.e. SVM with different kernels and NPKR method, have been adopted. In all cases the proposed tensor learning approach gives good performance for data reconstruction, nevertheless the PTE method is able to reconstruct data with minimum error. Additionally the proposed tensor learning approach has proven to be effective for classification problem, using data of reduced dimensionality. To show this ability, classification on several different datasets has been performed. The comparison of the results obtained with feature extracted by the proposed approach and state-of-the-art tensor learning methods (PCA, gLPCA, TD, GLTD, MPCA, TLPP, TNPE, OTNPE, STA), shows the effectiveness of the suggested model.

Appendix Local Parametrization of Data Embedded in a Manifold

Assume all the values of $\alpha$ embedded in the manifold $\mathcal {M}$ are described by the following parametrized surface in $\mathbb {R}^{L}$

$$\begin{equation*} \boldsymbol {\alpha } = F(\boldsymbol {\theta }), \;\; \boldsymbol {\theta } \in U \subset \mathbb {R}^{d}, \, \boldsymbol {\alpha } \in \mathbb {R}^{L}, \, d < L.\tag{71}\end{equation*}$$ This means that a bijective and differentiable function $f(\boldsymbol {x})$ defined on a subset $V = U \times \mathbb {R}^{m} \subset \mathbb {R}^{L}$ , given by $$\begin{align*} f(\boldsymbol {x}) = f(\boldsymbol {\theta }, \boldsymbol {t}) = F(\boldsymbol {\theta }) + \left ({\begin{array}{c} 0 \\ \boldsymbol {t} \\ \end{array} }\right), \; f: U \times \mathbb {R}^{m} \rightarrow \mathbb {R}^{L} \\\tag{72}\end{align*}$$ exists, where $\boldsymbol {x} = (\boldsymbol {\theta }, \boldsymbol {t})^{T} \in V,\,\,\; \boldsymbol {t} \in \mathbb {R}^{m}, \, m = L - d$ . This ensures that a one-to-one correspondence is established between a point $\boldsymbol {x} = (\boldsymbol {\theta }, 0)^{T}$ in $V$ and a point $\boldsymbol {\alpha }$ in the manifold $\mathcal {M}$ . As a consequence $f$ is invertible and unique. Rearranging (72) according to the dimension $d$ of $\boldsymbol {\theta }$ we have $$\begin{align*} f(\boldsymbol {x}) = \left ({\begin{array}{c} F''(\boldsymbol {\theta }) \\ F'(\boldsymbol {\theta }) + \boldsymbol {t} \end{array} }\right), \; F'' \in \mathbb {R}^{d}\tag{73}\end{align*}$$ and from differentiability of $f$ one gets $$\begin{align*} J(f) = \left ({J_{ \boldsymbol {\theta }}(f), J_{ \boldsymbol {t}}(f) }\right) = \left ({\begin{array}{cc} J(F'') &\quad 0 \\ J(F') &\quad I_{mm} \\ \end{array} }\right)\tag{74}\end{align*}$$ where $I_{mm}$ is an $(m \times m)$ diagonal identity matrix and $J(f)$ is the Jacobian of $f$ . In order that $f(\boldsymbol {x})$ to be invertible, as it is a bijective mapping, the condition $\det J(f) \neq 0$ on Jacobian must be satisfied. As a consequence from (74) we have $\det J(F'') \neq 0$ , meaning that the function $F''(\boldsymbol {\theta }) = \boldsymbol {\alpha ''}$ is invertible. Thus the inverse $F^{''-1}$ of $F''$ exists such that $$\begin{equation*} \boldsymbol {\theta } = F^{''-1} (\boldsymbol {\alpha ''}).\tag{75}\end{equation*}$$ Combining (71) and (75) it results $$\begin{align*} \boldsymbol {\alpha } = \left ({\begin{array}{c} \boldsymbol {\alpha ''} \\ F'\left ({F^{''-1} (\boldsymbol {\alpha ''}) }\right) \end{array} }\right) = \left ({\begin{array}{c} \boldsymbol {\alpha ''} \\ G(\boldsymbol {\alpha ''}) \end{array} }\right)\tag{76}\end{align*}$$ where $G(\cdot) = F'\left ({F^{''-1} (\cdot) }\right)$ , and this proves (30).