On tensor-product model based representation of neural networks

The approximation methods of mathematics are widely used in theory and practice for several problems. In the framework of the paper a novel tensor-product based approach for representation of neural networks (NNs) is proposed. The NNs in this case stand for local models based on which a more complex parameter varying model can numerically be reconstructed and reduced using the higher order singular value decomposition (HOSVD). The HOSVD as well as the tensor-product based representation of NNs will be discussed in detail.

I. Introduction

In non-linear system modelling beside analytic techniques heuristic approaches are highly welcome, as well, especially in case when black-box like systems have to be modelled. Approaches connecting these two main concepts, i.e. analytic and heuristics ones, may further improve their effectiveness and further extend their appli-cability. Many methods have been proposed to deal with multi-input, multi-output systems, by the literature. Linear parameter varying (LPV) structure is one by which nonlinear systems can be controlled on the basis of linear control theories. As another frequently used approach to approximate dynamic systems the Takagi-Sugeno fuzzy modelling can be mentioned. This interest relies on the fact that dynamic T-S models are easily obtained by linearization of the nonlinear plant around different operating points [8]. Furthermore, neural networks are another examples of frequently used tools in non-linear modeling, which have the ability to learn sophisticated non-linear relationships [9]. Tensor product (TP) transformation is a numerical approach, which makes a connection between linear parameter varying models and higher order tensors ([5], [4]). The approach is strongly related to the generalized SVD the so called higher order singular value decomposition (HOSVD). Singular value decomposition is one of the most powerful tools of linear algebra and it has a great variety of applications. In the last two decades the increasing computing power made it possible to work with large multidimensional arrays (tensors). Some properties of tensors are the same as of matrices in the “usual” linear algebra, but some of them is a bit trickier (see for example [10], [11]). With the help of the HOSVD the LPV systems can effectively be approximated by local linear time invariant (LTI) models. Furthermore, models expressible in tensor product form can be reduced efficiently, as well [7]. The HOSVD is highly welcome also in other fields, as for instance the image processing, where task similar to those performed in the frequency domain can be performed efficiently, e.g. image compression, resolution enhancement, noise elimination, etc. [2].