Numerical conditioning of neural network components in time series forecasting

Abstract

In this paper, an issue of stability of fully connected neural network (perceptron), forecasting time series with sliding window method, is considered. A task of finding conditions, providing best results on accuracy and stability, was set. To accomplish it, training of the neural network with various sets of hyperparameters, such as amount of neurons on input and hidden layers, as well as activation function, has been performed. Training was performed on modeled data, which have been perturbed with unbiased normally distributed random variables with various variance. Such training results as number of conditioning of weight matrix between input and hidden layers, number of conditioning of activation functions, as well as statistics on retrospective forecasting absolut error, have been shown. An approach for estimation of condition number for some activation functions has been proposed. Impact of activation function on accuracy and stability of forecasting perceptron has been shown.