Pre-processing data for neural networks
Summary from last week:
; We outlined the development cycle for a neural network application ; We discussed the importance of data selection
; To investigate different types of neural network data preprocessing. Objectives: you should be able to:
; Demonstrate an understanding of the key principles involved in neural
network application development.
; Demonstrate an understanding of the data selection and pre-processing
techniques used in constructing neural network applications.
Neural networks learn more quickly and give better performance if the input variables are pre-processed before being used to train the network. Bear in mind that exactly the same pre-processing should be done to the test set, if we are to avoid peculiar answers from the network.
Scaling the data:
One of the reasons for scaling the data is to equalise the importance of variables. For example, if one input variable ranges between 1 and 10000, and another ranges between 0.1 and 0.001, the network should be able to learn to use tiny input weights for the first variable and huge weights for the second variable. However we are asking a lot of the network to cope with these different ranges. We can make the networks life a lot easier by giving it data scaled in such a way that all the weights can remain in small, predictable ranges.
To scale the data for a particular input X, find the maximum X (maxX) for that input, the minimum X (minX) and find the scaled value of any input X (scaledX) using the following equation:
scaledX = (X - minX)/(maxX - minX)
so for example, if maxX = 80, minX = 20 and we want to scale the value of X which is 50:
scaledX = (50-20)/(80-20)
A common way of dealing with data that is not normally distributed is to perform some form of mathematical transformation on the data that shifts it towards a normal distribution.
Statistical and numerical measures of trends in historical data can also be quite useful. For example a financial analyst might be interested in the pattern of cash flow for a company over the previous five years, or a marketing specialist might be interested in the trend in sales of certain product over the past six months. In situations such as these, it is often useful to extract the most salient information from the trend data and present the neural network with only a summary measure. Typically analysts are interested in three aspects of a variable’s trend:
What is the current status of the variable? This is just the most recently
available value for the variable.
How volatile is the variable over time? This can be measured using the
standard deviation of the data series. This can then be normalised by dividing it by the absolute value of the mean of the points in the series (assuming the mean is not 0). Normalisation is necessary to make comparisons across series with differing scales.
In what direction is the variable moving? & To what degree is the variable moving? The simplest way of doing this is to calculate the percentage change in the variable from the previous period:
However this only captures the most recent change in the variable and may be misleading if the underlying data series is highly volatile. Also, the percent change is only valid if both values are positive. More robust numerical values calculate the first derivative of the line through a series of data points using numerical differentiation techniques, a five-point method is shown below. The formula uses five data points to calculate the numerical derivative at the most recent point (X): 0
In practice it is best to normalise the slope, and since we are calculating the derivative at Xwe normalise by dividing by the absolute value of that point: 0
These indicators can add a great deal of ‘intelligence’ to the data being
presented to the neural network.
Another aspect of trend analysis that could affect the results is the seasonal or time-lagged aspect of some types of data. Some phenomena have a natural cyclicality associated with them. If a variable exists which has this form, it may not be appropriate to compare values if they are taken from different phases of a cycle. For example, when examining the change in quarterly sales volume in a department store, we should consider that the volume in the last quarter of the year (i.e. around Christmas) will probably be much higher than in the first quarter. To address this problem many analysts compare quarterly data on a lagged basis. Lagging data simply means comparing the current period’s data with the
previous corresponding periods in the cycle. In the example above, this would mean comparing the fourth-quarter data from this year with the fourth-quarter data from previous years, etc.
For higher frequency time series such as those found in market prediction or signal-processing problems, it may be more appropriate to use more sophisticated curve fitting techniques such as fast-fourier transforms or wavelets, both of which estimate curves by building equations of mathematical and/or trigonometric functions.
Sometimes the variables we present to neural networks are fundamentally circular. Examples are a rotating piece of machinery, or the dates in the calendar year. These variables introduce a special problem due to the fact that they have a discontinuity, i.e. we have a serious problem as the object passes from 360 degrees to 0 degrees or from 31st December to 1st January (i.e. day 365 to day 1). If we use a single input neuron to represent this value, we find that two extremely close values such as 359 degrees and 1 degree are represented by two extremely different activations, nearly full on and nearly full off. The best way to handle circular variables is to encode them using two neurons. We find new variables which change smoothly as the circular variable changes, and whose values when taken together are in a one-to one relationship with the circular variable. The most common transformation is to make sure our original variable is normalised to lie in the range 0 to 360 and then create two new variables as inputs to the network. If our original variable is X then the two new variables are:
Var 1 = sin(X)
Var 2 = cos(X)
Summary, today we have:
; Looked at data transformation, one of the key principles involved in neural
network application development.
; Described a number of pre-processing techniques used to make the learning
process easier for the network.
Practical neural network recipes in C++, by Timothy Masters, Academic Press, 1993, ISBN 0-12-479040-2