Overfitting...

[Dr. Brian J. Dowd]

Is there a "rule-of-thumb" to tell when a problem is being "over-fit" or
over trained?

I have a training set of 6044 items and a test set of 1511 items (20%).
Each set has 37 independent variables and 1 output variable. MSE's are:

10 Neurons trains with 0.061920 Training set, 0.079540 Test set
20 Neurons trains with 0.051659 Training set, 0.084523 Test set
50 Neurons trains with 0.26156 Training set, 0.109737 Test set
250 Neurons trains with 0.009798 Training set, 0.146125 Test set

With how many neurons should this training set be "learned"?
Comments? Algorithms?

-Brian Dowd



-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

[Conor Robinson]

You could try a grid search. Use 4 fold or so cross validation
(create 4 separate test sets 1/4 holdout, each randomly sampled from
the original set). I would run for each neuron arch 3 to 50, 4 times
(one for each fold) times # of epoch steps, example: 100 - 3000epochs
in 50 or 100 epoch steps. To get a better idea for max epochs, run a
net with 50 neurons on a test set a few times and see were the mse
bottoms out. If you have multiple cpus you can run all four folds at
once for a given hidden and epochs.

Find the lowest ave test error for a given hidden-epochs across the four folds.

You can also use a GA to search for input selection and hidden, but I
would use RBF nets (faster) to take out any epoch guessing, since
you'll probably only need ~100epochs to optimize the last layer.

Realistically speaking, it looks like you need 20 or less hidden
neurons, you are overfitting somewhere below 20 with your current
number of epochs since you test error went up after 10 hidden. I
would test a few manually in that range, unless you want to get crazy.
Remember that nets with less neurons have to generalize more and may
be more robust classifiers, at 250 you are memorizing the entire
training set.

ftp://ftp.sas.com/pub/neural/FAQ.html#questions

-Conor

Post by Dr. Brian J. Dowd
Is there a "rule-of-thumb" to tell when a problem is being "over-fit" or
over trained?
I have a training set of 6044 items and a test set of 1511 items (20%).
10 Neurons trains with 0.061920 Training set, 0.079540 Test set
20 Neurons trains with 0.051659 Training set, 0.084523 Test set
50 Neurons trains with 0.26156 Training set, 0.109737 Test set
250 Neurons trains with 0.009798 Training set, 0.146125 Test set
With how many neurons should this training set be "learned"?
Comments? Algorithms?
-Brian Dowd
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Fann-general mailing list
https://lists.sourceforge.net/lists/listinfo/fann-general

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

[Adrian Spilca]

Post by Dr. Brian J. Dowd
Is there a "rule-of-thumb" to tell when a problem is being "over-fit" or
over trained?
I have a training set of 6044 items and a test set of 1511 items (20%).
10 Neurons trains with 0.061920 Training set, 0.079540 Test set
20 Neurons trains with 0.051659 Training set, 0.084523 Test set
50 Neurons trains with 0.26156 Training set, 0.109737 Test set
250 Neurons trains with 0.009798 Training set, 0.146125 Test set
With how many neurons should this training set be "learned"?
Comments? Algorithms?
-Brian Dowd

Yes, I would say that what you experience here _is_ overfitting. Apart
from the 50 hidden neurons experiment that it's probably stuck in a
local minimum, you can see clearly that the training error goes deeper
and deeper without improvement in testing. The network has too much
"power" and will learn your training set "by-heart", as a set of
numbers, rather than the rule behind it. It is obvious with 250 hidden
neurons that the network can learn very well the samples in the training
set, but performs pretty badly on samples that follow the same rule, but
are slightly different (or in fact, are not "exactly" the ones in the
training set).

I would try even less than 10 hidden units, or explore the space between
10 and 20 to see perhaps the test error gets lower.

And then, if you can get more training samples, you can increase the
number of hidden neurons, with better performance on test set.

Hope this helps,
Adrian



-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/