The basic idea is you train once on all your data, and then test on your
validation set. Then you monitor whichever performance metric you care
about. MSE, accuracy, cross-entropy, precision, recall, preferably cost if
you can.
If fann doesn't support your performance metric, then you need to either
write a test call back, or your own test_on_data function.
Then, you stop training when your performance stops improving. How you
define that is also up to you. I do something like, if there has been no
improvement in 100 epochs, I randomly choose from the best nets I've seen to
tie-break.
I've wanted to try something like a sign test to measure statistical
significance between epochs, but then I got into Bayesian methods where you
don't need a validatoin set.
--Josh
Post by Chris SpencerBut how do you do it? Do you just call train_on_data until test_data
gives a small enough MSE for your validation set?
Regards,
Chris
Post by Josh MenkeIt's not built in to fann, but you can do it yourself. With ANNs I don't
usually use cross-validation, but instead just a single validation set.
This
Post by Josh Menkeis mostly due to the size of the sets I use though (millions).
--Josh
Post by Chris SpencerIs there any way to perform cross-validation during training with
either the backprop or cascade correlation methods? I don't see any
explicit methods for this in the docs. I see the short paragraph
"Avoid Over-Fitting" on the advanced usage page, but it doesn't
actually describe how to do this.
Regards,
Chris
-------------------------------------------------------------------------
Post by Josh MenkePost by Chris SpencerTake Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to
share
Post by Josh Menkeyour
Post by Chris Spenceropinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
Post by Josh MenkePost by Chris Spencer_______________________________________________
Fann-general mailing list
https://lists.sourceforge.net/lists/listinfo/fann-general
--
Joshua Menke
Machine Learning Scientist
Trust and Safety Applied Research
ebay, Inc
-------------------------------------------------------------------------
Post by Josh MenkeTake Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
Post by Josh Menkeopinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
Post by Josh Menke_______________________________________________
Fann-general mailing list
https://lists.sourceforge.net/lists/listinfo/fann-general
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Fann-general mailing list
https://lists.sourceforge.net/lists/listinfo/fann-general
--
Joshua Menke
Machine Learning Scientist
Trust and Safety Applied Research
ebay, Inc
josh-***@public.gmane.org