numpy integration ?

Discussion:

numpy integration ?

Ondrej Pacovsky

2008-09-23 16:42:15 UTC

Hello again,

I was thinking it would be a great idea to integrate fann with numpy. If
you have a large numpy array of training examples already loaded, it
would be great to be able to pass to fann for training without having to
convert it to the fann dataset format (and thus use twice as much memory).

Anyone worked with fann + numpy ? (Asking before I start thinking about
implementing that myself ... :) )

-- Ondrej

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

Ondrej Pacovsky

2008-09-24 07:03:12 UTC

Permalink

Gaziz,

sounds like a good start. So you convert the data-files to the fann
format and then load 'em with "native" fann methods ? That what I used
in the previous project ...

Did you try fiddle with the C/SWIG part, so that you could e.g. pass a
python 2D array directly to fann without having to have the training set
twice in RAM ?

-- Ondrej

Hi Ondrej,
I am using python binding and it is pain to convert data from DB into
files and feed them to fann :(
I didn't work with numpy yet but I hope I can help a bit.
Gaziz
Hello again,
I was thinking it would be a great idea to integrate fann with numpy. If
you have a large numpy array of training examples already loaded, it
would be great to be able to pass to fann for training without having to
convert it to the fann dataset format (and thus use twice as much memory).
Anyone worked with fann + numpy ? (Asking before I start thinking about
implementing that myself ... :) )
-- Ondrej
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
<http://moblin-contest.org/redirect.php?banner_id=100&url=/>
_______________________________________________
Fann-general mailing list
https://lists.sourceforge.net/lists/listinfo/fann-general

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

Ondrej Pacovsky

2008-09-24 10:17:41 UTC

Permalink

Update:

I had a quick look at the fann structs and numpy C interfacing.

Looks like it would be quite easy to create a fann function like
fann_train_from_mem(unsigned data, unsigned input, unsigned output,
fann_type *in, fann_type *out), that would create the appropriate
fann_train_data struct:

struct fann_train_data
{
enum fann_errno_enum errno_f;
FILE *error_log;
char *errstr;

unsigned int num_data;
unsigned int num_input;
unsigned int num_output;
fann_type **input;
fann_type **output;
};

(BTW: the way fann handles 2D arrays seems a bit inefficient, given that
in many cases, fann_type *'s size is comparable to the size of one data
vector itself. At least the actual data are allocated in one continuous
block of memory)

Then one could use ctypes (http://www.scipy.org/Cookbook/Ctypes) to pass
contiguous numpy array to fann. Or maybe use swig for that. The only
issue is ownership of the memory buffer: the newly created
fann_train_data must perish before the numpy array is gc'ed, otherwise
you're in trouble.

I've also been looking at ffnet, which seems to implement about the same
training algos as fann, plus GA. Dunno how fast it is, I think I will
give it a try.

-- Ondrej