Haven't tried that myself yet, but maybe the new network namespace support in the git master branch might be an easy alternative to start with?
You could first try to compile and load the batman-adv kernel module. Then you'd create a namespace for each virtual node you want to have. Then you connect the namespace via veth-pairs. And finally, within each network namespace you would hook some veth ends into a batX interface.
First you could try wiring two, then three nodes manually. If that works and looks good you could then write a script to setup the testbed with an arbitrary amount of virtual nodes.
Regards, Linus