[B.A.T.M.A.N.] Memory leak

List overview All Threads
Download

newer

older

[B.A.T.M.A.N.] batman OpenWRT...

[B.A.T.M.A.N.] batmand_0.3~r875...

Freifunk Dresden

4 Jan 2008 4 Jan '08

9:57 a.m.

Hello, I have a problem where batman (rev871 and rev908) consumes more and more memory until the router has no memory. I have added the output of the "top" command and the command line arguments of my setup. (WRT 10-1).....wlan.......(wrt 10-2) =======tinc vpn=====(laptop debian 0-1) kernel 2.4.32 kernel 2.4.32 kernel 2.6 I have seen that one batman thread changes the PID where the other stay the same. Below you will find the "top" output for batmand when it was started and after about one hour. The batmand running on kernal 2.6 does not increment its memory needs. Perhaps there are some allocation of memory that is not freed because of the arguments of the batmand. There is no special traffic on those nodes, they just are connected. Any Ideas? Regards Stephan linux 2.6. (0-1) /usr/bin/batmand -g 1024/200 -a 104.61.0.0/16 -a 141.56.20.5/32 -s 10.12.0.1 --no-unreachable-rule --no-throw-rules --no-prio-rules --resist-blocked-send wifi tbb /t 1 /i /A wifi is a unused bridge 1 batmand PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6612 root 15 0 19136 684 532 S 0.3 0.1 0:27.85 batmand #################################################################### linux 2.4.32 (wrt GL 10-2) /sbin/batmand -s 10.12.0.1 -a 10.12.10.16/28 -r 2 --t 63 --no-unreachable-rule --no-throw-rules --no-prio-rules --resist-blocked-send eth1 tbb /t 1 /i /A 3 batmands PID PPID USER STAT VSZ %MEM %CPU COMMAND 847 846 root S 1196 4% 0% /sbin/batmand -s 10.12.0.1 -a 10.12.10.16/28 -r 2 --t 63 --no-unreachable-rule 3937 847 root S 1196 4% 0% /sbin/batmand -s 10.12.0.1 -a 10.12.10.16/28 -r 2 --t 63 --no-unreachable-rule 848 847 root S 1196 4% 0% /sbin/batmand -s 10.12.0.1 -a 10.12.10.16/28 -r 2 --t 63 --no-unreachable-rule ---------- 848 847 root S 2076 7% 0% /sbin/batmand -s 10.12.0.1 -a 10.12.10.16/28 -r 2 --t 63 --no-unreachable-rule 847 846 root S 2076 7% 0% /sbin/batmand -s 10.12.0.1 -a 10.12.10.16/28 -r 2 --t 63 --no-unreachable-rule 22570 847 root S 2076 7% 0% /sbin/batmand -s 10.12.0.1 -a 10.12.10.16/28 -r 2 --t 63 --no-unreachable-rule #################################################################### linux 2.4.32 (wrt GL 10-1) /sbin/batmand -s 10.12.0.1 -a 10.12.10.0/28 -r 2 --t 63 --no-unreachable-rule --no-throw-rules --no-prio-rules --resist-blocked-send eth1 4 batmands PID PPID USER STAT VSZ %MEM %CPU COMMAND 837 1 root S 1292 9% 0% /sbin/batmand -s 10.12.0.1 -a 10.12.10.0/28 -r 2 --t 63 --no-unreachable-rule - 838 837 root S 1292 9% 0% /sbin/batmand -s 10.12.0.1 -a 10.12.10.0/28 -r 2 --t 63 --no-unreachable-rule - 11854 838 root S 1292 9% 0% /sbin/batmand -s 10.12.0.1 -a 10.12.10.0/28 -r 2 --t 63 --no-unreachable-rule - 839 838 root S 1292 9% 0% /sbin/batmand -s 10.12.0.1 -a 10.12.10.0/28 -r 2 --t 63 --no-unreachable-rule - --------- 837 1 root S 2172 15% 0% /sbin/batmand -s 10.12.0.1 -a 10.12.10.0/28 -r 2 --t 63 --no-unreachable-rule - 838 837 root S 2172 15% 0% /sbin/batmand -s 10.12.0.1 -a 10.12.10.0/28 -r 2 --t 63 --no-unreachable-rule - 30544 838 root S 2172 15% 0% /sbin/batmand -s 10.12.0.1 -a 10.12.10.0/28 -r 2 --t 63 --no-unreachable-rule - 839 838 root S 2172 15% 0% /sbin/batmand -s 10.12.0.1 -a 10.12.10.0/28 -r 2 --t 63 --no-unreachable-rule -

Show replies by date

Axel Neumann

4 Jan 4 Jan

8:24 p.m.

Hi, perhaps the problem is not related to 2.4/2.6 kernels but to the thread implementation of ulibc (used by openWrt). On your notebook you are probably running glibc. Glibc correctly shows only one process and not one per running thread. The changing PID might be because of the tunnel thread which is restarted when (re-)connecting to a GW. I could not reproduce the high memory consumption but recognized that memory usage increases slightly (e.g. when opening a verbose debug-level) and never decreases again to its old value. I thought the used memory (as indicated by ps and friends) is not freed until required by another program. ciao, axel On Freitag 04 Januar 2008, Freifunk Dresden wrote:

...

Freifunk Dresden

5 Jan 5 Jan

8:47 p.m.

Hi, I have checked the problem a little more. Beside the "top" command on debian can show the same as the "top" does on WRTs. If you press 'H' the threads are also shown. Perhaps the implementation of the "top" or "ps" show the threads too. There are four batmand threads running. After a while one thread is terminated for about 100 seconds (25 x 4 seconds) and a new thread is created after this time. At this time the memory needs is increased. It seams that at each thread-kill-restart 16 Kbytes are wasted. The virtual assinged memory increases and also the %-Value used. Below you will see the values (memory and pid) delivered by "top" for one incrementation. Hope it gives you a hint. Why is the thread killed and created? There is no client conntected to any router. rdate is restarted periodically and the laptop is telling that it has an internet connection (configured through batmand arguments, which is wanted). Note that the laptop does not have a connection to internet. It just pretends it has one. MEM: 1180 8% PIDS:1 -> 893 -> 894 -> 895 -> 2673 Pid 2673 terminates MEM: 1180 8% .... 100 seconds .... MEM: 1180 8% Pid 3751 is created MEM: 1196 9% 1 -> 893 -> 894 -> 895 -> 3751 MEM: 1196 9% Bye Stephan

Axel Neumann

6 Jan 6 Jan

2:03 p.m.

Hi, On Samstag 05 Januar 2008, Freifunk Dresden wrote:

...

Does it really increase endlessly with each new thread ? If its the GW-client thread then you can enforce the termination and recreation on-the-fly with batmand -c -r0 (to destroy the tunnel) and -c -r3 (to create the tunnel)

...

Below you will see the values (memory and pid) delivered by "top" for one incrementation. Hope it gives you a hint. Why is the thread killed and created? There is no client conntected to any router. rdate is restarted periodically and the laptop is telling that it has an internet connection (configured through batmand arguments, which is wanted). Note that the laptop does not have a connection to internet. It just pretends it has one.

there is a blackhole detection in the default two-way-tunnel. This can cause the client to disconnect from the unresponsive (-blackhole-) GW and put the GW on a blacklist for some time. It should all be reported on debug-level 3. If no other GW is available, the GW-client node will reconnect to the GW after a while (might be 100 sec). If you have a dummy GW, either use --no-unresp-gw-check or use --one-way-tunnel 3 at the client and the GW side /axel

...

MEM: 1180 8% PIDS:1 -> 893 -> 894 -> 895 -> 2673 Pid 2673 terminates MEM: 1180 8% .... 100 seconds .... MEM: 1180 8% Pid 3751 is created MEM: 1196 9% 1 -> 893 -> 894 -> 895 -> 3751 MEM: 1196 9% Bye Stephan _______________________________________________ B.A.T.M.A.N mailing list B.A.T.M.A.N(a)open-mesh.net https://list.open-mesh.net/mm/listinfo/b.a.t.m.a.n

Freifunk Dresden

3:52 p.m.

Hi Axel,

...

When I call batmand with those parameters I can reproduce this much faster until no memory is available anymore. This happens on each call with these parameters.

...

I like to have the blackhole detection enabled, it is a good way to detect an inactive GW beside of the ping-check-algorithm in the firmware. I assume that default is the two-way-tunnel where data goes from one node directly to the gw and back. Whatfor is the one-way-tunnel and what are the condtion for the one-way-tunnel being active. Why is the gw-thread deleted? If the problem lays in the pthread implementation it would be a workaround to keep the gw thread always active. I also think that the memory consumtion is more than 16kbyte, because the virtual memory is incremented and also the percentage used. Bye Stephan

Axel Neumann

7 Jan 7 Jan

9:45 a.m.

hi, On Sonntag 06 Januar 2008, Freifunk Dresden wrote:

...

Hi Axel,

When I call batmand with those parameters I can reproduce this much faster until no memory is available anymore. This happens on each call with these parameters.

ok, I'll check that again. Just to be sure, you observed this on WRT54GL with openWrt Whiterussioin Freifunk firmware? Did you compile the sources yourself or have you used precompiled binaries (which ones)? Because, last time I tried, I applied a lot of -cr0 and -cr3 and the memory usage did not increase endlessly.

...

one-way-tunnel are completely stateless. No assigned virtual IPs, no need for SNAT,... Data is only tunnelled from the GW-Client to the GW-Node. THere it is de-tunnelled and forwarded. We used this setup at the 24c3 to let batman client nodes be accessible via global unique IPs.

...

what are the condtion for the one-way-tunnel being active.

if you type -cbd2 at a client node you may see: WARNING: You are using BatMan-eXp 0.3-alpha rv898 (compatibility version 10) ! Originator bestNextHop # => 103.131.41.5 103.131.83.5 100, gw_class 49 - 4MBit/1024KBit, reliability: 0, supported tunnel types 2WT, 1WT 2WT indicates the GW capabilities for two-way-tunnel, 1WT indicates ... --dangerous shows: ... Gateway and tunneling options: --one-way-tunnel <value> : set preference for one-way-tunnel mode. For GW-nodes: 0 disables this tunnel mode, a larger value enables this tunnel mode. For GW-cliets: 0 disables this tunnel mode, a larger value sets the preference for this mode. default: 1, allowed values: 0 <= value <= 4 --two-way-tunnel <value> : set preference for two-way-tunnel mode. For GW-nodes: 0 disables this tunnel mode, a larger value enables this tunnel mode. For GW-cliets: 0 disables this tunnel mode, a larger value sets the preference for this mode. default: 2, allowed values: 0 <= value <= 4 So essentially, to force a client node to only use one-way tunnel you must start the client with two-way-tunnel disabled (--two-way-tunnel 0 ) . To configure the client node to prefer one-way-tunnel over two-way-tunnel start it with a higher preference for the one-way tunnel (e.g. --one-way-tunnel 3) bye /axel

...

Why is the gw-thread deleted? If the problem lays in the pthread implementation it would be a workaround to keep the gw thread always active. I also think that the memory consumtion is more than 16kbyte, because the virtual memory is incremented and also the percentage used. Bye Stephan _______________________________________________ B.A.T.M.A.N mailing list B.A.T.M.A.N(a)open-mesh.net https://list.open-mesh.net/mm/listinfo/b.a.t.m.a.n

Freifunk Dresden

12:56 p.m.

Hi, The batmand is running on WRT54GL and WRT54GS. The firmware is whiterussian_rc6 and the batmand was self-compiled from within whiterussian devel env. I have removed the "-static" LDFLAG and also removed the DEBUG/MEMORY... CFLAGS. As optimation I have used -Os to keep the batmand small which also produces a lot of warnings. I'm not at home to give you the correct CFLAGS.

...

I'm still confused about the data traveling over the one/two way tunnels and what the differences are for data travel (request/answer). What IP does the tunnel interface have for the one-way tunnel? Same as the main interface? If a client creates a tcp connection then ACKs and answers should come back through this one-way-tunnel, doesn't it? What dis/advantages does a two-way-tunnel have against the one-way-tunnel? For the one-way-tunnel how many tunnel interfaces are created? For each connecting client one interface or only one interface for all? How does the GW route packets to a specific client if the GW has a route over the wlan interface and a route to the tunnel interface? Please let me know if there is already a documentation for this. Currently I'm using the two-way-tunnel and this works find. the only "bad" thing is that the proxy running on the GW does display the 169er IP addresses and not the node main ip. But this is ok. Perhaps the one-way-tunnel would be better because the memory consumption on the GW. If you have some hints on that, please let me know. /Stephan

Axel Neumann

4:48 p.m.

New subject: [B.A.T.M.A.N.] one-way / two-way tunnel (was: Memory leak)

On Montag 07 Januar 2008, Freifunk Dresden wrote:

...

if you have 30k left, try -O1. Then there should be no warnings. playing with different optimization levels (all dynamically linked and stripped) : 194783 b with -O1 -DDEBUG_MALLOC -DMEMORY_USAGE -DPROFILE_DATA 190675 b with -O1 162019 b with -O2 174247 b with -O3 I once did some cpu-load experiments to estimate the benifits of -O1..-O3 and could not identify any significant differences between the different compiler optimizations. By the way, at the 24c3 somebody managed to stip more useless data from the binaries than I achieved with the whiterussion sstrip. Does anybody know how this is possible and what's the difference between strip and sstrip ?

...

I'm not at home to give you the correct CFLAGS.

I'm still confused about the data traveling over the one/two way tunnels and what the differences are for data travel (request/answer).

an original packet, generated at the client node and forwarded into the bat0 tunnel will be wrapped as UDP payload and send to the GW node. IP src and dst address of the inner (original packet) are not changed. The inner (original) packet does usually contain: - Any global unique IP address from a distant server - the GW-Client node primary batman interface as the src address. The outer IP header will contain: - the GW-Nodes' primary batman interface as the dst address and - the GW-Client nodes' primary batman interface as the src address. Once arrived at the GW-node the inner (original) packet is detunnelled ( unwrapped ) and forwarded to the destination address of the original packet. Once a reply/answer to the original packet arrives at the GW node it'll contain the GW-Client node primary batman interface as the dst address and is forwarded. Thats it.

...

What IP does the tunnel interface have for the one-way tunnel? Same as the main interface?

- on the GW-Client node: Yes, same as the primary batman interface - on the GW-Node, depending of what you configured as the GW-tunnel range ( --dangerous: ) --gw-tunnel-network <ip-address/netmask> : set tunnel IP-address range leased out by GW nodes. Only relevant for GW-nodes in two-way-tunnel mode default: 169.254.0.0/22, allowed netmask values: 20 <= value <= 30 If you only want to use one-way-tunnel and do not want to see an ip address here you can use --gw-tunnel-network 0.0.0.0/30 .

...

If a client creates a tcp connection then ACKs and answers should come back through this one-way-tunnel, doesn't it?

no, answers are not tunnelled, see above.

...

What dis/advantages does a two-way-tunnel have against the one-way-tunnel?

more cpu load (every downlink packet must be entunnelled and usually you have much more downlink traffic than uplink), less protocol overhead (no additional IP/UDP header), less need for SNAT. See also this thread: https://list.open-mesh.net/pipermail/b.a.t.m.a.n/2007-November/000407.html

...

For the one-way-tunnel how many tunnel interfaces are created? For each connecting client one interface or only one interface for all?

Always only one, no matter how many clients, secondary interfaces, one-way or two-way tunnel

...

How does the GW route packets to a specific client if the GW has a route over the wlan interface and a route to the tunnel interface?

If you are talking about a batman GW-node which is participating in the mesh then it should have a host, HNA, or IF route to each other batman node. This will be used for forwarding the packet towards the client node. If you are talking about a non-batman GW, you must set a network route explicitly to the batman GW-node.

...

Please let me know if there is already a documentation for this.

in progress... (i see, this answer gets boring)

...

Currently I'm using the two-way-tunnel and this works find. the only "bad" thing is that the proxy running on the GW does display the 169er IP addresses and not the node main ip. But this is ok. Perhaps the one-way-tunnel would be better because the memory consumption on the GW.

Memory consumption should usually not be an issue.

...

If you have some hints on that, please let me know. /Stephan _______________________________________________ B.A.T.M.A.N mailing list B.A.T.M.A.N(a)open-mesh.net https://list.open-mesh.net/mm/listinfo/b.a.t.m.a.n

Freifunk Dresden

8:52 p.m.

New subject: [B.A.T.M.A.N.] one-way / two-way tunnel (was: Memory leak)

Hi Axel,

...

On Montag 07 Januar 2008, Freifunk Dresden wrote:

If you use option -Os the resulting size is 153815 bytes. I like to save flash memory as much as possible. But I will try if the optimation level does solve the problem. The CFLAGS I have used are -Wall -Os.

...

I once did some cpu-load experiments to estimate the benifits of -O1..-O3 and could not identify any significant differences between the different compiler optimizations.

If this is true then I tend to keep -Os as compile option.

...

By the way, at the 24c3 somebody managed to stip more useless data from the binaries than I achieved with the whiterussion sstrip. Does anybody know how this is possible and what's the difference between strip and sstrip ?

I don't know, but it would be a good Idea if I could configure the batman make process. For instance I disabled the routing rule settings of batman. Perhaps there are more thinks that can be disabled if they are not used. Thanks for the explaination about the GW/tunnel. Will need a deeper look into your description. If I still have questions, I will ask. :) /Stephan

Freifunk Dresden

10 Jan 10 Jan

8:25 a.m.

New subject: [B.A.T.M.A.N.] one-way / two-way tunnel (was: Memory leak)

Hi, I have compiled batmand (whiterussian_rc6) with the CFLAGS you provide. There is no change in the behavior when calling "batmand -c -r0" and "batmand -c -r3". On every thread creation the memory consumption rises. I also have tried to apply "--one-way-tunnel 1 --two-way-tunnel 0" Using the commands above I have the same result. One advantage of the one-way-tunnel is, that the gateway thread is not killed and re-created. /Stephan

Freifunk Dresden

17 Jan 17 Jan

1:04 p.m.

Hi, has anyone could verify the memory-leak problem (see previous thread). I'm currently use the -one-way-tunnel which does not destroy/recreate the gw thread that is causing the memory consumtion. jfi: the system is whiterussian rc6, busybox 1.7.2/1.8.2, uclibc 0.27. I still couldn't get uclibc 29 running to check if this causes the problem. Can somebody check this or first verify the problem? /Stephan

Axel Neumann

20 Jan 20 Jan

11:13 a.m.

Hi, can you check if the problem is fixed with rv 957. regards, axel On Donnerstag 17 Januar 2008, Freifunk Dresden wrote:

...

Freifunk Dresden

26 Jan 26 Jan

9:04 p.m.

Hi, I have checked the revision 966 and the memory leak is gone. calling batmand -c -r3 creates a task and the memory size is set to 1164. After calling batmand -c -r0 the size is back to 1148. I can repeat these calles and all looks fine. /Stephan Zitat von Axel Neumann <axel(a)open-mesh.net>et>:

...

Hi, can you check if the problem is fixed with rv 957. regards, axel On Donnerstag 17 Januar 2008, Freifunk Dresden wrote:

_______________________________________________ B.A.T.M.A.N mailing list B.A.T.M.A.N(a)open-mesh.net https://list.open-mesh.net/mm/listinfo/b.a.t.m.a.n

5542

days inactive

5564

days old

b.a.t.m.a.n@lists.open-mesh.org

Manage subscription

12 comments

2 participants

tags (0)

participants (2)

Axel Neumann
Freifunk Dresden