Hi, i'm participating in the development of a wireless mesh network in Delta de Tigre, near Buenos Aires, Argentina. Landscape is absolutely flat, with dense forest 15-20mts tall. Nodes are being installed along the river, 50 - 150mts apart, with clear LOS at ground level. I collaborated with Nico Echaniz last month, bulding another community network in Cordoba, and given the success, I decided to start one more here. I'm also really thankful with Elektra, for recommending the tl-m3220 to Nico! We chose batman over olsr, in part because of avahi stuff, but also to have a real-world implementation and thus collaborate in testing / debugging :) So far, there are only 4 working nodes, but luckily I'm already facing unexpected behaviour.
I'm sorry for the long email, it's mostly terminal logs. In short, iperf between two neighbouring nodes reports >20mbps, yet running the same iperf between nodes separated by one hop reports inexplicably low <5mbps.
Links are point to point, with different radios on different channels, no freq overlap.
A=[ath9k]--(50m)--[ath9k]=B=[ath9k_htc]--(150m)--[ath9k_htc]=C=[ath9k]--(50m)--[ath9k]=D
A is a tplink mr3420 + wn722n B and C are tplink mr3220 + wn722n D is a tplink mr3220 (single interface)
All of them are running openwrt trunk r30919. the internal iface of the mr3x20 uses ath9k, and the wn722n runs with ath9k_htc
the link between B and C is done in infrastructure mode, B is ap and C is managed. Radios are set to channel 9, HT40-. C <-> D is made with ath9k , in channel 1 , HT20.
(all hostnames have been obscured for readability :P )
nodeB -> nodeC = 31mbps nodeC -> nodeD = 21mbps nodeB -> nodeD = 3mbps
nodeD -> nodeC = 21mbps nodeC -> nodeB = 21mbps nodeD -> nodeB = 10mbps
nodeD# batctl tr B_mesh traceroute to B_mesh (56:e6:fc:be:29:d3), 50 hops max, 20 byte packets 1: C_mesh (56:e6:fc:b9:b6:47) 1.155 ms 0.834 ms 0.796 ms 2: B_mesh (56:e6:fc:be:29:d3) 6.523 ms 2.719 ms 1.917 ms
nodeB# batctl tr D_mesh traceroute to D_mesh (56:e6:fc:b9:b7:01), 50 hops max, 20 byte packets 1: C_mesh (56:e6:fc:b9:b6:47) 1.323 ms 1.117 ms 0.974 ms 2: D_mesh (56:e6:fc:b9:b7:01) 2.278 ms 1.839 ms 2.164 ms
### iperf between nodeB -> nodeC root@nodeB:~# iperf -c nodeC -w 320k -i 1 ------------------------------------------------------------ Client connecting to nodeC, TCP port 5001 TCP window size: 320 KByte ------------------------------------------------------------ [ 3] local 10.6.0.1 port 35759 connected with 10.6.0.32 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 3.63 MBytes 30.4 Mbits/sec [ 3] 1.0- 2.0 sec 3.63 MBytes 30.4 Mbits/sec [ 3] 2.0- 3.0 sec 4.00 MBytes 33.6 Mbits/sec [ 3] 3.0- 4.0 sec 3.63 MBytes 30.4 Mbits/sec [ 3] 4.0- 5.0 sec 3.88 MBytes 32.5 Mbits/sec [ 3] 5.0- 6.0 sec 3.88 MBytes 32.5 Mbits/sec [ 3] 6.0- 7.0 sec 4.00 MBytes 33.6 Mbits/sec [ 3] 7.0- 8.0 sec 3.75 MBytes 31.5 Mbits/sec [ 3] 8.0- 9.0 sec 3.75 MBytes 31.5 Mbits/sec [ 3] 9.0-10.0 sec 3.75 MBytes 31.5 Mbits/sec [ 3] 0.0-10.1 sec 38.0 MBytes 31.7 Mbits/sec
### iperf between nodeC -> nodeD nodeC# iperf -c nodeD -w 320k -i 1 ------------------------------------------------------------ Client connecting to nodeD, TCP port 5001 TCP window size: 320 KByte ------------------------------------------------------------ [ 3] local 10.6.0.32 port 43655 connected with 10.6.0.16 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 2.50 MBytes 21.0 Mbits/sec [ 3] 1.0- 2.0 sec 2.63 MBytes 22.0 Mbits/sec [ 3] 2.0- 3.0 sec 2.50 MBytes 21.0 Mbits/sec [ 3] 3.0- 4.0 sec 2.88 MBytes 24.1 Mbits/sec [ 3] 4.0- 5.0 sec 2.63 MBytes 22.0 Mbits/sec [ 3] 5.0- 6.0 sec 2.63 MBytes 22.0 Mbits/sec [ 3] 6.0- 7.0 sec 2.50 MBytes 21.0 Mbits/sec [ 3] 7.0- 8.0 sec 2.25 MBytes 18.9 Mbits/sec [ 3] 8.0- 9.0 sec 2.63 MBytes 22.0 Mbits/sec [ 3] 9.0-10.0 sec 2.63 MBytes 22.0 Mbits/sec [ 3] 0.0-10.1 sec 25.9 MBytes 21.5 Mbits/sec
### Now, enter the mistery.. ### Best iperf run after several attemps yielding 1 - 2 mbps # iperf -c charly-muelle -w 320k -i 1 -t 20 ------------------------------------------------------------ Client connecting to charly-muelle, TCP port 5001 TCP window size: 320 KByte ------------------------------------------------------------ [ 3] local 10.6.0.1 port 55093 connected with 10.6.0.16 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 896 KBytes 7.34 Mbits/sec [ 3] 1.0- 2.0 sec 256 KBytes 2.10 Mbits/sec [ 3] 2.0- 3.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 3.0- 4.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 4.0- 5.0 sec 256 KBytes 2.10 Mbits/sec [ 3] 5.0- 6.0 sec 384 KBytes 3.15 Mbits/sec [ 3] 6.0- 7.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 7.0- 8.0 sec 384 KBytes 3.15 Mbits/sec [ 3] 8.0- 9.0 sec 256 KBytes 2.10 Mbits/sec [ 3] 9.0-10.0 sec 512 KBytes 4.19 Mbits/sec [ 3] 10.0-11.0 sec 512 KBytes 4.19 Mbits/sec [ 3] 11.0-12.0 sec 512 KBytes 4.19 Mbits/sec [ 3] 12.0-13.0 sec 640 KBytes 5.24 Mbits/sec [ 3] 13.0-14.0 sec 512 KBytes 4.19 Mbits/sec [ 3] 14.0-15.0 sec 640 KBytes 5.24 Mbits/sec [ 3] 15.0-16.0 sec 512 KBytes 4.19 Mbits/sec [ 3] 16.0-17.0 sec 640 KBytes 5.24 Mbits/sec [ 3] 17.0-18.0 sec 512 KBytes 4.19 Mbits/sec [ 3] 18.0-19.0 sec 640 KBytes 5.24 Mbits/sec [ 3] 19.0-20.0 sec 512 KBytes 4.19 Mbits/sec [ 3] 0.0-20.3 sec 8.88 MBytes 3.66 Mbits/sec
### and the way back... (i'll snip the logs generously) nodeD# iperf -c nodeC -w320k [ 3] 0.0-10.1 sec 25.5 MBytes 21.2 Mbits/sec
nodeC# iperf -c nodeB -w 320k [ 3] 0.0-10.0 sec 25.5 MBytes 21.3 Mbits/sec
nodeD# iperf -c nodeB -w320k [ 3] 0.0-10.2 sec 12.3 MBytes 10.0 Mbits/sec
This can be reproduced at will, and i don't understand what's happening.
Looks like node C is getting .. overwhelmed..(?) when forwarding the packets between B and D Even more intriguing is that the problem is much worse when going B -> D (tenfold slower) than from D -> B (twice slower)
##### more screenlogs follow
### similar in node B and C (in D, there's no wlan1) # batctl -v batctl 2012.0.0 [batman-adv: 2012.0.0] # batctl if wlan0-1: active # ath9k, internal, adhoc mode wlan1: active # ath9k_htc, usb, infrastructure mode # brctl show bridge name bridge id STP enabled interfaces br-lan 8000.228082e1fd06 no wlan0 eth0 bat0
### originators seen from nodeB nodeB# batctl o | cut -b -100 [B.A.T.M.A.N. adv 2012.0.0, MainIF/MAC: wlan0-1/56:e6:fc:be:29:d3 (bat0)] Originator last-seen (#/255) Nexthop [outgoingIF]: Potential nexthops ... A_wlan1 0.830s (250) A_wlan1 [ wlan1]: C_mesh ( 0) D_mesh 0.320s (230) C_wlan1 [ wlan1]: C_wlan1 (230) A_mesh 0.890s (250) A_wlan1 [ wlan1]: A_wlan1 (250) C_mesh 0.800s (241) C_wlan1 [ wlan1]: C_wlan1 (241) C_wlan1 0.720s (241) C_wlan1 [ wlan1]: A_wlan1 (226)
### originators seen from nodeC nodeC# batctl o |cut -b -100 [B.A.T.M.A.N. adv 2012.0.0, MainIF/MAC: wlan0-1/56:e6:fc:b9:b6:47 (bat0)] Originator last-seen (#/255) Nexthop [outgoingIF]: Potential nexthops ... B_mesh 0.830s (255) B_wlan1 [ wlan1]: D_mesh ( 0) A_wlan1 0.180s (242) A_wlan1 [ wlan1]: B_wlan1 (229) B_wlan1 0.280s (255) B_wlan1 [ wlan1]: A_wlan1 (233) D_mesh 0.030s (235) D_mesh [ wlan0-1]: A_wlan1 (194) A_mesh 0.430s (242) A_wlan1 [ wlan1]: B_wlan1 (230)
Any ideas, thoughts, pointers? Any additional info needed?
I've RTFW, RTFM, and even RTF-PDF concerning bandwidth degradation on single-interface-node mesh networks, and that's the reason why i'm (so far unsuccessfully) trying to overcome that problem with the additional USB interface.
Thanks a lot for the attention!
Guido
Hi,
i'm participating in the development of a wireless mesh network in Delta de Tigre, near Buenos Aires, Argentina. Landscape is absolutely flat, with dense forest 15-20mts tall. Nodes are being installed along the river, 50 - 150mts apart, with clear LOS at ground level. I collaborated with Nico Echaniz last month, bulding another community network in Cordoba, and given the success, I decided to start one more here. I'm also really thankful with Elektra, for recommending the tl-m3220 to Nico! We chose batman over olsr, in part because of avahi stuff, but also to have a real-world implementation and thus collaborate in testing / debugging :) So far, there are only 4 working nodes, but luckily I'm already facing unexpected behaviour.
I'm sorry for the long email, it's mostly terminal logs. In short, iperf between two neighbouring nodes reports >20mbps, yet running the same iperf between nodes separated by one hop reports inexplicably low <5mbps.
[..]
Any ideas, thoughts, pointers? Any additional info needed?
welcome on the list! I admit not having dived into all the details of your mail - the bigger part of the batman team is sitting in Athens at the moment (see battlemesh.org). The best next step is to find out whether this performance issue is a batman- adv or a wifi driver problem. May I suggest you remove batman-adv and repeat the same test using static routing ? If you still see the problem it is the wifi that is not working otherwise we have to fix batman-adv.
Regards, Marek
Hey thanks marek! I'm sorry i didn't think of trying that in the first place. Ath9k_htc has indeed a few issues so i wouldn't be suprised it's the culprit.
I was aware of the battlemesh v5 and last week i even piperlurked the mail archives, but didn't realize it would happen this week. Best luck in athens then! I will continue my little mesh-battle here. Will try static routes tomorrow and report anything interesting.
Cheers, guido
On 3/26/12, Marek Lindner lindner_marek@yahoo.de wrote:
Hi,
i'm participating in the development of a wireless mesh network in Delta de Tigre, near Buenos Aires, Argentina. Landscape is absolutely flat, with dense forest 15-20mts tall. Nodes are being installed along the river, 50 - 150mts apart, with clear LOS at ground level. I collaborated with Nico Echaniz last month, bulding another community network in Cordoba, and given the success, I decided to start one more here. I'm also really thankful with Elektra, for recommending the tl-m3220 to Nico! We chose batman over olsr, in part because of avahi stuff, but also to have a real-world implementation and thus collaborate in testing / debugging :) So far, there are only 4 working nodes, but luckily I'm already facing unexpected behaviour.
I'm sorry for the long email, it's mostly terminal logs. In short, iperf between two neighbouring nodes reports >20mbps, yet running the same iperf between nodes separated by one hop reports inexplicably low <5mbps.
[..]
Any ideas, thoughts, pointers? Any additional info needed?
welcome on the list! I admit not having dived into all the details of your mail - the bigger part of the batman team is sitting in Athens at the moment (see battlemesh.org). The best next step is to find out whether this performance issue is a batman- adv or a wifi driver problem. May I suggest you remove batman-adv and repeat the same test using static routing ? If you still see the problem it is the wifi that is not working otherwise we have to fix batman-adv.
Regards, Marek
(this time i'll keep the mail short, insane details are attached as text :) )
Well, luckily for batman-adv devs, but unfortunately for me, the problem is in fact related to relaying between wlan interfaces in mr3x20 hardware (at least)
I understand this is not anymore batman related, but maybe someone has experience on this, or can point me into the right direction / maillist ? Given you have a good deal of equipments at the BattleMesh, maybe someone has already tried adding a wifi dongle and can confirm seeing this behaviour? (in a line of 2-radio nodes with static routing, packet throughput is halved on each hop)
========
I've repeated the iperf tests in a controlled environment, this time using different subnets on each interface and static routes built with "route add". iptables flushed with -F and policy -P FORWARD ACCEPT no batman at all.
Packets coming in through wlan1 and out through wlan0 (or viceversa) experience (unexpected) "bandwidth degradation". Throughput drops from 30mbps to 15mbps (rough avg) The speed is closely comparable to using only 1 radio (packet comes in through wlan0, goes out through wlan0), so adding the dongle makes no net difference. (besides alternating the channel for transmission, throughput is halved over one hop anyway) 2 consecutive hops, alternating wifi radios, yield 1/3 throughput. (like it happens in a 1-radio mesh)
this does not happen if the packet comes in through eth0 and goes out through wlanX (or viceversa) . Throughput is maintained high at 30mbps.
=========
So the hardware (or kernel??) is uncapable of keeping up with the transfer speed *when two radios are involved*.
Has anyone bumped into a similar case, or has experience with these equipments? Maybe (again!) i am doing something wrong?
If this is confirmed, my only alternative to maintain good throughput along the mesh will be to install 2 x mr3220 on each node, connected back-to-back with ethernet cable. This will definitely work but the cost for each node owner evidently rises ;(
Thanks a lot!
Guido
Hey Guido,
the problem using one radio is explained further here (from another company, but this applies to any WiFi mesh technology)
http://revolutionwifi.blogspot.com/2012/02/mesh-network-performance-impact.h...
Also, if you use two-radio nodes, make sure that you use one module on 2.4 GHz and one 5 GHz - if you use two modules in the same frequency band with omni antennas, even if you use separate channels (like 1 and 11), they may interfere with each other if the antennas are near to each other. There are quite a few ppl who can confirm this, e.g. check this:
https://hackerspace.be/Wbm2009v2/TestInterferenceResults
Possible Solutions: * use different bands * use directional or sector antennas * have a reasonable distance between your omni antennas
Cheers, Simon
On Wed, Mar 28, 2012 at 07:41:05PM -0300, Guido Iribarren wrote:
(this time i'll keep the mail short, insane details are attached as text :) )
Well, luckily for batman-adv devs, but unfortunately for me, the problem is in fact related to relaying between wlan interfaces in mr3x20 hardware (at least)
I understand this is not anymore batman related, but maybe someone has experience on this, or can point me into the right direction / maillist ? Given you have a good deal of equipments at the BattleMesh, maybe someone has already tried adding a wifi dongle and can confirm seeing this behaviour? (in a line of 2-radio nodes with static routing, packet throughput is halved on each hop)
========
I've repeated the iperf tests in a controlled environment, this time using different subnets on each interface and static routes built with "route add". iptables flushed with -F and policy -P FORWARD ACCEPT no batman at all.
Packets coming in through wlan1 and out through wlan0 (or viceversa) experience (unexpected) "bandwidth degradation". Throughput drops from 30mbps to 15mbps (rough avg) The speed is closely comparable to using only 1 radio (packet comes in through wlan0, goes out through wlan0), so adding the dongle makes no net difference. (besides alternating the channel for transmission, throughput is halved over one hop anyway) 2 consecutive hops, alternating wifi radios, yield 1/3 throughput. (like it happens in a 1-radio mesh)
this does not happen if the packet comes in through eth0 and goes out through wlanX (or viceversa) . Throughput is maintained high at 30mbps.
=========
So the hardware (or kernel??) is uncapable of keeping up with the transfer speed *when two radios are involved*.
Has anyone bumped into a similar case, or has experience with these equipments? Maybe (again!) i am doing something wrong?
If this is confirmed, my only alternative to maintain good throughput along the mesh will be to install 2 x mr3220 on each node, connected back-to-back with ethernet cable. This will definitely work but the cost for each node owner evidently rises ;(
Thanks a lot!
Guido
Simon, you saved my day,
On Thu, Mar 29, 2012 at 9:51 AM, Simon Wunderlich simon.wunderlich@s2003.tu-chemnitz.de wrote:
the problem using one radio is explained further here (from another company, but this applies to any WiFi mesh technology)
http://revolutionwifi.blogspot.com/2012/02/mesh-network-performance-impact.h...
I was fully aware of this problem with single-radio nodes, and that's why I was trying to build cheap two-radio nodes using mr3220 + usb dongles.
Also, if you use two-radio nodes, they may interfere with each other if the antennas are near to each other. There are quite a few ppl who can confirm this, e.g. check this:
Ah! you hit the nail right in the head! I walked around the battlemesh wiki a couple of weeks ago, interested into past events results, but only came across an rsync daemon serving a tree with videos, photos and results, mostly in raw data form, collected during the last (before athens) WBM edition.
I think i was a bit short-sighted, since the Wbm2009v2 link (and the rest of that wbm wiki) was a wonderful read!
I did a couple of tests, and indeed increasing the distance between the dongle and the router makes the whole difference.
I googled more on the subject and found a paper with thorough testing results http://edoc.hu-berlin.de/oa/conferences/reSNAVwf8bA/PDF/28TjAjjEtU7ts.pdf
and even finer-grained testing here http://userver.ftw.at/~valerio/files/wons.pdf
None of them use 802.11n equipment, so i set out to make my own experiment. In my case, i needed about 1.5 meter between omni antennas to achieve full throughput, placing both omni antennas perpendicular to the ground and on the same plane. On the other hand, if i place them one over the other (that is, align their vertical axis, while holding them at different altitudes) a distance as little as 20cm was enough to overcome interference. (Sounds reasonable given the omnis gain spatial pattern)
I was able to get 20mbps from A to C (and viceversa), hopping on B. [A ch=11] -- [ ch=11 B(2 radio) ch=1] -- [ch=1 C] This was done using either BATMAN or static routing, giving almost equal results (~1mbps less using batman)
If i put the two radios in B really close to each other (<5 cm) throughput fell down to 8 mbps. It's a relief it was such a simple issue!
Thanks a lot Simon and Marek for getting me into the right path,
So far we haven't heard of anyone implementing OpenWRT+batman-adv on mr3220+usb to build two-radio-nodes mesh network, which means we're mostly in a "solo adventure" so your help and attention was even more appreciated!
Guido
b.a.t.m.a.n@lists.open-mesh.org
-
Guido Iribarren -
Marek Lindner -
Simon Wunderlich