Hi community,
We have batman-adv working on OpenWRT Chaos Calmer. - Atheros ar9331 MIPS platform + built-in ath9k WiFi - batman-adv version 2016.1 - routing_algos = BATMAN_IV - Wireless interface MTU = 1532, adhoc network encryption "psk2-ccmp" - bat0 interface MTU = 1500
We have batman-adv running on 10+ sites. For each site, there are 10-20 nodes in the mesh network.
batman-adv runs almost perfectly (*almost*). Occasionally (occurrence rate is low), node drops off the batman-adv / adhoc mesh. - Sometimes, node can recover (re-joins the mesh network automatically), but not always.
Any suggestions? Does batman-adv works perfectly in the field (i.e. running for 1 year with 100+ nodes without any issues)?
What about I use one node as Master, and other nodes ping this Master every 10s (or 30 seconds) (to keep mesh from inactivity)? Does this help?
Thanks. Xuebing Wang
Hello Xuebing,
it sounds like you have WiFi driver issues. There are some effects like key cache corruption, deafness, and other effects known for the AR93xx series. To confirm, you can try to use IPv6 link local ping (ping6 fe80:...%wlan0) to your neighbors. If you can ping but batman can't (e.g. use batctl) it's a batman issue. If both pings don't get through, it's most likely a WiFi driver issue. In this case, a master or something doesn't help. :)
Cheers, Simon
On Thursday, April 6, 2017 10:32:29 AM CEST Xuebing Wang wrote:
Hi community,
We have batman-adv working on OpenWRT Chaos Calmer.
- Atheros ar9331 MIPS platform + built-in ath9k WiFi
- batman-adv version 2016.1
- routing_algos = BATMAN_IV
- Wireless interface MTU = 1532, adhoc network encryption "psk2-ccmp"
- bat0 interface MTU = 1500
We have batman-adv running on 10+ sites. For each site, there are 10-20 nodes in the mesh network.
batman-adv runs almost perfectly (*almost*). Occasionally (occurrence rate is low), node drops off the batman-adv / adhoc mesh.
- Sometimes, node can recover (re-joins the mesh network automatically),
but not always.
Any suggestions? Does batman-adv works perfectly in the field (i.e. running for 1 year with 100+ nodes without any issues)?
What about I use one node as Master, and other nodes ping this Master every 10s (or 30 seconds) (to keep mesh from inactivity)? Does this help?
Thanks. Xuebing Wang
On Donnerstag, 6. April 2017 09:04:50 CEST Simon Wunderlich wrote:
Hello Xuebing,
it sounds like you have WiFi driver issues. There are some effects like key cache corruption, deafness, and other effects known for the AR93xx series.
Correct, completely forgot about the deaf and 0xdeadbeef issues. There were also two workarounds for them [1,2] from Simon.
Kind regards, Sven
[1] https://patchwork.kernel.org/patch/9433619/ [2] https://patchwork.kernel.org/patch/9433621/
Hi Sven, Simon,
Thank you very much for your help. These 2 patches are not merged upstream? I did see a similar patch merged (as you 2 are the authors, you know it very well):
[1] https://patchwork.kernel.org/patch/9433619/ [2] https://patchwork.kernel.org/patch/9433621/
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
Thanks again. xuebing
On 2017年04月06日 15:13, Sven Eckelmann wrote:
On Donnerstag, 6. April 2017 09:04:50 CEST Simon Wunderlich wrote:
Hello Xuebing,
it sounds like you have WiFi driver issues. There are some effects like key cache corruption, deafness, and other effects known for the AR93xx series.
Correct, completely forgot about the deaf and 0xdeadbeef issues. There were also two workarounds for them [1,2] from Simon.
Kind regards, Sven
[1] https://patchwork.kernel.org/patch/9433619/ [2] https://patchwork.kernel.org/patch/9433621/
On Donnerstag, 6. April 2017 16:05:26 CEST Xuebing Wang wrote:
Hi Sven, Simon,
Thank you very much for your help. These 2 patches are not merged upstream? I did see a similar patch merged (as you 2 are the authors, you know it very well):
[1] https://patchwork.kernel.org/patch/9433619/ [2] https://patchwork.kernel.org/patch/9433621/
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
The deaf patch is not applied upstream. But yes, the deadbeef patch was modified by Felix and then applied upstream (the commit you've listed here).
And these patches (in patchwork) are much newer than Chaos Calmer. So I would not expect to see them in the Chaos Calmer mac80211 package.
Kind regards, Sven
Hi Simon / Sven,
Do you know any hardware platform on which batman-adv (or batmand) work perfectly and field-proven? Thanks.
Xuebing Wang
On 2017年04月06日 15:04, Simon Wunderlich wrote:
Hello Xuebing,
it sounds like you have WiFi driver issues. There are some effects like key cache corruption, deafness, and other effects known for the AR93xx series. To confirm, you can try to use IPv6 link local ping (ping6 fe80:...%wlan0) to your neighbors. If you can ping but batman can't (e.g. use batctl) it's a batman issue. If both pings don't get through, it's most likely a WiFi driver issue. In this case, a master or something doesn't help. :)
Cheers, Simon
On Thursday, April 6, 2017 10:32:29 AM CEST Xuebing Wang wrote:
Hi community,
We have batman-adv working on OpenWRT Chaos Calmer.
- Atheros ar9331 MIPS platform + built-in ath9k WiFi
- batman-adv version 2016.1
- routing_algos = BATMAN_IV
- Wireless interface MTU = 1532, adhoc network encryption "psk2-ccmp"
- bat0 interface MTU = 1500
We have batman-adv running on 10+ sites. For each site, there are 10-20 nodes in the mesh network.
batman-adv runs almost perfectly (*almost*). Occasionally (occurrence rate is low), node drops off the batman-adv / adhoc mesh.
- Sometimes, node can recover (re-joins the mesh network automatically),
but not always.
Any suggestions? Does batman-adv works perfectly in the field (i.e. running for 1 year with 100+ nodes without any issues)?
What about I use one node as Master, and other nodes ping this Master every 10s (or 30 seconds) (to keep mesh from inactivity)? Does this help?
Thanks. Xuebing Wang
Hi Xuebing,
actually, ath9k is the most used hardware platform used with batman-adv, at least as far as I'm aware. It is used commercially as well on some hundred thousand devices. I would assume that it's very hard (or impossible) to find a "perfect" running hardware platform and drivers. On other drivers, you will have trouble to even get IBSS/11s mode running.
The question is, how do you handle faults - usually, there are workarounds applied which happen automatically like the ones we have referenced in this thread. They don't happen very often (i.e. less than once a day), and many of them can be fixed quite fast (except for the deaf problem, which may take ~30 seconds to recover).
However, there are also some problems which happen more often in certain environments or on certain hardware (maybe due to bad choices in RF design ...). There also may be problems in certain driver versions (i.e. regressions). And of course, there can be issues which have not yet been addressed.
BTW, I've just seen that you use psk2-ccmp. There is a known problem with key cache corruption. I'm not sure if it was already solved and merged in Chaos Calmer [1]. You may want to try if the problem appears without encryption.
Cheers, Simon
[1] https://patchwork.kernel.org/patch/9381651/
On Saturday, April 22, 2017 3:12:54 PM CEST Xuebing Wang wrote:
Hi Simon / Sven,
Do you know any hardware platform on which batman-adv (or batmand) work perfectly and field-proven? Thanks.
Xuebing Wang
On 2017年04月06日 15:04, Simon Wunderlich wrote:
Hello Xuebing,
it sounds like you have WiFi driver issues. There are some effects like key cache corruption, deafness, and other effects known for the AR93xx series. To confirm, you can try to use IPv6 link local ping (ping6 fe80:...%wlan0) to your neighbors. If you can ping but batman can't (e.g. use batctl) it's a batman issue. If both pings don't get through, it's most likely a WiFi driver issue. In this case, a master or something doesn't help. :)
Cheers,
SimonOn Thursday, April 6, 2017 10:32:29 AM CEST Xuebing Wang wrote:
Hi community,
We have batman-adv working on OpenWRT Chaos Calmer.
- Atheros ar9331 MIPS platform + built-in ath9k WiFi
- batman-adv version 2016.1
- routing_algos = BATMAN_IV
- Wireless interface MTU = 1532, adhoc network encryption "psk2-ccmp"
- bat0 interface MTU = 1500
We have batman-adv running on 10+ sites. For each site, there are 10-20 nodes in the mesh network.
batman-adv runs almost perfectly (*almost*). Occasionally (occurrence rate is low), node drops off the batman-adv / adhoc mesh.
- Sometimes, node can recover (re-joins the mesh network automatically),
but not always.
Any suggestions? Does batman-adv works perfectly in the field (i.e. running for 1 year with 100+ nodes without any issues)?
What about I use one node as Master, and other nodes ping this Master every 10s (or 30 seconds) (to keep mesh from inactivity)? Does this help?
Thanks. Xuebing Wang
Hi Simon,
Thank you very much. Sven pointed to me this key cache corruption before, but I did not take it because it is a bit complicated. I did take deaf and 0xdeadbeef patches. Sven, thank you.
I am doing simple change to use software encryption (int ath9k_modparam_nohwcrypt = 1). It seems working ok for my simple test. Throughput is down about 2Mbps, which is acceptable. CPU usage is now 60% (on ar9331) for max throughput scp, it is acceptable for my application.
I will put this software encryption in field test, at least it will show if my issue is related to key cache corruption or not.
Thanks again. Xuebing Wang
On 2017年04月22日 16:35, Simon Wunderlich wrote:
Hi Xuebing,
actually, ath9k is the most used hardware platform used with batman-adv, at least as far as I'm aware. It is used commercially as well on some hundred thousand devices. I would assume that it's very hard (or impossible) to find a "perfect" running hardware platform and drivers. On other drivers, you will have trouble to even get IBSS/11s mode running.
The question is, how do you handle faults - usually, there are workarounds applied which happen automatically like the ones we have referenced in this thread. They don't happen very often (i.e. less than once a day), and many of them can be fixed quite fast (except for the deaf problem, which may take ~30 seconds to recover).
However, there are also some problems which happen more often in certain environments or on certain hardware (maybe due to bad choices in RF design ...). There also may be problems in certain driver versions (i.e. regressions). And of course, there can be issues which have not yet been addressed.
BTW, I've just seen that you use psk2-ccmp. There is a known problem with key cache corruption. I'm not sure if it was already solved and merged in Chaos Calmer [1]. You may want to try if the problem appears without encryption.
Cheers, Simon
[1] https://patchwork.kernel.org/patch/9381651/
On Saturday, April 22, 2017 3:12:54 PM CEST Xuebing Wang wrote:
Hi Simon / Sven,
Do you know any hardware platform on which batman-adv (or batmand) work perfectly and field-proven? Thanks.
Xuebing Wang
On 2017年04月06日 15:04, Simon Wunderlich wrote:
Hello Xuebing,
it sounds like you have WiFi driver issues. There are some effects like key cache corruption, deafness, and other effects known for the AR93xx series. To confirm, you can try to use IPv6 link local ping (ping6 fe80:...%wlan0) to your neighbors. If you can ping but batman can't (e.g. use batctl) it's a batman issue. If both pings don't get through, it's most likely a WiFi driver issue. In this case, a master or something doesn't help. :)
Cheers,
SimonOn Thursday, April 6, 2017 10:32:29 AM CEST Xuebing Wang wrote:
Hi community,
We have batman-adv working on OpenWRT Chaos Calmer.
- Atheros ar9331 MIPS platform + built-in ath9k WiFi
- batman-adv version 2016.1
- routing_algos = BATMAN_IV
- Wireless interface MTU = 1532, adhoc network encryption "psk2-ccmp"
- bat0 interface MTU = 1500
We have batman-adv running on 10+ sites. For each site, there are 10-20 nodes in the mesh network.
batman-adv runs almost perfectly (*almost*). Occasionally (occurrence rate is low), node drops off the batman-adv / adhoc mesh.
- Sometimes, node can recover (re-joins the mesh network automatically),
but not always.
Any suggestions? Does batman-adv works perfectly in the field (i.e. running for 1 year with 100+ nodes without any issues)?
What about I use one node as Master, and other nodes ping this Master every 10s (or 30 seconds) (to keep mesh from inactivity)? Does this help?
Thanks. Xuebing Wang
Hi Simon,
=> It is used commercially as well on some hundred thousand devices.
Are you saying that batman-adv has hundred of thousands (100,000+) nodes installed? Thanks for your always help.
Xuebing Wang
On 2017年04月22日 16:35, Simon Wunderlich wrote:
Hi Xuebing,
actually, ath9k is the most used hardware platform used with batman-adv, at least as far as I'm aware. It is used commercially as well on some hundred thousand devices. I would assume that it's very hard (or impossible) to find a "perfect" running hardware platform and drivers. On other drivers, you will have trouble to even get IBSS/11s mode running.
The question is, how do you handle faults - usually, there are workarounds applied which happen automatically like the ones we have referenced in this thread. They don't happen very often (i.e. less than once a day), and many of them can be fixed quite fast (except for the deaf problem, which may take ~30 seconds to recover).
However, there are also some problems which happen more often in certain environments or on certain hardware (maybe due to bad choices in RF design ...). There also may be problems in certain driver versions (i.e. regressions). And of course, there can be issues which have not yet been addressed.
BTW, I've just seen that you use psk2-ccmp. There is a known problem with key cache corruption. I'm not sure if it was already solved and merged in Chaos Calmer [1]. You may want to try if the problem appears without encryption.
Cheers, Simon
[1] https://patchwork.kernel.org/patch/9381651/
On Saturday, April 22, 2017 3:12:54 PM CEST Xuebing Wang wrote:
Hi Simon / Sven,
Do you know any hardware platform on which batman-adv (or batmand) work perfectly and field-proven? Thanks.
Xuebing Wang
On 2017年04月06日 15:04, Simon Wunderlich wrote:
Hello Xuebing,
it sounds like you have WiFi driver issues. There are some effects like key cache corruption, deafness, and other effects known for the AR93xx series. To confirm, you can try to use IPv6 link local ping (ping6 fe80:...%wlan0) to your neighbors. If you can ping but batman can't (e.g. use batctl) it's a batman issue. If both pings don't get through, it's most likely a WiFi driver issue. In this case, a master or something doesn't help. :)
Cheers,
SimonOn Thursday, April 6, 2017 10:32:29 AM CEST Xuebing Wang wrote:
Hi community,
We have batman-adv working on OpenWRT Chaos Calmer.
- Atheros ar9331 MIPS platform + built-in ath9k WiFi
- batman-adv version 2016.1
- routing_algos = BATMAN_IV
- Wireless interface MTU = 1532, adhoc network encryption "psk2-ccmp"
- bat0 interface MTU = 1500
We have batman-adv running on 10+ sites. For each site, there are 10-20 nodes in the mesh network.
batman-adv runs almost perfectly (*almost*). Occasionally (occurrence rate is low), node drops off the batman-adv / adhoc mesh.
- Sometimes, node can recover (re-joins the mesh network automatically),
but not always.
Any suggestions? Does batman-adv works perfectly in the field (i.e. running for 1 year with 100+ nodes without any issues)?
What about I use one node as Master, and other nodes ping this Master every 10s (or 30 seconds) (to keep mesh from inactivity)? Does this help?
Thanks. Xuebing Wang
Hi Xuebing, On Wednesday, April 26, 2017 10:49:37 AM CEST Xuebing Wang wrote:
Hi Simon,
=> It is used commercially as well on some hundred thousand devices.
Are you saying that batman-adv has hundred of thousands (100,000+) nodes installed? Thanks for your always help.
Yes, installed and actively in use. The devices having batman-adv installed are probably some millions, since many Linux PC distributions ship the module along with their Linux kernel. On Debian for example, batman-adv is just an insmod away. ;)
Cheers, Simon
On Donnerstag, 6. April 2017 10:32:29 CEST Xuebing Wang wrote:
Hi community,
We have batman-adv working on OpenWRT Chaos Calmer.
- Atheros ar9331 MIPS platform + built-in ath9k WiFi
- batman-adv version 2016.1
Please update your batman-adv version. This one is known to have problems which can cause crashes or memory corruptions.
If you are using the Chaos Calmer packages version 2016.1-3 then you should already have some of the fixes (I think all fixes up to 2016.3 or so). But no one is updating the Chaos Calmer branch in the openwrt-routing packages feed anymore. And I doubt that anyone will do that now because LEDE 17.01 was released a while ago.
[...]
- Wireless interface MTU = 1532, adhoc network encryption "psk2-ccmp"
[..]
We have batman-adv running on 10+ sites. For each site, there are 10-20 nodes in the mesh network.
batman-adv runs almost perfectly (*almost*). Occasionally (occurrence rate is low), node drops off the batman-adv / adhoc mesh.
- Sometimes, node can recover (re-joins the mesh network automatically),
but not always.
This sounds like problems in the wifi stack/hw. batman-adv is not handling adhoc or encryption - it is simply using the links which are provided by your wifi stack/hardware.
Your description therefore looks more like you should fix your wifi. I don't know why you have low rates but the IBSS problems with enabled encryption could be caused by an HW issue. You should check out a workaround to refresh the HW key cache [1] when incoming packets cannot be decrypted anymore.
Any suggestions? Does batman-adv works perfectly in the field (i.e. running for 1 year with 100+ nodes without any issues)?
There are a lot of other components which will fail during that time. For example your wifi stack
What about I use one node as Master, and other nodes ping this Master every 10s (or 30 seconds) (to keep mesh from inactivity)? Does this help?
I have absolutely no idea against what this should help.
Kind regards, Sven
b.a.t.m.a.n@lists.open-mesh.org
-
Simon Wunderlich -
Sven Eckelmann -
Xuebing Wang