Hi,
I tried the patch in production on our gateway as our problem is rather significant. I don't see an improvement in the pskb_expand_head() behaviour. (Flamegraph https://cloud.awlnx.space/owncloud/s/3wtoakc4mzFQSGs)
I tried to debug even further and added some debugging points in br_if.c and vxlan.c
This is what I get from the bridge when bat0 is enslaved with the vxlan interface as member of batman ( https://elixir.bootlin.com/linux/latest/source/net/bridge/br_if.c#L311 ) [ 36.959547] Bridge firewalling registered [ 522.221767] SKB Bridge br_if.c: max_headroom 0 [ 522.221781] SKB Bridge br_if.c: new_hr 0 [ 627.186129] SKB Bridge br_if.c: max_headroom 0 [ 627.186139] SKB Bridge br_if.c: new_hr 0 [ 627.616650] SKB Bridge br_if.c: new_hr 102
Also BATMAN reports itself when initialized and seems not to propagate stuff up the stack on change?: (https://github.com/open-mesh-mirror/batman-adv/blob/master/net/batman-adv/ha... ) [ 3350.212094] SKB hard-interface.h: soft_iface->needed_tailroom) 0 [3350.212105] SKB hard-interface.h: soft_iface->needed_headroom) 358 [ 3350.212116] SKB hard-interface.h: lower_headroom 70 [ 3350.212126] SKB hard-interface.h: needed_headroom 102
Also added some debugging to Fragmentation.c in BATMAN after the patch: Nov 25 17:48:26 raspi-1gb.awlnx.space kernel: SKB Fragmentation.c: ll_reserved 96 Nov 25 17:48:26 raspi-1gb.awlnx.space kernel: SKB Fragmentation.c: skb->len 762 Nov 25 17:48:26 raspi-1gb.awlnx.space kernel: SKB Fragmentation.c: header_size 20 Nov 25 17:48:26 raspi-1gb.awlnx.space kernel: SKB Fragmentation.c: fragment_size 762 Nov 25 17:48:26 raspi-1gb.awlnx.space kernel: SKB Fragmentation.c: ll_reserved 96
At the same time the VXLAN interface which is transported over Wireguard reports this (https://elixir.bootlin.com/linux/latest/source/drivers/net/vxlan.c#L2352 ) [ 567.515778] SKB VXLAN vxlan.c: min_headroom 200 [ 567.515792] SKB VXLAN vxlan.c: dst->header_len 0 [ 567.515805] SKB VXLAN vxlan.c: VXLAN_HLEN 16 [ 567.515819] SKB VXLAN vxlan.c: LL_RESERVED_SPACE(dst->dev) 144 [ 567.515831] SKB VXLAN vxlan.c: iphdr_len 40
So in my opinion the needed headroom reported by batman is wrong by maybe 200 ? As the min_headroom of vxlan seems to be 200 but BATMAN reports 102 up the stack to the bridge.
If you need any more input we are happy to help. Because the actual performance with running batman over vxlan is really bad. We have some figures here: https://gist.github.com/fadenb/9705059cf17eddf60e744e95bf926f05
Thank you for your help! Annika
-- Annika Wickert
EXARING AG
On 25.11.20, 13:25, "Sven Eckelmann" sven@narfation.org wrote:
TODO: write something about the extra headroom vxlan needs and why it reduced the performance significantly when only using the minimum reserved space.
Cc: Annika Wickert annika.wickert@exaring.de Signed-off-by: Sven Eckelmann sven@narfation.org --- net/batman-adv/fragmentation.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/net/batman-adv/fragmentation.c b/net/batman-adv/fragmentation.c index 97220e19..5039b201 100644 --- a/net/batman-adv/fragmentation.c +++ b/net/batman-adv/fragmentation.c @@ -391,6 +391,7 @@ bool batadv_frag_skb_fwd(struct sk_buff *skb,
/** * batadv_frag_create() - create a fragment from skb + * @net_dev: outgoing device for fragment * @skb: skb to create fragment from * @frag_head: header to use in new fragment * @fragment_size: size of new fragment @@ -401,22 +402,24 @@ bool batadv_frag_skb_fwd(struct sk_buff *skb, * * Return: the new fragment, NULL on error. */ -static struct sk_buff *batadv_frag_create(struct sk_buff *skb, +static struct sk_buff *batadv_frag_create(struct net_device *net_dev, + struct sk_buff *skb, struct batadv_frag_packet *frag_head, unsigned int fragment_size) { struct sk_buff *skb_fragment; unsigned int header_size = sizeof(*frag_head); unsigned int mtu = fragment_size + header_size; + int ll_reserved = LL_RESERVED_SPACE(net_dev);
- skb_fragment = netdev_alloc_skb(NULL, mtu + ETH_HLEN); + skb_fragment = dev_alloc_skb(ll_reserved + mtu); if (!skb_fragment) goto err;
skb_fragment->priority = skb->priority;
/* Eat the last mtu-bytes of the skb */ - skb_reserve(skb_fragment, header_size + ETH_HLEN); + skb_reserve(skb_fragment, ll_reserved + header_size); skb_split(skb, skb_fragment, skb->len - fragment_size);
/* Add the header */ @@ -439,11 +442,12 @@ int batadv_frag_send_packet(struct sk_buff *skb, struct batadv_orig_node *orig_node, struct batadv_neigh_node *neigh_node) { + struct net_device *net_dev = neigh_node->if_incoming->net_dev; struct batadv_priv *bat_priv; struct batadv_hard_iface *primary_if = NULL; struct batadv_frag_packet frag_header; struct sk_buff *skb_fragment; - unsigned int mtu = neigh_node->if_incoming->net_dev->mtu; + unsigned int mtu = net_dev->mtu; unsigned int header_size = sizeof(frag_header); unsigned int max_fragment_size, num_fragments; int ret; @@ -503,7 +507,7 @@ int batadv_frag_send_packet(struct sk_buff *skb, goto put_primary_if; }
- skb_fragment = batadv_frag_create(skb, &frag_header, + skb_fragment = batadv_frag_create(net_dev, skb, &frag_header, max_fragment_size); if (!skb_fragment) { ret = -ENOMEM; -- 2.29.2