Hi Martin, hi Sven, hi all
I couldn't help myself and started ntop after a while. Now I come to the conclusion that ntop has nothing to do with the crash. I think the machine simply crashes on the next task in the stack. (but really, I don't know what I'm talking about)
I understand, that the patch is quick & dirty and can't protect the VM from crash. But it's a big step forward identifying the cause of the bug and that your patch provides us with some output.
Best regards an happy hacking
Philipp
________________________ Freifunk Rheinland e. V. – Funkzelle Wuppertal –
KERNEL: /usr/src/linux-3.17.4-gentoo/vmlinux DUMPFILE: vmcore_20141130174721 CPUS: 1 DATE: Thu Jan 1 01:00:00 1970 UPTIME: 02:29:15 LOAD AVERAGE: 0.17, 0.17, 0.15 TASKS: 141 NODENAME: wolke RELEASE: 3.17.4-gentoo VERSION: #1 SMP Tue Nov 25 12:37:10 CET 2014 MACHINE: x86_64 (2593 Mhz) MEMORY: 511.6 MB PANIC: "" PID: 0 COMMAND: "swapper/0" TASK: ffffffff81a19480 [THREAD_INFO: ffffffff81a00000] CPU: 0 STATE: TASK_RUNNING (PANIC)
crash> bt PID: 0 TASK: ffffffff81a19480 CPU: 0 COMMAND: "swapper/0" #0 [ffff88001fc034e0] machine_kexec at ffffffff8103ab9e #1 [ffff88001fc03540] crash_kexec at ffffffff810bfa23 #2 [ffff88001fc03610] oops_end at ffffffff810060f8 #3 [ffff88001fc03640] die at ffffffff81006593 #4 [ffff88001fc03670] do_general_protection at ffffffff8100341a #5 [ffff88001fc036a0] general_protection at ffffffff81620388 [exception RIP: __kmalloc_node_track_caller+237] RIP: ffffffff8115c24d RSP: ffff88001fc03758 RFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88001a2d2c00 RCX: 0000000000391dc8 RDX: 0000000000391dc7 RSI: 0000000000000000 RDI: 0000000000015900 RBP: ffff88001fc03798 R8: ffff88001fc15900 R9: ffffffff81466f1b R10: b713fedcfcf04441 R11: ffff88001a2d2200 R12: ffff88001f001400 R13: 0000000000000740 R14: 00000000ffffffff R15: 0000000000010220 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #6 [ffff88001fc037a0] __kmalloc_reserve at ffffffff81464387 #7 [ffff88001fc037f0] __alloc_skb at ffffffff81466f48 #8 [ffff88001fc03850] skb_segment at ffffffff81467e29 #9 [ffff88001fc03970] tcp_gso_segment at ffffffff814e7e82 #10 [ffff88001fc039e0] inet_gso_segment at ffffffff814f7d57 #11 [ffff88001fc03a50] skb_mac_gso_segment at ffffffff81475d83 #12 [ffff88001fc03a90] __skb_gso_segment at ffffffff81475ebb #13 [ffff88001fc03ab0] dev_hard_start_xmit at ffffffff814760a3 #14 [ffff88001fc03b10] __dev_queue_xmit at ffffffff81476798 #15 [ffff88001fc03b60] dev_queue_xmit at ffffffff8147696b #16 [ffff88001fc03b70] ip_finish_output at ffffffff814c466b #17 [ffff88001fc03bd0] ip_output at ffffffff814c5128 #18 [ffff88001fc03c00] ip_forward_finish at ffffffff814c0d41 #19 [ffff88001fc03c20] ip_forward at ffffffff814c10fe #20 [ffff88001fc03c60] ip_rcv_finish at ffffffff814bef2c #21 [ffff88001fc03c90] ip_rcv at ffffffff814bf86c #22 [ffff88001fc03cd0] __netif_receive_skb_core at ffffffff81474152 #23 [ffff88001fc03d40] __netif_receive_skb at ffffffff81474691 #24 [ffff88001fc03d60] netif_receive_skb_internal at ffffffff81474878 #25 [ffff88001fc03d90] napi_gro_complete at ffffffff814749dc #26 [ffff88001fc03dc0] dev_gro_receive at ffffffff81474c0f #27 [ffff88001fc03e20] napi_gro_receive at ffffffff81475224 #28 [ffff88001fc03e50] gro_cell_poll at ffffffff81507e07 #29 [ffff88001fc03ea0] net_rx_action at ffffffff81474f31 #30 [ffff88001fc03f00] __do_softirq at ffffffff81052e28 #31 [ffff88001fc03f60] irq_exit at ffffffff81053205 #32 [ffff88001fc03f70] do_IRQ at ffffffff810046f2 --- <IRQ stack> --- #33 [ffffffff81a03de8] ret_from_intr at ffffffff8161f26d [exception RIP: tick_nohz_idle_exit+291] RIP: ffffffff810b1923 RSP: ffffffff81a03e98 RFLAGS: 00000202 RAX: ffff88001fc0d080 RBX: ffff88001fc0d4a0 RCX: 0000000000000020 RDX: 0000000000000000 RSI: 0000000000000086 RDI: 0000000000000008 RBP: ffffffff81a03ea8 R8: 00000000209d3188 R9: 0000000000000001 R10: 0000000000000005 R11: 0000000000000004 R12: ffffffff810a33d7 R13: ffffffff81a03e78 R14: 0000000000000086 R15: ffffffff81a03df8 ORIG_RAX: ffffffffffffff8e CS: 0010 SS: 0018 #34 [ffffffff81a03eb0] cpu_startup_entry at ffffffff810844c7 #35 [ffffffff81a03f10] rest_init at ffffffff81610332 #36 [ffffffff81a03f20] start_kernel at ffffffff81ad8062 #37 [ffffffff81a03f70] x86_64_start_reservations at ffffffff81ad75cc #38 [ffffffff81a03f80] x86_64_start_kernel at ffffffff81ad7714 crash> log […] [ 6.765959] Adding 1571836k swap on /dev/vda2. Priority:-1 extents:1 across:1571836k [ 13.008359] named (1695) used greatest stack depth: 11784 bytes left [ 62.407977] tun: Universal TUN/TAP device driver, 1.6 [ 62.407981] tun: (C) 1999-2004 Max Krasnyansky maxk@qualcomm.com [ 63.275215] batman_adv: B.A.T.M.A.N. advanced 2014.3.0-44-g650251a-dirty (compatibility version 15) loaded [ 63.541480] batman_adv: bat0: Adding interface: fastd0 [ 63.541484] batman_adv: bat0: The MTU of interface fastd0 is too small (1426) to handle the transport of batman-adv packets. Packets going over this interface will be fragmented on layer2 which could impact the performance. Setting the MTU to 1532 would solve the problem. [ 63.541493] batman_adv: bat0: Interface activated: fastd0 [ 63.553877] batman_adv: bat0: orig_interval: Changing from: 1000 to: 5000 [ 63.561411] batman_adv: bat0: bridge_loop_avoidance: Changing from: disabled to: enabled [ 63.566465] batman_adv: bat0: Changing gw mode from: off to: client [ 65.511629] ipip: IPv4 over IPv4 tunneling driver [ 77.521583] random: nonblocking pool is initialized [ 103.144238] batman_adv: bat0: Changing gw mode from: client to: server [ 103.144300] batman_adv: bat0: Changing gateway bandwidth from: '10.0/2.0 MBit' to: '90.0/90.0 MBit' [ 345.305038] nf_conntrack: automatic helper assignment is deprecated and it will be removed soon. Use the iptables CT target to attach helpers instead. [ 2998.421458] device eth0 entered promiscuous mode [ 8955.565935] batadv_frag_merge_packets: i: 1, size: 1380, entry->seqno: 16640, entry->size: 3512, entry->total_size: 34816 [ 8955.565940] skb->len: 84, skb->tailroom: 522, pkt->pkt_type: 64, pkt->version: 15, pkt->no: 0, pkt->seqno: 33848, pkt->total_size: 14578 [ 8955.565942] skb->len: 1400, skb->tailroom: 250, pkt->pkt_type: 65, pkt->version: 15, pkt->no: 0, pkt->seqno: 56874, pkt->total_size: 1464 [ 8955.571490] batadv_frag_merge_packets: i: 1, size: 1380, entry->seqno: 16384, entry->size: 3512, entry->total_size: 34816 [ 8955.571498] skb->len: 84, skb->tailroom: 522, pkt->pkt_type: 64, pkt->version: 15, pkt->no: 0, pkt->seqno: 33848, pkt->total_size: 14578 [ 8955.571500] skb->len: 1400, skb->tailroom: 250, pkt->pkt_type: 65, pkt->version: 15, pkt->no: 0, pkt->seqno: 56875, pkt->total_size: 1464 [ 8955.614084] general protection fault: 0000 [#1] SMP [ 8955.614225] Modules linked in: xt_nat iptable_nat nf_nat_ipv4 nf_nat ipip batman_adv(O) libcrc32c tun crc32c_intel aesni_intel aes_x86_64 glue_helper intel_agp lrw gf128mul intel_gtt ablk_helper agpgart cryptd psmouse mousedev evdev [ 8955.614822] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 3.17.4-gentoo #1 [ 8955.614919] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007 [ 8955.614919] task: ffffffff81a19480 ti: ffffffff81a00000 task.ti: ffffffff81a00000 [ 8955.614919] RIP: 0010:[<ffffffff8115c24d>] [<ffffffff8115c24d>] __kmalloc_node_track_caller+0xed/0x1b0 [ 8955.614919] RSP: 0018:ffff88001fc03758 EFLAGS: 00010246 [ 8955.614919] RAX: 0000000000000000 RBX: ffff88001a2d2c00 RCX: 0000000000391dc8 [ 8955.614919] RDX: 0000000000391dc7 RSI: 0000000000000000 RDI: 0000000000015900 [ 8955.614919] RBP: ffff88001fc03798 R08: ffff88001fc15900 R09: ffffffff81466f1b [ 8955.614919] R10: b713fedcfcf04441 R11: ffff88001a2d2200 R12: ffff88001f001400 [ 8955.614919] R13: 0000000000000740 R14: 00000000ffffffff R15: 0000000000010220 [ 8955.614919] FS: 0000000000000000(0000) GS:ffff88001fc00000(0000) knlGS:0000000000000000 [ 8955.614919] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 8955.614919] CR2: 00007ff981305000 CR3: 0000000002201000 CR4: 00000000000006f0 [ 8955.614919] Stack: [ 8955.614919] ffff88001fc03768 ffffffff81466f48 ffff88001fc037a8 ffff88001a2d2c00 [ 8955.614919] ffff88001fc03817 0000000000000020 0000000000000740 00000000ffffffff [ 8955.614919] ffff88001fc037e8 ffffffff81464387 0000000000000000 0000000000000000 [ 8955.614919] Call Trace: [ 8955.614919] <IRQ> [ 8955.614919] [ 8955.614919] [<ffffffff81466f48>] ? __alloc_skb+0x88/0x2a0 [ 8955.614919] [<ffffffff81464387>] __kmalloc_reserve.isra.58+0x37/0xa0 [ 8955.614919] [<ffffffff81466f48>] __alloc_skb+0x88/0x2a0 [ 8955.614919] [<ffffffff81467e29>] skb_segment+0x4b9/0x940 [ 8955.614919] [<ffffffffa009cfa3>] ? tun_net_xmit+0x263/0x320 [tun] [ 8955.614919] [<ffffffff814e7e82>] tcp_gso_segment+0x122/0x4f0 [ 8955.614919] [<ffffffff814f7d57>] inet_gso_segment+0x137/0x390 [ 8955.614919] [<ffffffff81475d83>] skb_mac_gso_segment+0x93/0x170 [ 8955.614919] [<ffffffff81475ebb>] __skb_gso_segment+0x5b/0xc0 [ 8955.614919] [<ffffffff814760a3>] dev_hard_start_xmit+0x183/0x580 [ 8955.614919] [<ffffffff814c4000>] ? ip_finish_output2+0x300/0x300 [ 8955.614919] [<ffffffff81476798>] __dev_queue_xmit+0x2f8/0x4b0 [ 8955.614919] [<ffffffff8147696b>] dev_queue_xmit+0xb/0x10 [ 8955.614919] [<ffffffff814c466b>] ip_finish_output+0x66b/0x7f0 [ 8955.614919] [<ffffffff814c5128>] ip_output+0x88/0x90 [ 8955.614919] [<ffffffff814c0d41>] ip_forward_finish+0x61/0x80 [ 8955.614919] [<ffffffff814c10fe>] ip_forward+0x39e/0x430 [ 8955.614919] [<ffffffff814bef2c>] ip_rcv_finish+0x7c/0x320 [ 8955.614919] [<ffffffff814bf86c>] ip_rcv+0x2dc/0x3f0 [ 8955.614919] [<ffffffff81474152>] __netif_receive_skb_core+0x222/0x740 [ 8955.614919] [<ffffffff81474691>] __netif_receive_skb+0x21/0x70 [ 8955.614919] [<ffffffff81474878>] netif_receive_skb_internal+0x28/0x90 [ 8955.614919] [<ffffffff814e7cee>] ? tcp4_gro_complete+0x6e/0x70 [ 8955.614919] [<ffffffff814749dc>] napi_gro_complete+0x9c/0xd0 [ 8955.614919] [<ffffffff81474c0f>] dev_gro_receive+0x1ff/0x300 [ 8955.614919] [<ffffffff81475224>] napi_gro_receive+0x34/0x100 [ 8955.614919] [<ffffffff81507e07>] gro_cell_poll+0x77/0xb0 [ 8955.614919] [<ffffffff81474f31>] net_rx_action+0x141/0x240 [ 8955.614919] [<ffffffff81052e28>] __do_softirq+0xe8/0x280 [ 8955.614919] [<ffffffff81053205>] irq_exit+0x95/0xa0 [ 8955.614919] [<ffffffff810046f2>] do_IRQ+0x62/0x110 [ 8955.614919] [<ffffffff8161f26d>] common_interrupt+0x6d/0x6d [ 8955.614919] <EOI> [ 8955.614919] [ 8955.614919] [<ffffffff810b1923>] ? tick_nohz_idle_exit+0x123/0x1a0 [ 8955.614919] [<ffffffff810b196d>] ? tick_nohz_idle_exit+0x16d/0x1a0 [ 8955.614919] [<ffffffff810844c7>] cpu_startup_entry+0x137/0x330 [ 8955.614919] [<ffffffff81610332>] rest_init+0x72/0x80 [ 8955.614919] [<ffffffff81ad8062>] start_kernel+0x422/0x42f [ 8955.614919] [<ffffffff81ad7a2d>] ? set_init_arg+0x58/0x58 [ 8955.614919] [<ffffffff81ad7117>] ? early_idt_handlers+0x117/0x120 [ 8955.614919] [<ffffffff81ad75cc>] x86_64_start_reservations+0x2a/0x2c [ 8955.614919] [<ffffffff81ad7714>] x86_64_start_kernel+0x146/0x155 [ 8955.614919] Code: 00 4c 89 d0 48 8b 5d d8 4c 8b 65 e0 4c 8b 6d e8 4c 8b 75 f0 4c 8b 7d f8 c9 c3 0f 1f 40 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <49> 8b 1c 02 4c 89 d0 65 48 0f c7 0f 0f 94 c0 84 c0 0f 84 56 ff [ 8955.614919] RIP [<ffffffff8115c24d>] __kmalloc_node_track_caller+0xed/0x1b0 [ 8955.614919] RSP <ffff88001fc03758>