[clue] [tech] [fixed] CentOS 6.3 tg3 (BCM5700 hardware) network driver bug causes kernel panics

[Jim Ockers]

Hi CLUEbies,

This is just for the record.  The CentOS 6.3 kernels 2.6.32-279.el6.i686 
and 2.6.32-279.11.1.el6.i686 have a bug in the tg3 network device driver 
that causes a kernel panic (along with CPU #0 stuck for 63s!) messages 
whenever you run "ifconfig eth0 down" or try to reboot the system.  
Because rebooting obviously does an ifconfig eth0 down.

We have an old Dell PowerEdge 2650 server and I'm trying to get CentOS 
6.3 to run reliably on it.  The server works OK I guess except for this 
error.  Apparently this bug has already been fixed by the ELRepo people 
in April 2012, but the fix has not been incorporated into the RHEL6 
kernel tree yet.  If they could go on ahead and include this fix in the 
RHEL6 kernel, that'd be GREAT:

http://lists.elrepo.org/pipermail/elrepo/2012-April/001187.html

The solution to get my server to not kernel panic and lock up all the 
time was to install the kmod-tg3 package from ELREPO as follows:

rpm --import http://elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://elrepo.org/elrepo-release-6-4.el6.elrepo.noarch.rpm
yum --disablerepo=\* --enablerepo=elrepo install kmod-tg3

Then I copied the tg3.ko module file that came with that RPM into the 
right directory for the kernel I was running - I just replaced the 
tg3.ko module file that was in there.  The ELRepo module loads OK 
presumably because the binary interfaces for EL6/CentOS6 are compatible 
enough.

Here is the bug report from abrt from one such kernel 
lockup/crash/panic, for googly pattern matching goodness for anyone else 
who is having this problem.  Note that the tg3_get_stats is the 
offending function according to the call trace.

:BUG: soft lockup - CPU#0 stuck for 63s! [irqbalance:1461]
:Modules linked in: autofs4 sunrpc p4_clockmod ipt_REJECT 
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT 
nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter 
ip6_tables i
pv6 uinput sg microcode dcdbas tg3 i2c_piix4 i2c_core ipmi_si 
ipmi_msghandler ext3 jbd mbcache raid10 raid1 sd_mod crc_t10dif aic7xxx 
scsi_transport_spi sr_mod cdrom pata_acpi ata_generic pata_serverworks 
dm_mir
ror dm_region_hash dm_log dm_mod [last unloaded: mperf]
:Modules linked in: autofs4 sunrpc p4_clockmod ipt_REJECT 
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT 
nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter 
ip6_tables i
pv6 uinput sg microcode dcdbas tg3 i2c_piix4 i2c_core ipmi_si 
ipmi_msghandler ext3 jbd mbcache raid10 raid1 sd_mod crc_t10dif aic7xxx 
scsi_transport_spi sr_mod cdrom pata_acpi ata_generic pata_serverworks 
dm_mir
ror dm_region_hash dm_log dm_mod [last unloaded: mperf]
:Pid: 1461, comm: irqbalance Not tainted 2.6.32-279.11.1.el6.i686 #1 
Dell Computer Corporation PowerEdge 2650             /0H3014
:EIP: 0060:[<c083ed83>] EFLAGS: 00000283 CPU: 0
:EIP is at _spin_lock_bh+0x13/0x30
:EAX: f593e0de EBX: c15033e4 ECX: c0882488 EDX: f8110bf0:ESI: f542d000 
EDI: 00000000 EBP: c15033e4 ESP: f5931e94
:DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
:Process irqbalance (pid: 1461, ti=f5930000 task=f639e000 task.ti=f5930000)
:Stack:
:c15033e0 f8110d8c f5634280 f5634280 c1503020 c1503020 000001be c0792440
: f5634280 c0792ba9 f5634280 c09e34c0 f5914020 0015cf30 00003585 00000000
: 00000000 00000000 00000000 00000000 000004c3 000231c2 00000130 00000000
:Call Trace:
:[<f8110d8c>] ? tg3_get_stats+0x19c/0x210 [tg3]
:[<c0792440>] ? dev_get_stats+0x10/0x30:[<c0792ba9>] ? 
dev_seq_show+0x19/0xe0:[<c054a90f>] ? seq_read+0x22f/0x3d0
:[<c054a6e0>] ? seq_read+0x0/0x3d0:[<c057bb24>] ? 
proc_reg_read+0x64/0xa0:[<c057bac0>] ? proc_reg_read+0x0/0xa0
:[<c052df6d>] ? vfs_read+0x9d/0x190
:[<c052e0a1>] ? sys_read+0x41/0x70
:[<c083ef94>] ? syscall_call+0x7/0xb
:[<c0830000>] ? quirk_usb_early_handoff+0x53d/0x632
:Code: 89 d8 f0 83 28 01 79 05 e8 ab fd ff ff 5b c3 89 f6 8d bc 27 00 00 
00 00 53 89 c3 e8 e8 e0 c1 ff 66 b8 00 01 f0 66 0f c1 03 38 e0 <74> 0e 
f3 90 8a 03 eb f6 66 83 3b 00 75 f4 eb e5 5b c3 8d 74 26

Just for the record,
Jim

-- 
Jim Ockers, P.E., P.Eng. (ockers at ockers.net)
Contact info: http://www.ockers.net/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://cluedenver.org/pipermail/clue/attachments/20121030/49893768/attachment.html