[clue-tech] I/O Error when attempting to remove an
empty (corrupt?)
William
wlist-clue at kimballstuff.com
Thu Feb 9 14:03:28 MST 2006
Jim Ockers wrote:
>Hi William,
>
>I need some more information to assist with this, but I love
>this kind of problem. :)
>
>
No problem. Aside from rebooting or taking down active services, I'll
provide anything you need.
>William wrote:
>
>>[root at kenshin ~]# cd /mnt/broken
>>[root at kenshin broken]# dir
>>total 16
>>drwxr-xr-x 3 root root 4096 Feb 7 14:42 .
>>drwxr-xr-x 4 root root 4096 Feb 9 09:59 ..
>>?--------- ? ? ? ? ? dnszones
>>[root at kenshin broken]# rm -rf dnszones
>>rm: cannot lstat `dnszones': Input/output error
>>
>>I broke this directory when I attempted to mount it to a samba share on
>>another (Windows NT 4) machine. While the logon I used should have
>>worked, it did not and I am unable to recover.
>>
>>
>
>I'm not sure I understand. Did you use smbmount or
>mount -t smbfs or what command did you use to mount it?
>
>
I wrote, then ran the following script (trimmed and censored here --
five other smbmounts to the same machine are not shown and did not cause
problems):
#!/bin/sh
if [ -d /mnt/nt4svr/ ]; then
smbumount /mnt/nt4svr/dnszones 2>/dev/null
rmdir /mnt/nt4svr/dnszones
rmdir /mnt/nt4svr
fi
mkdir /mnt/nt4svr
mkdir /mnt/nt4svr/dnszones
smbmount //nt4svr/DNSZones /mnt/nt4svr/dnszones -o
username=user%password,TTL=10000
echo Shares mounted on \\\\nt4svr:
ls -la /mnt/nt4svr/
>Can you reboot in singleuser mode and fsck the filesystem?
>What kind of filesystem is /mnt/broken on? I'm assuming
>ext2 or ext3.
>
>
Sadly, I cannot reboot this machine until off-hours (next weekend) -- it
is a production server. I admit that I was screwing around, researching
a theory, when I broke it. Yes, I should have been on a dev box,
instead. The HDDs are using ext3 (Linux LVM).
>Are there any kernel error messages from the hard disk
>device on which this filesystem lives? Run dmesg | less
>and troll through there looking for IDE errors or disk
>errors from your disk. If you don't see anything have a
>look at /var/log/messages.
>
>
This machine uses an array of SCSI drives attached to an Adaptec
back-plane. When I grep dmesg for SCSI or HDD, I find nothing
troublesome. /var/log/messages is equally disinteresting, except for:
[root at kenshin ~]# grep -B2 -A2 "dnszones" /var/log/messages*
/var/log/messages-Feb 9 09:59:53 kenshin mount.smbfs[11282]:
tdb(/var/log/samba/locks/gencache.tdb): tdb_lock failed on list 128
ltype=1 (Bad file descriptor)
/var/log/messages-Feb 9 09:59:53 kenshin mount.smbfs[11282]:
[2006/02/09 09:59:53, 0] client/smbmount.c:send_fs_socket(348)
/var/log/messages:Feb 9 09:59:53 kenshin mount.smbfs[11282]:
mount.smbfs[11282]: can't open /mnt/nt4svr/dnszones
/var/log/messages-Feb 9 09:59:53 kenshin mount.smbfs[11282]:
[2006/02/09 09:59:53, 0] client/smbmount.c:smb_umount(276)
/var/log/messages:Feb 9 09:59:53 kenshin mount.smbfs[11282]: 11282:
Could not umount /mnt/nt4svr/dnszones: No such file or directory
>Of course if you still want help with this issue then please
>post your findings from dmesg with your next message to the
>list.
>
>Then please reboot and fsck -f the filesystem.
>
>
Not possible today. If there is any chance, I need a solution that does
not require a reboot.
>Then run the following commands in sequence and post the
>output to the list, in the same manner as you did above.
>Note that dumpe2fs might output a lot of information. :)
>
>/bin/ls -ald /mnt/broken
>/bin/ls -ald /mnt/broken/dnszones
>/usr/bin/lsattr -d /mnt/broken/dnszones
>/bin/mount
>/bin/df /mnt/broken
>/bin/df /mnt/broken/dnszones
>/sbin/dumpe2fs /dev/hdd1 (or whatever the device name is for
> the filesystem on which /mnt/broken resides)
>
>
>
[root at kenshin ~]# /bin/ls -ald /mnt/broken
drwxr-xr-x 3 root root 4096 Feb 7 14:42 /mnt/broken
[root at kenshin ~]# /bin/ls -ald /mnt/broken/dnszones
/bin/ls: /mnt/broken/dnszones: Input/output error
[root at kenshin ~]# /usr/bin/lsattr -d /mnt/broken/dnszones
/usr/bin/lsattr: Input/output error while trying to stat
/mnt/broken/dnszones
[root at kenshin ~]# /bin/mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
//nt4svr/DNSZones on /mnt/nt4svr/dnszones type smbfs (0)
//nt4svr/DNSZones on /mnt/nt4svr/dnszones type smbfs (0)
[root at kenshin ~]# /bin/df /mnt/broken
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
33672984 1612364 30350132 6% /
[root at kenshin ~]# /bin/df /mnt/broken/dnszones
/bin/df: `/mnt/broken/dnszones': Input/output error
[root at kenshin ~]# /sbin/dumpe2fs /dev/sda1
dumpe2fs 1.35 (28-Feb-2004)
Filesystem volume name: /boot
Last mounted on: <not available>
Filesystem UUID: 7e830d4d-b4cd-47c3-aca3-67a01a83f55f
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index
filetype needs_recovery sparse_super
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 26104
Block count: 104388
Reserved block count: 5219
Free blocks: 88622
Free inodes: 26066
First block: 1
Block size: 1024
Fragment size: 1024
Reserved GDT blocks: 256
Blocks per group: 8192
Fragments per group: 8192
Inodes per group: 2008
Inode blocks per group: 251
Filesystem created: Tue Sep 20 13:37:11 2005
Last mount time: Sun Jan 8 12:17:47 2006
Last write time: Sun Jan 8 12:17:47 2006
Mount count: 4
Maximum mount count: -1
Last checked: Tue Sep 20 13:37:11 2005
Check interval: 0 (<none>)
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
Default directory hash: tea
Directory Hash Seed: 65dda302-17c0-4350-8e64-ac02182b617c
Journal backup: inode blocks
Group 0: (Blocks 1-8192)
Primary superblock at 1, Group descriptors at 2-2
Block bitmap at 259 (+258), Inode bitmap at 260 (+259)
Inode table at 261-511 (+260)
5 free blocks, 1987 free inodes, 2 directories
Free blocks: 4644-4648
Free inodes: 22-2008
Group 1: (Blocks 8193-16384)
Backup superblock at 8193, Group descriptors at 8194-8194
Block bitmap at 8451 (+258), Inode bitmap at 8452 (+259)
Inode table at 8453-8703 (+260)
4429 free blocks, 1991 free inodes, 1 directories
Free blocks: 11897-12288, 12303-12800, 12809-13312, 13323-13824,
13833-14336, 14338-14343, 14352-14848, 14859-16384
Free inodes: 2011, 2027-4016
Group 2: (Blocks 16385-24576)
Block bitmap at 16385 (+0), Inode bitmap at 16386 (+1)
Inode table at 16387-16637 (+2)
7939 free blocks, 2008 free inodes, 0 directories
Free blocks: 16638-24576
Free inodes: 4017-6024
Group 3: (Blocks 24577-32768)
Backup superblock at 24577, Group descriptors at 24578-24578
Block bitmap at 24835 (+258), Inode bitmap at 24836 (+259)
Inode table at 24837-25087 (+260)
7681 free blocks, 2008 free inodes, 0 directories
Free blocks: 25088-32768
Free inodes: 6025-8032
Group 4: (Blocks 32769-40960)
Block bitmap at 32769 (+0), Inode bitmap at 32770 (+1)
Inode table at 32771-33021 (+2)
7939 free blocks, 2008 free inodes, 0 directories
Free blocks: 33022-40960
Free inodes: 8033-10040
Group 5: (Blocks 40961-49152)
Backup superblock at 40961, Group descriptors at 40962-40962
Block bitmap at 41219 (+258), Inode bitmap at 41220 (+259)
Inode table at 41221-41471 (+260)
7681 free blocks, 2008 free inodes, 0 directories
Free blocks: 41472-49152
Free inodes: 10041-12048
Group 6: (Blocks 49153-57344)
Block bitmap at 49153 (+0), Inode bitmap at 49154 (+1)
Inode table at 49155-49405 (+2)
7939 free blocks, 2008 free inodes, 0 directories
Free blocks: 49406-57344
Free inodes: 12049-14056
Group 7: (Blocks 57345-65536)
Backup superblock at 57345, Group descriptors at 57346-57346
Block bitmap at 57603 (+258), Inode bitmap at 57604 (+259)
Inode table at 57605-57855 (+260)
7681 free blocks, 2008 free inodes, 0 directories
Free blocks: 57856-65536
Free inodes: 14057-16064
Group 8: (Blocks 65537-73728)
Block bitmap at 65537 (+0), Inode bitmap at 65538 (+1)
Inode table at 65539-65789 (+2)
7939 free blocks, 2008 free inodes, 0 directories
Free blocks: 65790-73728
Free inodes: 16065-18072
Group 9: (Blocks 73729-81920)
Backup superblock at 73729, Group descriptors at 73730-73730
Block bitmap at 73987 (+258), Inode bitmap at 73988 (+259)
Inode table at 73989-74239 (+260)
7681 free blocks, 2008 free inodes, 0 directories
Free blocks: 74240-81920
Free inodes: 18073-20080
Group 10: (Blocks 81921-90112)
Block bitmap at 81921 (+0), Inode bitmap at 81922 (+1)
Inode table at 81923-82173 (+2)
7939 free blocks, 2008 free inodes, 0 directories
Free blocks: 82174-90112
Free inodes: 20081-22088
Group 11: (Blocks 90113-98304)
Block bitmap at 90113 (+0), Inode bitmap at 90114 (+1)
Inode table at 90115-90365 (+2)
7939 free blocks, 2008 free inodes, 0 directories
Free blocks: 90366-98304
Free inodes: 22089-24096
Group 12: (Blocks 98305-104387)
Block bitmap at 98305 (+0), Inode bitmap at 98306 (+1)
Inode table at 98307-98557 (+2)
5830 free blocks, 2008 free inodes, 0 directories
Free blocks: 98558-104387
Free inodes: 24097-26104
>>I have tried stopping smb before issuing the rm -- no change. I have
>>tried remounting the share (with smb running) to no avail. At this
>>point, I just want this broken directory off my HDD.
>>
>>
>
>It seems most likely to me that the filesystem is corrupt
>and fsck will fix it. If the filesystem is corrupt perhaps
>a likely culprit is an actual disk i/o error. If you have
>IDE-SMART installed can you see if SMART is indicating any
>errors?
>
>
These are SCSI disks. Disk damage is possible -- one of these disks has
been removed due to failure in the last several months (before I
installed RHEL/CentOS4).
>Hope this helps,
>Jim
>
>
>
Thank you for your guidance. Sorry I cannot reboot the machine any time
soon.
_______________________________________________
CLUE-tech mailing list
CLUE-tech at cluedenver.org
http://cluedenver.org/mailman/listinfo/clue-tech
More information about the clue-tech
mailing list