PDA

View Full Version : why file system became corrupted



sharfuddin
28-Aug-2015, 13:28
OS: SLES 11 SP 3 + online updates.

Customer started noticing the following error:
Jul 22 13:45:30 node1 kernel: [23580.074503] EXT3-fs error (device dm-3): ext3_lookup: deleted inode referenced: 1352479

and then on 20th Aug, it was noticed 32 errors in b/w 08:47:11 and 08:47:14
Aug 20 08:47:11 node1 kernel: [2510147.739895] EXT3-fs error (device dm-3): ext3_lookup: deleted inode referenced: 1352539
.
.
.
Aug 20 08:47:14 node1 kernel: [2510150.315486] EXT3-fs error (device dm-3): ext3_lookup: deleted inode referenced: 1352539

They also noticed that Application(SAP) using this file system became unfunctional, then a reboot and running the fsck.ext3 was issued to repair the file system.

Now question here is why it got corrupted ? and what should one do to avoid such errors in future.

Regards,

jmozdzen
28-Aug-2015, 14:06
Hi sharfuddin,

> ext3_lookup: deleted inode referenced:
> Now question here is why it got corrupted ? and what should one do to avoid such errors in future.

in addition to file system implementation bugs (we all know these don't exist ;) ), this type of error is typical for system crashes or defective disk hardware.

Have a look at the server syslog, searching for errors indicating problems when accessing disks (i.e. "kernel.*scsi:" messages). Also, on more complicated setups like SANs, have a look at the storage back-end system and the connecting network.

Generally, with stable storage environments these errors are pretty rare. Nevertheless it is a good idea to have a file system check every year or two - typically requiring a reboot, though, to make sure the file systems aren't accessed and are unmounted at the time of checking.

If this re-occurs in rather short time frames, i.e. weeks, I'd suspect the storage hardware to be less reliable than required.

Regards,
Jens

maikcat
17-Sep-2015, 06:22
can you post the output of lsblk in this system? just curious...

Michael