IO error at virtual disk

Its a VM(SLES 12 SP2) running on ESX 6.5. ESX is not showing any error, while at OS we are getting following errors:

kernel: [787030.198763] end_request: I/O error, dev sda, sector 3759139416
kernel: [1530802.448197] end_request: I/O error, dev sda, sector 3754016448
kernel: [1912330.306074] end_request: I/O error, dev sda, sector 3719306720
kernel: [1954566.577314] end_request: I/O error, dev sda, sector 3722956392

The disk(sda) is a virtual disk.

Please help, what and where should I check ?

Comments

  • Jens-U.Jens-U. Knowledge Partner
    Hi sharfuddin,

    > kernel: [787030.198763] end_request: I/O error, dev sda, sector 3759139416
    > Please help, what and where should I check ?

    are those reported sector numbers within the virtual disk's size? If not, those errors could be the result of a on-disk corruption (invalid sector numbers/offsets stored in whatever you put on that disk).

    Everything else depends on how that virtual disk is provisioned... is it from some VMFS pool? SAN? If it's some back-end storage system, does it report any errors?

    Regards,
    J
  • sharfuddinsharfuddin Senior Member
    jmozdzen wrote: »
    Hi sharfuddin,

    > kernel: [787030.198763] end_request: I/O error, dev sda, sector 3759139416
    > Please help, what and where should I check ?

    are those reported sector numbers within the virtual disk's size? If not, those errors could be the result of a on-disk corruption (invalid sector numbers/offsets stored in whatever you put on that disk).

    Everything else depends on how that virtual disk is provisioned... is it from some VMFS pool? SAN? If it's some back-end storage system, does it report any errors?

    Regards,
    J

    >are those reported sector numbers within the virtual disk's size?
    >
    How can I find this.. via "fdisk -l" ? Actually don't have access to the system right now.

    >is it from some VMFS pool?
    Yes.

    >SAN?
    No, locally attached disk on ESX host
  • Jens-U.Jens-U. Knowledge Partner
    Hi sharfuddin,
    sharfuddin wrote: »
    >are those reported sector numbers within the virtual disk's size?
    >
    How can I find this.. via "fdisk -l" ? Actually don't have access to the system right now.

    yes, can be a good information source. Have a look at the start and end sector information for your partitions.
    sharfuddin wrote: »
    >SAN?
    No, locally attached disk on ESX host

    You wrote that ESX had nothing in the logs - did you also look at ESX-system-related logs or just at those directly related to your virtual machine? If the latter, a look at the system logs might reveal information about intermittent disk errors reported by the system. Also, while I do not trust SMART further than I can throw our servers, maybe some SMART pre-failure error indicators may have increased. For below to work, you'll of course need to know which disks are actually used - or if it's hardware RAID, will have to take a totally different approach. Speaking of (hardware or software) RAID: If it's in use, do you give your arrays a check run at least once a week? The older the disks, the more important this can get to detect silently introduced consistency errors (this is, of course, completely unrelated to ESX, but a generally applicable advice).
    # esxcli storage core device list
    # esxcli storage core device smart get -d device
    

    Regards,
    J
  • KEVINKEVIN Knowledge Partner
    sharfuddin wrote:

    >
    > Its a VM(SLES 12 SP2) running on ESX 6.5. ESX is not showing any
    > error, while at OS we are getting following errors:
    >
    > kernel: [787030.198763] end_request: I/O error, dev sda, sector
    > 3759139416
    > kernel: [1530802.448197] end_request: I/O error, dev sda, sector
    > 3754016448
    > kernel: [1912330.306074] end_request: I/O error, dev sda, sector
    > 3719306720
    > kernel: [1954566.577314] end_request: I/O error, dev sda, sector
    > 3722956392
    >
    > The disk(sda) is a virtual disk.
    >
    > Please help, what and where should I check ?


    If SLES is reporting these errors, I would do a filesystem check from
    within SLES and see what it reports.

    --
    Kevin Boyle - Knowledge Partner
    If you find this post helpful and are logged into the web interface,
    please show your appreciation and click on the star below this post.
    Thank you.
Sign In or Register to comment.