Detection of modes of NVMe devices in SLES

SmashSmash New or Quiet Member

I am using NVMe devices on SLES15 sp1 and sp2 both. These NVMe devices are exposed through SAN. Because they exported from SAN, these devices can go readonly or back to readwrite due to some changes by host or internally. I am trying to find out a way in sales host if the device is readonly without doing any IOs to it. The small dd command that I did, succeeded but IO errors appeared in dmesg. At a later point when device turned rw, again I couldn't find if the device is rw now. dd command succeeded and no bad messages in dmesg.
In summary, I would like to understand if there is a programmatic way to find out if device is read only. I do see a file called “/sys/block/nvmeXnY/ro” but that always remained 0 even when device is readonly. Moreover, when I try "blockdev –rereadpt /dev/nvmeXnY; blockdev -report /dev/nvmeXnY", the output is always RW.

Comments

  • malcolmlewismalcolmlewis Knowledge Partner

    Hi
    Are there nvme* kernel modules in use lsmod | grep nvme? If so, then can look at the module options (parm) via the /sbin/modinfo <module> and systool -vm <module> to see the parm value in use.

    For example;

    /sbin/modinfo nvme-core | grep parm:
    
    parm:           fail_request:charp
    parm:           multipath:turn on native support for multiple controllers per subsystem (bool)
    parm:           admin_timeout:timeout in seconds for admin commands (uint)
    parm:           io_timeout:timeout in seconds for I/O (uint)
    parm:           shutdown_timeout:timeout in seconds for controller shutdown (byte)
    parm:           max_retries:max number of retries a command may have (byte)
    parm:           default_ps_max_latency_us:max power saving latency for new devices; use PM QOS to change per device (ulong)
    parm:           force_apst:allow APST for newly enumerated devices even if quirked off (bool)
    parm:           streams:turn on support for Streams write directives (bool)
    
    systool -vm nvme_core
    
    Module = "nvme_core"
    ....
      Parameters:
        admin_timeout       = "60"
        default_ps_max_latency_us= "0"
        force_apst          = "N"
        io_timeout          = "30"
        max_retries         = "5"
        multipath           = "Y"
        shutdown_timeout    = "5"
        streams             = "N"
    
  • SmashSmash New or Quiet Member

    Thanks for the response. I couldn't find systool utility for my sles15sp1. But in spite of that, I don't find any parameter indicating me the state of the particular device/namespace. ..
    e.g. in case of SCSI, I could do "blockdev --rereadpt ; blockdev --report " to get the latest state of the device, but the same isn't working for NVMe.
    Following is what I find from modinfo -
    /sbin/modinfo nvme-core | grep parm:
    parm: fail_request:charp
    parm: multipath:turn on native support for multiple controllers per subsystem (bool)
    parm: admin_timeout:timeout in seconds for admin commands (uint)
    parm: io_timeout:timeout in seconds for I/O (uint)
    parm: shutdown_timeout:timeout in seconds for controller shutdown (byte)
    parm: max_retries:max number of retries a command may have (byte)
    parm: default_ps_max_latency_us:max power saving latency for new devices; use PM QOS to change per device (ulong)
    parm: force_apst:allow APST for newly enumerated devices even if quirked off (bool)
    parm: streams:turn on support for Streams write directives (bool)

  • malcolmlewismalcolmlewis Knowledge Partner
    edited October 17

    Hi
    The cnf command helps :)

    cnf systool
    
    Program 'systool' is present in package 'sysfsutils', which is installed on your system.
    
    Absolute path to 'systool' is '/usr/bin/systool'. Please check your $PATH variable to see whether it contains the mentioned path.
    
    zypper if sysfsutils
    
    Information for package sysfsutils:
    -----------------------------------
    Repository     : SLE-Module-Basesystem15-SP1-Updates
    Name           : sysfsutils
    Version        : 2.1.0-3.3.1
    
    zypper in sysfsutils
    

    So are the nvme modules in use at all lsmod | grep nvme?

  • SmashSmash New or Quiet Member

    Below are the outputs requested. But still my query is same, how does this output help in determining device level state ?
    $ lsmod | grep nvme
    nvme 36864 4
    nvme_core 102400 7 nvme
    $ /sbin/modinfo nvme-core | grep parm:
    parm: fail_request:charp
    parm: multipath:turn on native support for multiple controllers per subsystem (bool)
    parm: admin_timeout:timeout in seconds for admin commands (uint)
    parm: io_timeout:timeout in seconds for I/O (uint)
    parm: shutdown_timeout:timeout in seconds for controller shutdown (byte)
    parm: max_retries:max number of retries a command may have (byte)
    parm: default_ps_max_latency_us:max power saving latency for new devices; use PM QOS to change per device (ulong)
    parm: force_apst:allow APST for newly enumerated devices even if quirked off (bool)
    parm: streams:turn on support for Streams write directives (bool)
    systool -vm nvme_core
    Module = "nvme_core"

    Attributes:
    coresize = "102400"
    initsize = "0"
    initstate = "live"
    refcnt = "7"
    srcversion = "7EBDDE9C727C600341E6CA1"
    supported = "Yes"
    taint = ""
    uevent =
    version = "1.0"

    Parameters:
    admin_timeout = "60"
    default_ps_max_latency_us= "100000"
    force_apst = "N"
    io_timeout = "30"
    max_retries = "5"
    multipath = "N"
    shutdown_timeout = "5"
    streams = "N"

    Sections:
    .bss = "0xffffffffc01b3340"
    .data..read_mostly = "0xffffffffc01b2d98"
    .data = "0xffffffffc01b1120"
    .exit.text = "0xffffffffc01a9e41"
    .gnu.linkonce.this_module= "0xffffffffc01b3000"
    .init.text = "0xffffffffc018b000"
    .note.gnu.build-id = "0xffffffffc01aa000"
    .orc_unwind = "0xffffffffc01adf73"
    .orc_unwind_ip = "0xffffffffc01ac4a7"
    .parainstructions = "0xffffffffc01b0cd0"
    .ref.data = "0xffffffffc01b2e20"
    .rodata.str1.1 = "0xffffffffc01ab515"
    .rodata.str1.8 = "0xffffffffc01aaab8"
    .rodata = "0xffffffffc01aa340"
    .smp_locks = "0xffffffffc01abfe0"
    .strtab = "0xffffffffc01906c8"
    .symtab = "0xffffffffc018c000"
    .text = "0xffffffffc01a0000"
    __bpf_raw_tp_map = "0xffffffffc01b2da0"
    __bug_table = "0xffffffffc01b2bc8"
    __jump_table = "0xffffffffc01b1000"
    __kcrctab_gpl = "0xffffffffc01aa290"
    __ksymtab_gpl = "0xffffffffc01aa030"
    __ksymtab_strings = "0xffffffffc01ac034"
    __mcount_loc = "0xffffffffc01b07a8"
    __param = "0xffffffffc01ac2e0"
    __tracepoints_ptrs = "0xffffffffc01ac448"
    __tracepoints = "0xffffffffc01b2f40"
    __tracepoints_strings= "0xffffffffc01ac460"
    __verbose = "0xffffffffc01b2c80"
    _ftrace_events = "0xffffffffc01b2e00"

  • malcolmlewismalcolmlewis Knowledge Partner

    Hi
    There are web (AWS) reports with nvme-core timing out, fixes include increasing the max_retries (10) and io_timeout (255) parameters via grub.

    What model NVME devices? I see intel has over provisioning issues;
    https://www.intel.com/content/www/us/en/support/articles/000022714/memory-and-storage.html

    The smartctl command can give insight to the condition of the device as well;
    For example;

    smartctl -a /dev/nvme0
    ....
    Available Spare:                    100%
    Available Spare Threshold:          10%
    Percentage Used:                    1%
    ....
    

    Have you looked through the journalctl output for errors?

  • SmashSmash New or Quiet Member

    Actually in my case the SSDs are not locked to read-only mode. The IOs go through fine when the NVMe device is made read-write and no errors are seen in /var/log/messages too, but when device is read-only, IOs still pass (echo $? reports 0), but IO failure errors are reported in /var/log/messages file. My question is - is there any attribute on device which indicate the right or current state ?

  • malcolmlewismalcolmlewis Knowledge Partner

    Hi
    Depending on the device, smartctl is the only one. Depending on the device manufacturer there maybe some tools available to run on linux. Power/cabling ok?

Sign In or Register to comment.