Hi,

we have the following problem with some HPC nodes. After some time (7-30 days) large parts of the memory
are occupied. There are no active or inactive processes using the memory. Any idea how do find the memory (ps,lsof , /proc/.. did not help).

System SLES 11SP3, 64GB RAM

Code:
BAD node:
node22:~ # cat /proc/meminfo 
MemTotal:       66066804 kB
MemFree:        49673020 kB
Buffers:        12298540 kB
Cached:          1467596 kB
SwapCached:            0 kB
Active:         12875052 kB
Inactive:         948312 kB
Active(anon):      42288 kB
Inactive(anon):       88 kB
Active(file):   12832764 kB
Inactive(file):   948224 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      16779260 kB
SwapFree:       16779260 kB
Dirty:                52 kB
Writeback:             0 kB
AnonPages:         42236 kB
Mapped:            13780 kB
Shmem:               140 kB
Slab:            1638204 kB
SReclaimable:    1578768 kB
SUnreclaim:        59436 kB
KernelStack:        6400 kB
PageTables:         2696 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    49812660 kB
Committed_AS:     115352 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      383588 kB
VmallocChunk:   34359352372 kB
HardwareCorrupted:     0 kB
AnonHugePages:     14336 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      227328 kB
DirectMap2M:    11290624 kB
DirectMap1G:    55574528 kB


GOOD node:
node08:~ # cat /proc/meminfo 
MemTotal:       66066804 kB
MemFree:        63056704 kB
Buffers:          814328 kB
Cached:           668740 kB
SwapCached:         6528 kB
Active:          1056692 kB
Inactive:         511812 kB
Active(anon):      32872 kB
Inactive(anon):    47336 kB
Active(file):    1023820 kB
Inactive(file):   464476 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      16779260 kB
SwapFree:       16761860 kB
Dirty:                88 kB
Writeback:             0 kB
AnonPages:         74692 kB
Mapped:            12376 kB
Shmem:                52 kB
Slab:             507524 kB
SReclaimable:     364540 kB
SUnreclaim:       142984 kB
KernelStack:        6416 kB
PageTables:         2980 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    49812660 kB
Committed_AS:     150608 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      383588 kB
VmallocChunk:   34359352372 kB
HardwareCorrupted:     0 kB
AnonHugePages:     32768 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       71680 kB
DirectMap2M:     4106240 kB
DirectMap1G:    62914560 kB