I'm running SLES 11 SP3 with all the available updates.

I have two different fiber channel JDOB arrays with Intel 520 SSD's attached to the server and the write latency/IOPS is awful. I'm using a QLE2464 Qlogic HBA.

So here the scenerio. The server is modestly spec'd with 8 cores and 24GB or ram. I'm testing with IOMeter. When writing to an disk in the array the write latency is fairly steady 2100ms. I can't get above 30 IOPS. Even if I create a RAID 0 with 14 SSD's, the write latency is still 2100ms and IOPS barely go above 60 IOPS. Testing to the local RAID5 via the onboard HP SmartArray yields about 40ms response and 400 IOPS. Which is what I'd expect from the 4 disk SAS RAID 5. Read IOPS to the Fiberchannel SSDdisk looks ok. I get latency in the 20ms range and IOPS in the 4K to 5k range. So the issue appears to be write only.

Now, just to confirm I didnt have a hardware issue, I installed Windows 2008 R2 in a dual boot and tested with the same IOMETER spec. With a single SSD I can get about 20ms latency and 3000 IOPS. Which is more like it should be. This is with a plain install of 2008 and new tweaks to the drivers etc. I didnt change any of the hardware. Reboot back to SLES, and its slow.

So now that I know it isnt a hardware issue here is what I tried:
1. Disabled atime
2. Tried several file systems EXT3 and XFS (made sure the partitions where aligned and did NOT use LVM)
3. Changed the scheduler from CFS to NOOP
4. Disabled write barriers
5. Played with /proc/sys/vm/dirty_ratio and /proc/sys/vm/dirty_background_ratio as noted here: http://www.novell.com/support/kb/doc.php?id=7010287
6. Update the drivers from qlogics website and the HBA card firmware ,
7. HP DL360 bios and firmwares updated

nothing made any noticeable different.

I'm not seeing any errors in /var/logs or dmesg.

I'm stumped where to look next. Does anyone have any troubleshooting tips to track this down?

-M