Local Disk Hotspot Analysis

One common mistake I see is NFS/Samba mounted partitions use to either stage data into memory or to write application log files to. There's no real excuse for this, and the argument that local storage is not allowed by intrastructure engineering policy is one which I easily argued against at a tier 1. We wanted about half a gig of file systems space to store logs and model data which could be loaded quickly in the event of a system or application crash. We were told that this was not allowed under any circumstances as it was against policy.

The issue we had was quite simple - on our Solaris box, restarting the application took 2 hours, pulling the data from an an ancient disk array with really poor service times and some config stuff NFS mounted partition. When we moved to Linux and used our local home dir to stage the models and store the logs, this went to 15 mins - an acceptable restart time for the business. So I arranged a meeting with the Head of Unix Infrastructure and told him that he needed to explain to the business why they couldn't trade for one and three quarter hours in the event of a system failure. For some reason he changed his mind and we managed to get two fast SATA II hard drives in addition to the two OS drives.

JBOD Arrays

If you are using a JBOD array with some form of disk suite management, frequently there's several performance problems that go unnoticed and unchecked - hardly surprising knowing the extreme workloads of some to the sys admins I've worked with. This leaves them no time to be proactive.

The first is what I call disk hot-spotting - where a bunch of disks are idle yet one is maxed due to poor partitioning. To analyse this, I wrote diskanalyse which aggregates sar output to highlight possible issue. Here's an example of two disks which just so happen to be the OS disks. It shows that there's a poor average service of 17ms per operation. Now that's going to slow things down. The cure here is quite simple - there's too much activity in /tmp - get the applicaiton guys to move it to a partition local to the application and the problem will be lessened.

avserv[ sd1,h ] = 0
avserv[ sd0 ] = 17
avserv[ sd320 ] = 2
avserv[ sd30 ] = 0
avserv[ sd1 ] = 17
avserv[ sd330 ] = 0

busy[ sd0 ] = 28
busy[ sd320 ] = 0
busy[ sd230 ] = 0
busy[ sd203 ] = 0
busy[ sd30 ] = 0
busy[ sd1 ] = 29

rwpersec[ sd0 ] = 57
rwpersec[ sd320 ] = 1
rwpersec[ sd230 ] = 0
rwpersec[ sd203 ] = 0
rwpersec[ sd30 ] = 0
rwpersec[ sd1 ] = 57

blkpersec[ sd0 ] = 920
blkpersec[ sd320 ] = 10
blkpersec[ sd230 ] = 0
blkpersec[ sd203 ] = 0
blkpersec[ sd30 ] = 0
blkpersec[ sd1 ] = 909

avwait[ sd0 ] = 7
avwait[ sd320 ] = 0
avwait[ sd230 ] = 0
avwait[ sd203 ] = 0
avwait[ sd30 ] = 0
avwait[ sd1 ] = 7