Today's topic - Disk Performance. A warning to the squeamish...Math ahead.
Throughput refers to the amount of data read or written per unit of time. Generally measured in units like Megabytes per second (MB/s), or Gigabytes per Hours (GB/h). Often when dealing with networks we see Kilobits per second (Kb/s) or Megabits per second (Mb/s). Note that the abbreviations of some of those units look similar, pay attention to the capitalization because the differences are a factor of at least 8x.
It's easy to talk about a hard drive, or a backup job, or even a network interface providing 250MB/s throughput and understand that if I have 500GB of data that it's going to take a little over a half hour to transfer the data. (500GB * 1024MB/GB / 250MB/s / 3600s/h = 0.56h)
Throughput is by far the most talked about and in general most understood measure of performance. By the same token when people ask about network performance they often go to speedtest.com and tell me that "my network is fast, because I get 15Mb/s download." I agree that's a pretty decent throughput, but that's not the only measure of performance that's important.
A second measure is Response Time (or latency). This is a measure of how long it takes a request to complete. In the network world we think about this being how long it takes a message to arrive at it's destination after being sent. In the disk world we think about how long from when we request an IO operation happen until the system completes it. Disk latency (and network latency) are often measured in direct units of time - milliseconds (ms) or microseconds (us), and occasionally in seconds (s). Hopefully you never see IT technology latency measured in hours or days unless you're using an RFC1149 circuit.
The combination of a request response time and throughput, combined with the size of the request (amount of data moved at a time) yields a metric which amounts to how many requests can be completed per unit of time. We see this most often in the disk world as I/O operations per second or IOPS. We talk about IOPS a way of thinking about how "fast" a given disk system is, but it's arguably more of a measure of workload capability than either latency or throughput; however both latency and throughput contribute to the maximum IO operations per second a given disk can mange.
For example - if we have a hard disk with an maximum physical throughput of 125MB/sec, which is capable of processing requests at a rate of 80 requests per second, what is the throughput of the drive if my workload consists of 4KB reads and writes? Well in theory at 125MB/sec throughput the drive could process 125MB/s * 1024KB/MB / 4KB/IO = 32,000 IO/s. Hold on, the drive is only capable of 80 IOPS so the maximum physical throughput won't be achieved. 80IO/s * 4KB/IO = 320KB/s. If we wanted to maximize this drive's throughput we need to increase the size (or payload) of the IO requests. Ideally we'd perform reads an writes in blocks equal to the maximum throughput divided by the maximum IO rate (125MB/s / 80IO/s = 1.562MB).
This last trick by the way is what many vendors use to improve the performance of relatively "slow" hard disks; referred to as IO coalescing they take many small IO operations and buffer them until they can perform one large physical IO.
What governs the drive's maximum IOPS is actually a function of multiple factors.
Physical media throughput is one of them - which is governed by the physical block size (often 512bytes), the number of blocks per track on the platter, the number of platters, and the rotational velocity of the drive (typically measured in revolutions per minute - RPM). The idea here being that the drive can only transfer data to or from the platters at the rate at which the data is moving under the heads. If we have a drive spinning at 7200RPM, with say 100 blocks per track, and 512bytes/block and a single platter/head we have a drive with a maximum physical media transfer rate of 512B/block * 100blocks/track * 7200 tracks/minute / 60seconds/minute / 1024bytes/KB / 1024KB/MB = 5.85MB/s. Under no circumstances can the physical throughput of the drive exceed this value, because the data simply isn't passing under the head any faster.
To improve this value you can add more heads (with two platters and 4 heads this drive could potentially move 23.4MB/s). You can increase the number of blocks per track (with 200 blocks per track and one head the drive would have a throughput of 11.7MB/s). Or you can increase the velocity at which the drive spins (at 36,000 RPM this drive would move 29.25MB/sec). As you can see though this maximum throughput is governed by the physical characteristics of the drive.
A second factor impacting the IOPS is the question of how long it takes to position the head to read or write a particular block from the disk. IO operations start at a given block and then proceed to read or write subsequent sequential blocks until the size of the IO request has been fulfilled. So on our sample drive above a 4KB request is going to read or write 8 adjacent (sequential) blocks. We know what the physical transfer rate for the drive is, but how long does it take to physically move the mechanism so that the 8 blocks we care about will pass under the head? Two things have to happen, first we have to position the head over the right track, and then we have to wait for the right block to pass under the head. This is the combination of "seek time" and "rotational latency". Our 7200RPM drive completes one revolution every (7200RPM / 60 seconds/minute = 120 revolutions/second or every 120th of a second or 0.00833 seconds or 8.33 milliseconds). On average then every IO operation will take 4.16ms to start performing IO after the head are aligned. Again we can reduce the rotational latency by spinning the drive faster. Seek time (how long it takes to align the heads) varies by drive, but if it takes 6ms then the average physical access time for the drive would be 10.15ms. Drives which are physically smaller will have to move the heads shorter distances and will have lower seek times, and therefore lower access times. Larger drives, or drives with heavier head assemblies (more heads) will have higher seek times. For a given drive you can typically look up the manufacture's specs to see what the average seek time is. So, let's say that it takes 10ms to typically position a head and read a block, then our drive could potentially position the head 100 times per second. That means the maximum IOPS for this drive is 100 per second.
So, IOPS is governed by physics, and throughput (from media) is governed by physics. What else is a factor? There are additional latencies introduced by controllers, interfaces, and other electronics. Generally these are fairly small, measured in micro-seconds (us) relative to the latencies we've talked about generally become negligible. The other side is physical interface throughput ATA-133 for instance had a maximum throughput on the channel of 133MB/s; where Ultra-160 SCSI was limited to 160MB/s. The maximum throughput of a given drive will be limited to the throughput of the channel to which it's attached. The old ATA and SCSI interfaces noted earlier also attached multiple devices to a channel which limited the sum of all devices to the bandwidth of the channel. Newer SAS and SATA architectures generally dedicated a channel per device, however the use of expanders serves to connect multiple devices to the same channel. The net of this being that if you have 10 devices at 25MB/sec throughput each connected to a channel with a maximum throughput of 125MB/sec then the maximum throughput you'll see is 125MB/sec.
So that covers what governs the speed of a hard disk. Some may ask "what makes SSD's so fast?" The short answer is that it's because they aren't spinning magnetic devices, and therefore don't have the same physical limits. The long answer is a topic for another blog.
