An example from
Configuration and Capacity Planning for Solaris Servers (Wong) shows that a dual-processor SPARCstation 20/712 with four Ethernets and six 2.1 GB disks will spend all its time waiting for the disks to return some data. If it was loaded with disks (Brian Wong suggests as many as 34 of them), it would still be held below 1,200KB/s by the Ethernet cards. To get the performance the machine is capable of, we would need to configure multiple Ethernets, 100 Mbps Fast Ethernet, or 155 Mbps FDDI.
The progression you'd work through to get that conclusion looks something like
Table B.5.
Table B.5: Tuning a Medium-Sized Server
Machine |
Disk Throughput |
CPU Throughput |
Network Throughput |
Actual Throughput |
Dual SPARC 10, 1 disk |
560
|
6000
|
1,113
|
560
|
Add 5 more disks |
3,360
|
6000
|
1,113
|
1,113
|
Add 3 more Ethernets |
3,360
|
16000
|
4,452
|
3,360
|
Change to using a 20-disk array |
11,200
|
6000
|
4,452
|
4,452
|
Use dual 100 Mbps ether |
11,200
|
6000
|
13,000
|
11,200
|
Initially, the bottleneck is the disk with only 560 MB/s of throughput available. Our solution is to add five more disks. This gives us more throughput on the disks than on the Ethernet, so then the Ethernet becomes the problem. Consequently, as we continue to expand, we go back and forth several times between these two. As you add disks, CPUs, and network cards, the bottleneck moves. Essentially, the strategy is to add more equipment to try to avoid each bottleneck until you reach your target performance, or (unfortunately) you either can't add any more or run out of money.
Our experience bears out this kind of calculation; a large SPARC 10 file server that one author maintained was quite capable of saturating an Ethernet plus about a third of an FDDI ring when using two processors. It did nearly as well with a single processor, albeit with a fast operating system and judicious over-optimization.
The same process applies to other brands of purpose-designed servers. We found the same rules applied to DECstation 2100s as to the newest Alphas or Compaqs, old MIPS 3350s and new SGI O2s. In general, a machine offering multi-CPU server configurations will have enough bus bandwidth and CPU power to reliably bottleneck on hard disk I/O when doing file service. As one would hope, considering the cost!