Concept vs. Reality

The "information superhighway" or "digital highway" concept has garnered popular attention and press coverage over the past year largely due to the mental imagery those catchy phrases create. For the general population it is easy to imagine a broad, open, eight-lane freeway, with "blocks" of different types of data (like video and graphics) moving at high speed like buses, trucks and cars to various locations all over the world.

Unfortunately, while much has been made of the concept in the computing world, there has been little explanation of how it will actually be implemented. The technical solution used will not only have to be flexible and inexpensive, it will also require high performance characteristics to support extremely high data transfer rates. One solution that addresses those issues head on, but has not been widely publicized, is a relatively new technology called Fibre Channel.

Fibre Channel is an industry-standard interface adopted by the American National Standards Institute. It is usually thought of as a system-to-system or system-to-subsystem interconnection architecture that uses optical cable between systems in a point-to-point (or switch) configuration. Certainly, this is how it was first envisioned, and among the many protocols defined for it are IPI (Intelligent Peripheral Interface) and IP (Internet Protocol) which are ideal in those configurations.

Fibre Channel has since evolved to include electronic (non-optical) implementations and the ability to connect many devices to a host port--including disc drives--in a relatively low-cost manner. This addition to the greater set of Fibre Channel specifications is called Fibre Channel - Arbitrated Loop (FC-AL). FA-CL has made it possible for Fibre Channel to be used as a direct disc attachment interface, opening whole new levels of I/O performance up to designers of high-throughput, performance-intensive systems. SCSI-3 (Small Computer Systems Interface-3) has been defined as the disc protocol, which is also technically refered to as the SCSI-FCP (Fibre Channel Protocol) for FC-AL.

Fibre Channel for Disc Storage

Fibre Channel has been viewed as too expensive and power-hungry for lower-level functions like peripheral attachment, raising important questions as to why it should be used as a peripheral interface. What is Fibre Channel-Arbitrated Loop? Why use it? How can it be implemented? It is these questions that we will examine in detail.

What is Fibre Channel-Arbitrated Loop?

The Fibre Channel interface is a loop architecture as opposed to being a bus-like standard SCSI or IPI. The Fibre Channel loop can have any combination of hosts and discs up to a maximum of 126 devices.

Loop architecture

Note: Although the figure above suggests that the drives are connected by a cable running from device to device, the preferred approach is to attach disc drives directly to a backplane. This eliminates cable congestion, makes hot plugging practical and simplifies mechanical designs. In fact, because the Fibre Channel drives use a shortened version of the SCA connector used on parallel SCSI drives, they can easily be designed into existing SCA drive cabinets.

The loop structure enables the rapid exchange of data from device to device. A PBC (Port Bypass Circuit), which is located on the backplane, is the logic that enables devices to be removed or inserted without disrupting the operation of the loop. In addition, the PBC logic can take drives off-line or bring them back on-line by sending a command to any device to remove it from loop operation or reinstall it onto the loop.

(Fibre channel loops with backplane detail.)The figure above shows the use of a backplane with dual-ported drives and dual loops. This would be typical of a disc array subsystem with hot insertion capability. The diagram also shows the PBC chips contained on the backplane. The drives attach directly to the backplane for both signal and power. Neither jumpers nor switches need be set on the drive; it simply plugs into the backplane. The controller determines a drive's address from either the relative position on the backplane or the drives unique IEEE (Institute of Electronic and Electrical Engineers) Fibre Channel address; each Fibre Channel interface device has a unique world-wide address.

A summary of the features of FC-AL is listed below:

Why use Fibre Channel?

SCSI has become the interface of choice for medium- and large-sized computing systems. At the same time, the computer industry's experience with SCSI has brought to light the need for improvements in cable distance, number of variations, command overhead, array feature support and connectability.

Cable Distance Limitations SCSI comes in several different versions, but the majority of products shipping today are of the single-ended variety. (That is primarily because single-ended SCSIs cost less and are widely available.) If a SCSI bus is limited to connecting devices found within a single cabinet and the interface cable length does not exceed three meters, it usually is not a problem to use single-ended SCSI. If the bus must link several cabinets, however, and in the course of doing so must convert from an unshielded ribbon cable in one cabinet to a shielded one externally, then back again to a ribbon cable within the second cabinet, the potential for SCSI signal problems increases. Differential SCSI solves this cabling issue, but usually requires that a system have both a single-ended and differential ports because most non-disc peripherals use only single-ended SCSI (single-ended SCSI devices cannot be attached to a differential bus).

SCSI Variations

The previous example hints at the problem created by the availability of multiple versions of the SCSI interface. Here is a list of the SCSI versions supported by a single Seagate product (the Barracuda 2LP, ST32550):

For many companies building computers, this creates a problem of logistical complexity. A systems manufacturer might have to buy two or three versions of the same drive simply because various system models require different flavors of the SCSI. Costs can get high when a company has to account for inventory, sparing, qualification and testing for multiple versions. Such costs could be avoided if a single version of the interface were used.

Excessive command overhead.

SCSI's utility has improved with increased data rates several times since its introduction. Unfortunately, only the data rate has improved, which is the rate at which data is transfered off the disc. The rate at which SCSI interface information passes over a SCSI bus has not changed at all. SCSI command overhead is taking up an increasingly larger proportion of bus time, which makes it difficult to support multiple drives before the bus is saturated. Often peak transfer rate is achieved with three drives per each eight-bit bus, because three drives kept busy will produce so much SCSI protocol traffic that no additional drives can get any practical bandwidth that would contribute to the aggregate data rate.

Higher Data Rates

Magnetic hard disc areal density is increasing at about 60% per year. Since bit density (measured in bits per inch), one of the two components of areal density, increases at about 30% per year, data rate automatically increases proportionately. Disc rotation speeds, which have doubled over the last five years from 3,600 to 7,200 as they continue to climb, also contribute to higher data rates. In the next few years drives will be introduced which can sustain transfers in excess of 20 MB/s, thanks solely to improvements in bit density and rotational speed. Already there are disc drives in the market which can transfer data at rates over 10 MB/s. An interface limited to 20 MB/s can only support one drive at that data rate, making it technically impractical for future applications.

Arrays and Hot Plugging

The increased use of SCSI discs in arrays and fault-tolerant configurations has revealed further limitations and problems of SCSI, particularly when devices must be inserted or removed from the interface without disrupting operations. It has taken considerable ingenuity to get SCSI to run in the presence of the glitches caused by insertions and removals.

Limited Connectibility

New applications, like video and image processing have created a demand for huge increases in storage capacity. Some capacity requirements are so large that it is difficult to configure enough SCSI buses to make sufficient drive addresses available to attach the needed number of drives. Simply increasing the addressability of SCSI, such as making it possible to have more than 15 devices per SCSI Wide bus, would not be a solution because more bus bandwidth is needed to support the additional drives.

How Does Fibre Channel Stack Up?

A disc interface based on the Fibre Channel standard can resolve each of these problems and provide functionality that has only been dreamed about up to now. As the following chart demonstrates, Fibre Channel resolves the limitations that arise with SCSI:

Fibre Channel also has benefits that go beyond current disc drive interface solutions. Fibre Channel provides significantly higher bandwidth, superior drive array features and improved network storage capabilities than conventional interface technologies.

Bandwidth

Fibre Channel loop supports data rates up to 100 MB/s (megabytes per second). Video storage and retrieval, supercomputer modeling, and image processing are among the applications growing in popularity that demand this kind of data rate. Moreover, as file servers are looked upon as replacements for mainframe computers, they will require ever-higher transaction rates to provide comparable levels of service. Since most UNIX and Windows servers lack the sophisticated I/O channel and controller structures of mainframe computers, they have not been able to match the large number of high-performance disc drives enterprise systems can support. Fibre Channel loops attached to such high-performance buses as S-Bus, Turbochannel or PCI-- all of which run 70 MB/s or faster--offer I/O configurations that can sustain mainframe-like I/O rates. Performance estimates suggest that if a system requests the relatively short I/O transfers typical of business transaction processing (8K or less), more than 60 drives can be supported without saturating the loop and bogging down performance. Comparing the single host adapter to the many channels and controllers mainframes employ to attach as many drives illustrates the remarkable economics of Fibre Channel-attached disc storage.

The chart above estimates the capacity of a Fibre Channel interface to support large numbers of drives in a transaction environment. With up to 9 GB per disc available 63 drives on a loop would make over 570 GB available to a user on a single FC-AL host port. It will be possible for any workstation or system that has a single FC port or backplane slot to become a file server. No longer will the large rack-mounted systems characterize file servers. With FC they can be as small and unobtrusive as any office system.

Array Improvements

Array controllers have traditionally been architected with multiple standard SCSI interfaces for drive attachment, which enables the controller to supply data and I/O rates equal to several times those achievable from a single interface. Designing a specific number of drive interfaces into a given controller, however, has forced the customer to deal with the parity amortization, granularity and controller cost associated with that decision. It severely limits the designer's choices for configuring the optimal combination for economy- that is, maximizing the number of data drives per parity drive, granularity - minimizing the atomic unit of capacity per array, and performance. A fully populated controller is in danger of bogging down under the burden of supporting the level of I/O activity that several rows of drives could generate.

SCSI drive array

For example, an array that has six rows amortizes the parity data over five data drives, and usually requires adding six drives each time any increase in capacity is required.Using Fibre Channel it is possible for the first time to have more than enough bandwidth on a single bus to configure an array along a single interface (the array shown uses two loops to take advantage of FC dual porting for better performance and reliability, but it could as easily be constructed with one FC loop if full fault tolerance is not called for). Seagate Fibre Channel drives also have an exclusive XOR logic engine which allows easy implementation of RAID 5, thereby avoiding the high costs associated with traditional RAIDs.

Fibre Channel longitudinal array

Some of the more important benefits from using FC as the basis of a longitudinal array include:

1. The customer can decide the economy of the array. He can use five, eight, 18 or 24 data drives per parity drive, with no additional controller cost.
2. The granularity is a single drive. The customer need not add a whole row of drives as on a traditional array; he can increase the capacity of his subsystem by just the amount he requires.
3. Performance. Because the controller has been significantly simplified, it is much less expensive than a multiple drive interface version. The customer can add controllers - and spread his drives across them - as he needs more performance, but only add drives if he needs extra capacity.

With Fibre Channel provides the customer with complete control over the dimensions of economy and performance.

Remote On-Line Storage

Since the Fibre Channel interface is part of (and fully compatible with) the Fibre Channel standard, optical cabling can be used in any part of a subsystem, excluding the backplane. This makes it possible to have a disc subsystem quite a distance from the computer system to which it is attached. Using single-mode fibre optics, on-line disc storage could be as far as 10 km away.

Fibre Channel with optical transmissionIn the case above the computer system on the left could have discs within the system unit attached via the FC loop. The loop can be extended by using an electronic-to-optical adapter and lengths of fibre-optic cable. On the same loop running the internal discs, remote discs would appear to the system to be directly attached exactly like the local discs. These, then, could be used in any fashion, including shadowing of the local discs. It makes a very attractive means for having a remote on-line copy of critical data, which could be used to continue operations should anything happen to the original computer system.

LAN Compatibility

FC is a generic, standard interface. It has multiple uses and supports many different protocols, such as:

• SCSI
• IPI-3 Disc & Tape
• Link Encapsulation
• Internet Protocol
• ATM

All of these can run on the same FC facility! In fact, some of the first FC disc drive host adapters will support both SCSI and networking protocol. It makes for a very attractive investment: install a network loop and get 100 MB/s disc channel for free, or install a 100 MB/s disc loop and get a 100 MB/s network interface for free. Today's common discussions about 10 Mb/s or 100 Mb/s limitations of Ethernet seem trivial in comparison to the reality of a network that runs at mainframe backplane speeds.

Network Attached Storage

While there is, as yet, no software support for it, Network Attached Storage offers intriguing possibilities for the future and Fibre Channel makes it architecturally practical for the first time.

Wide area network In a typical non-fibre channel network the storage devices would be attached to a file server which would service the data needs of all other attached systems. The wide area network in diagram shows an FC loop with discs and systems attached. The systems could be communicating with the discs using SCSI and with each other using IP!. Since the discs are directly attached to the same network as the systems there is no reason why all the systems could not communicate with the discs directly! All of the systems in the network could access the data at Fibre Channel speeds and avoid the file server bottleneck. These features would be very beneficial in a video server application where the data could be sent directly from the drive to a cable head or satellite at speeds much faster than any file server could move the data.

How can it be done?

The complexity of the Fibre Channel standard, since it covers so many and because it is so fast, may lead to the assumption that any implementation would be extremely expensive. The fact is several recent developments have made it very economical.

Protocol-Specific Devices

Putting Fibre Channel on a disc drive will not be significantly more expensive than SCSI because the interface can be tailored to specifically run the SCSI FCP protocol. Using the SCSI protocol eliminates much of the complexity that would come with a comprehensive Fibre Channel implementation. In fact the logic associated with Fibre Channel is only marginally more complex than today's SCSI disc drives.

1-GHz Transmission Speeds

Optical cable is usually thought to be the most promising vehicle for this kind of frequency. Recent developments in semiconductor chips have resulted in inexpensive 1 GHz transmitters and receivers. These chips include all the logic needed to connect inexpensive CMOS digital logic needed for the rest of the Fibre Channel interface to a 1 GHz serial link.

High-Speed Memory

The memory speed needed to support 100 MB/s transfers concurrently with 20+ MB/s disc media transfer would have been a problem several years ago. The introduction of Cache DRAMS has made it possible to implement full 100 MB/s data/cache buffers for the same cost as supporting fast/wide SCSI.

Host adapters and drives

Several companies have announced their intention to supply products with a Fibre Channel interface. By the end of 1994 the first host adapters should be appear and drives will be available from Seagate and other suppliers. Fibre Channel testers will also be available from Peer Protocols and I-Tech.

A Fibre Channel Future

The elements necessary to build a Fibre Channel-based system will be available by the end of 1994. By the middle of 1995 end-users should be able to purchase and configure Fibre Channel disc subsystems. Early introductions of Fibre Channel will occur on mainframes, supercomputers, workstations, and workstation-class file servers. Gradually, the technology will migrate to Novell-class file servers and eventually personal workstations. Each step in the migration path will draw closer to the technology infrastructure necessary to support the "information superhighway."