VelocityQ and Storage Area Networks (SAN)

 

Review contents:

Data storage systems for TV Application technology of the uniform disk arrays from Avid Technology Decisions of Ehance Tech company SANStream from Huge Systems DriveBox from JC System Integration VelocityQ and Storage Area Networks (SAN) Network decisions from Medea Palladium architecture SAN/NAS decisions from Raidtec Network storage decisions of Sanbolic Company Storage in the broadcast environment

Introduction

Storage Area Networks, or SANs, are a powerful collaborative architecture for facilities with multiple non-linear editing workstations. This article provides an introductory overview of Storage Area Network (SAN) architecture, and the considerations in the use of SANs with Leitch’s award-winning VelocityQ multi-stream non-linear editing system. This overview is not intended to comprehensively cover all technical details and considerations of SAN technology; some sections have been intentionally simplified, omitting some low-level details to offer a “high level” presentation of the general concepts for those unfamiliar with SANs.

What is a SAN?

A SAN, short for Storage Area Network, is essentially a dedicated, specialized network connecting multiple workstations to a shared storage pool for high-performance data transfer. Through a combination of SAN hardware and software, all of the workstations can simultaneously access the shared storage pool, using the media in the centralized storage much like if it was on the workstation’s local hard drive. Many SAN configurations also provide file-sharing management that allows multiple users to access the same file at the same time whenever possible. SANs tend to be the shared storage architecture of choice for applications in which predictable response time, availability and high performance are essential.

As opposed to file transfer over IP-based networks (such as a LAN), a SAN provides significantly greater performance in a predictable amount of time. (Accessing files over a LAN provides no guaranteed level of performance, nor predictable delivery time, which makes this method ill-suited for video). Rather than doing file transfers, the shared storage on a SAN appears to the workstation just as if it was a local hard drive connected directly to the workstation, reading and writing data seamlessly. SANs typically still use an IP-based LAN, but primarily for transferring metadata, file access requests, and very small auxiliary files. The actual media files are transferred over a higher-performance connection such as Fibre Channel.

Velocity editing systems within SAN architecture

In the context of VelocityQ, a SAN serves as a large shared media pool accessible to multiple VelocityQ editing systems at the same time. Multiple VelocityQ systems can be combined with an SAN to form a powerful collaborative editing environment in which the SAN provides a common centralized storage pool. Benefits of this shared architecture include the ability for multiple editors to access the same raw footage to create different variations of projects simultaneously, for multiple people to be working on different parts of the same project in parallel, and for the shared storage to serve as a central repository for media assets, providing easy access and availability.

While SAN interfaces can be based on a number of different protocols, the Fibre Channel protocol is by far the most popular implementation currently. The Fibre Channel protocol provides a high degree of predictability and performance, and is well-suited to the high data-throughput requirements of video applications over shared storage. (Note the particular spelling of Fibre, and that its name is not related to the physical connection layer — the physical connections for Fibre Channel SAN implementation can actually be optical or copper). All currently certified SAN solutions for VelocityQ, including configurations featuring ImageSAN from Rorke Data and MelioFS from Sanbolic, use the Fibre Channel protocol.

VelocityQ and SAN

Maximum Number of SAN Seats

The maximum number of VelocityQ systems that can be running at optimal real-time performance from shared storage on a SAN is dependent on the SAN implementation, and is not constrained by VelocityQ. VelocityQ itself imposes no inherent limits on the number of SAN seats it can work with; the limiting factors include the available sustained data rates, seek times and other characteristics of the overall SAN implementation.

At a basic level, this number of VelocityQ systems is constrained by (among other factors) a function of the number of real-time streams being used on each system, the recording and playback data rates being used on each system, and the total aggregate data rate available through the SAN. For example, a SAN configuration that is able to handle four systems, each running two real-time streams of 10MB/sec each, might be able to handle only two (half as many) VelocityQ systems that are each running four real-time streams (twice as many) of the same data rate.

In comparing VelocityQ’s possible SAN configurations with that of NLE systems with lower real-time performance, the initial impression may be that VelocityQ is limited to fewer simultaneous SAN seats, or higher-priced SAN configurations. Again, it’s important to recognize that this is because of VelocityQ’s four-stream real-time performance. Dual-stream systems, for example, won’t consume as much SAN data throughput, as they each play no more than two real-time video streams at a time through the storage. And of course, non-linear editing solutions that are not real-time (ie. full-quality playback first requires rendering, or background rendering) may put even less demands on the SAN. A similar number of SAN seats could be achieved with VelocityQ systems by using only a portion of VelocityQ’s real-time streams, but in doing so, the user would be sacrificing significant real-time performance to achieve more SAN seats — defeating VelocityQ’s tremendous real-time performance advantage.

Structure scheme of a storage system for VelocityQ

While full-quality video streams consume the greatest portion of SAN bandwidth, they might not be alone. If the corresponding audio, graphics, and project files are stored on the same SAN, then even more data throughput is required of the SAN. (While the size and data throughput of audio and graphics files is very small relative to that of video, it still all adds up!)

While this may seem straightforward (and possibly even somewhat obvious), this is of course over-simplified. Beyond the total required data throughput, number of other factors affect the maximum number of seats of VelocityQ that work with a given SAN configuration, including the time required for the hard drive spindles to “seek” to the required locations on the disks, the “switching speed” and caching characteristics of the Fibre Channel switch, the performance of the SAN management software or file system, and more. For example, a SAN that supports three VelocityQ systems running clips at 14MB/sec might not necessarily support six VelocityQ systems running clips at 7MB/sec. While the total data throughput would stay constant, the shared storage array’s drive spindles now would be accessing twice as many separate clips, “seeking” much more across the drive platters and dramatically increasing the “seek time” to each clip. Unfortunately, there is no simple mathematical formula to determine how many systems will work comfortably on a SAN, as the “bandwidth loss” to factors such as “seek time”, network overhead, and caching and more aren’t easily calculable. For this reason, it’s important not to make any performance assumptions about particular configurations unless they’ve been specifically tested and certified.

Real-Time Performance Considerations

Based on the above, it’s clear that in choosing a SAN configuration that will provide optimal real-time performance for VelocityQ, you will need to determine your total data throughput requirements (the maximum data rates you’ll be using, times 4 real-time video streams per system, times the number of systems…plus additional throughput for audio, graphics, and project files).

The discussion so far, however, has focused on the maximum possible data throughput you may require. In practice, this could be quite different from the average or typical data throughput you require. During the editing process, it may be rare for all editing workstations to be playing back four streams each at the same time. It’s more likely that some systems will be playing back just one or two streams, while others play back as many as four. Furthermore, VelocityQ’s dynamic compression during recording means that the data rate specified during capture is a maximum limit; actual captured clips may use a lower data rate, though, if the full maximum date rate isn’t needed. (For example, capturing low-entropy footage, such as pure full-field black, won’t use a full 14 MB/sec, even if that’s specified as the capture limit).

As such, while the number of VelocityQ seats possible on a SAN configuration may be limited to a particular number assuming maximum data throughput requirements, you may be able to have more seats running perfectly well on the SAN under normal operating conditions. This does take the risk of real-time performance being affected under extreme usage, and whether that’s problematic for you will depend on your workflow.

If maintaining guaranteed real-time performance across all systems is important to your workflow, then base your number of SAN seats on your maximum possible data throughput requirements, not the average or typical data throughput you expect. Because all of the systems on the SAN are sharing the data throughput, if the SAN is chosen based on your average data rates, and one (or more) of the editing workstations then uses a higher data rate, the overall maximum data throughput of the SAN may be exceeded. If this occurs, it could affect the performance to all workstations on the SAN, resulting on dropped frames on video playback. Thus, activities elsewhere on the SAN by other users may impact upon the data throughput a particular workstation receives.

VelocityQ editing system interface

If the risk of such occasional performance interruptions is acceptable in your workflow, you may instead be able to base your number of SAN seats on the average or typical data throughput you expect. Some SAN management software may allow the SAN administrator to set guaranteed minimum data rates for specified workstations to ensure that critical workstations would not be affected in situations exceeding the maximum data throughput of the SAN (thus, the other non-guaranteed workstations would be affected even more dramatically), or restrict the maximum data rate available to particular systems (in an attempt to avoid this situation from occurring at all). But if such administration settings are not available in the SAN solution (or available, but not used), the shared nature of the SAN may make it impossible to guarantee real-time performance on all workstations. This can be particularly problematic when doing a final print-to-tape for delivery to a client. If this is a concern, you can use the Localize function of VelocityQ to transfer the relevant media onto the editing station’s local storage for print-to-tape, thus isolating it from other SAN traffic. (Alternately, you can just ask the other users on the SAN to limit their activities until the print-to-tape is complete).

Conclusion

The combination of VelocityQ with a Storage Area Network solution provides a powerful collaborative editing environment with a common centralized storage pool. In choosing the ideal SAN configuration for your environment, there are a significant number of variables to consider, and hopefully this article has helped by giving you some points to consider in making your choice.

Material provided by Leitch Company