RTMP can do dynamic streaming, where the video quality automatically adjusts to changes in bandwidth of the viewer's internet connection at that moment. It changes frame by frame, up and down different bitrate streams to provide the best user experience.
Dynamic streaming is the process of efficiently delivering streaming video to users by dynamically switching among different streams of varying quality and size during playback. This provides users with the best possible viewing experience their bandwidth and local computer hardware (CPU) can support. Another major goal of dynamic streaming is to make this process smooth and seamless to users, so that if up-scaling or down-scaling the quality of the stream is necessary, it is a smooth and nearly unnoticeable switch without disrupting the continuous playback.
This article provides an overview of the enhanced capabilities and concepts of dynamic streaming with Adobe Flash Player 10 and Adobe AIR 1.5 with the new Adobe Flash Media Server 3.5 (using either Flash Media Streaming Server 3.5 or Flash Media Interactive Server 3.5). Although dynamic streaming was somewhat possible in previous versions of Flash Player and Flash Media Server, the process was very complicated to implement from a developer's perspective and the end result was not a smooth user experience.
The need for dynamic streaming
With faster-performing client hardware and users with higher bandwidth becoming the norm, the promise of high-definition (HD) video on the web is a reality. HD web video is generally considered larger video starting at 640 × 480 pixel dimensions and increasing up through 720p towards 1080p. The issues facing this trend have been around since the beginning of streaming video. Now Flash Media Server and Flash Player can handle streaming HD video in ways that greatly improve the user's experience without the need for them to do anything besides sit back and enjoy high-quality material.
One of the biggest issues facing publishers trying to stream longer duration (longer than five minutes) and higher quality video—especially HD video—is the standard fluctuations of users' Internet connections. This is a standard issue on most networks and can be exacerbated when multitaskers, wireless network fluctuations, or multiple, simultaneous users sharing a connection are involved.
The end result is a moving target for actual available bandwidth, and this can leave users continually having to rebuffer and wait for their video if the selected stream bandwidth is unsustainable on their network. Dynamic streaming detects fluctuating bandwidth and switches among streams of different bit rates in order to match the content stream to the user's bandwidth (see Figure 1).
Figure 1. Matching bandwidth changes to maintain QoS
On the other hand, some users may start the stream with low available bandwidth, and then free up more bandwidth after the start of the video. In this scenario, dynamic streaming can offer the ability to up-scale the video quality to a higher level, once again improving the user's experience.
In the past, the alternative was to perform initial or frequent bandwidth detection routines. Although better than nothing, these tests were costly in time and often didn't provide the accuracy needed due to the normal fluctuations and changes in bandwidth. Now, with the dynamic streaming capabilities and Quality of Service (QoS) information available, bandwidth detection tests have lost much of their value.
Another issue that can hinder playback, especially with large-dimension HD video and full-screen playback, can be the user's hardware performance limitations. If the CPU cannot decode the video stream fast enough, it will result in dropped frames, which can adversely affect the smoothness of the user's video display. In this case, using a lower-quality video file would enable less strain on the CPU to decode in synch and maintain performance.
Multi-bitrate solutions with dynamic streaming
Dynamic streaming can provide an optimal solution to network fluctuations and CPU overloading. By continually monitoring key QoS metrics on the client, dynamic streaming can effectively identify when to switch up or down to different-quality streams. The Adobe solution does not require specially encoded files for dynamic streaming, which means existing files already encoded in multiple bit rates can be used. With the dynamic streaming support for H.264 video with AAC/AAC+ as well as FLV files, Adobe supports the standard streaming video formats preferred by nearly all major broadcasters.
In addition, Adobe offers excellent content protection with dynamic streaming with the RTMPE encrypted real-time protocol, as well as new support for streaming rights-managed encrypted content with Flash Media Rights Management Server (FMRMS). Another very powerful feature offered by Adobe with FMS 3.5 and Flash Player 10 is the ability to do dynamic streaming with broadcast (live) video. Paired with the updated free Flash Media Live Encoder (FMLE), which now supports multi-bitrate live publishing, Adobe really has a great suite of products to deliver the ultimate user experience.
Table 1 compares the features and functionality of dynamic streaming offered by Adobe in Flash Media Server 3.5 and Flash Player 10 against Microsoft Silverlight 2.0.
Table 1. Dynamic streaming solutions comparison
|Adobe RTMP Dynamic Streaming (Flash Player 10, FMS 3.5)||Adobe HTTP Dynamic Streaming (Flash Player 10.1, AIR 2)||Microsoft Silverlight 4.0 (IIS Media Services 3)|
|File format||Multiple files are used to generate a multi-bitrate experience.
Tweak bit rates at any time.
Standardized formats (H264/AAC, FLV: nonfragmented VP6/H263).
Full support for existing multiple-bitrate–encoded media.
|Standardized formats (VP6/MP3, H264/AAC).
On-demand files are fragmented and use the FMF manifest.
|Single files or multiple files standardized formats (VC-1, H264/AAC Fragmented MP4).|
|Live||Full support of multiple bit rates with live streaming.
All live encoders supporting live Flash streaming will be supported.
|Live streaming with DVR support.
Multiple bitrates supported.
Requires real-time fragmenting and packaging server.
|All encoders that support Live Smooth Streaming are supported.|
|Bit-rate limits||No limits.||No limits.||No limits.|
|Encoding workflow||Support for any media encoder available on the market today
No special encoding requirements.
Adobe Media Encoder CS4.
|On-demand follows normal encoding workflow with an additional post-encoding step to add a manifest.
Live requires real-time fragmenting and packaging.
|No special encoding requirements.
Microsoft Expression Media Encoder 4.
Smooth streaming uses server-side fragmenting
|Content protection||Real Time Encryption using RTMPE.
Encrypted files (Flash Access 2).
|Encrypted files (Flash Access 2) for both live and on-demand.
No over-the-wire encryption.
|Play Ready support (on-demand and live).
No over-the-wire encryption.
Stream Encryption using SSL.
No media player-based protection.
|QoS hueristics||Developer has complete control when to switch and how often.
Helps control costs.
|Developer has complete control when to switch and how often.
OSMF adjusts automatically if desired.
Controlled through a configuration file.
No claims of developer configuration.
|CDN||Over 20 CDNs worldwide are participating in Adobe's beta programs for early adoption to support dynamic streaming.||All support it if you package the file and deliver via the CDN through your origin.||About the same.|
|Audio||Full support for multi-bitrate audio.||Support for MP4 multi-bitrate audio.||One bit rate only.|
|Media framework||Open Source Media Framework (OSMF).||Open Source Media Framework (OSMF).||Silverlight Media Framework (SMF).|
How dynamic streaming works
Dynamic streaming is made possible by leveraging the new Flash Media Server 3.5 with Flash Player 10 and AIR 1.5. It is available on all server versions, including Flash Media Interactive Server, Flash Media Streaming Server, and Flash Media Development Server, and does not require any special server setup.
Flash Media Server handles the actual switching of the streams for the user based on the client-originated request to do so. Once the server receives the request to switch the user's stream to a different stream (bandwidth), it will wait a short period for the optimal point to switch with minimal playback impact to the user. This occurs on the nearest keyframe for video-only content, the nearest audio sample for audio-only streams, and the combination of the two for video content with audio. This means that video encoded with a keyframe every six seconds could take up to six seconds, once the command to switch streams has been received by the server, to actually switch the content and send the transition-complete notification.
This is also important in considering the available buffer amount. You need to make sure there is enough buffer to cover the transition command sequence, since the buffer will not continue to fill for the old stream once the switch transition request has been received by the server. The client-side application is notified at the key point of the dynamic streaming transition sequence. After the call to the server is received and is beginning to be processed, a NetStatusEvent will be fired with an info Object and a code property (event.info.code) with the value NetStream.Play.Transition. This indicates the server has received the transition command and is awaiting the next keyframe match to switch streams.
An undocumented valuable property is the reason property, also on the info Object of the NetStatusEvent instance (event.info.reason). The reason property provides additional information about the state of the transition request. Normally, this property will contain the value NetStream.Transition.Success, indicating that the transition request is processing normally. If streams that are being switched do not have aligned content/timelines, or if the keyframes are not aligned between the two streams, it is possible that the server will have to force a hard transition. This can also happen with broadcast (live) dynamic streaming if the server cannot find a synchronization point within the two streams. If this happens, the reason property will have the value NetStream.Transition.Forced. This situation can occur under the following circumstances:
- The two streams being switched don't have the same timeline, and therefore the server cannot select a time to perform the switch.
- The keyframe interval of the new stream is longer than the client's playback buffer, which is the maximum amount of time the server will wait for a transition. Since the server did not see a keyframe, it cannot select a frame for the switch.
- The live queue delay for the live streams is longer than the client's playback buffer, which creates a delay similar to a long keyframe interval.
Once the streams have successfully been switched and the new stream is being played by the client application, a play status event is fired and the onPlayStatus() handler of the NetStream client Object is invoked. Its single parameter is a generic Object and will have a code property with a value of NetStream.Play.TransitionComplete, indicating the transition is complete.
Client applications can be run from Flash Player 10 (and later) or AIR 1.5 (and later). Flash Player 9,0,115 achieved over 85% adoption in under six months. With this amazing trend to adoption, Flash Player 10 should be available for more than 85% of the entire worldwide Internet user base within six months. The industry-standard applications that create these rich experiences are Adobe Flash Professional CS4 and Adobe Flex Builder 3, or you can use the Adobe Flex 3 SDK. Within the new development tools, the new ActionScript-based API will provide you with the power both to monitor the performance of streams to determine the need and direction for stream switching, and to invoke the stream switch.
The new client-side ActionScript APIs includes a new info property on the existing NetStream class. When the NetStream.info property is accessed within a NetStream instance, it returns a snapshot of all the available QoS data points and returns them in a NetStreamInfo object (see NetStreamInfo in the ActionScript 3.0 Language and Components Reference). This class contains 19 properties, each providing valuable key points of information regarding the stream connection and playback performance. The wide spectrum of data the NetStreamInfo class provides gives granular real-time metrics about the stream performance, including audio, video, and data streams, both independently and as a whole.
Two of the most important properties it provides are NetStreamInfo.maxBytesPerSecond and NetStreamInfo.droppedFrames. The NetStreamInfo.maxBytesPerSecond property is one of the simplest and most direct ways to see the maximum capacity of the client's network for the stream. NetStreamInfo.droppedFrames indicates the total number of dropped frames to the point of the stream when NetStream.info was accessed. To monitor this property properly, you should likely be polled at a set interval and store the previous count of dropped frames and the target frame rate. If the percent of dropped frames is greater than 20%, you should consider swapping streams if managing manually.
NetStreamInfo public properties
Here are the 19 public properties exposed in the NetStreamInfo object:
- audioBufferByteLength: Number (read-only): Provides the NetStream audio buffer size in bytes.
- audioBufferLength: Number (read-only): Provides NetStream audio buffer size in seconds.
- audioByteCount: Number (read-only): Specifies the total number of audio bytes that have arrived in the queue, regardless of how many have been played or flushed.
- audioBytesPerSecond: Number (read-only): Specifies the rate at which the NetStream audio buffer is filled in bytes per second.
- audioLossRate: Number (read-only): Specifies the audio loss for the NetStream session.
- byteCount: Number (read-only): Specifies the total number of bytes that have arrived into the queue, regardless of how many have been played or flushed.
- currentBytesPerSecond: Number (read-only): Specifies the rate at which the NetStream buffer is filled in bytes per second.
- dataBufferByteLength: Number (read-only): Provides the NetStream data buffer size in bytes.
- dataBufferLength: Number (read-only): Provides the NetStream data buffer size in seconds.
- dataByteCount: Number (read-only): Specifies the total number of bytes of data messages that have arrived in the queue, regardless of how many have been played or flushed.
- dataBytesPerSecond: Number (read-only): Specifies the rate at which the NetStream data buffer is filled in bytes per second.
- droppedFrames: Number (read-only): Returns the number of video frames dropped in the current NetStream playback session.
- maxBytesPerSecond: Number (read-only): Specifies the maximum rate at which the NetStream buffer is filled in bytes per second.
- playbackBytesPerSecond: Number (read-only): Returns the stream playback rate in bytes per second.
- SRTT: Number (read-only): Specifies the Smooth Round Trip Time for the NetStream session.
- videoBufferByteLength: Number (read-only): Provides the NetStream video buffer size in bytes.
- videoBufferLength: Number (read-only): Provides the NetStream video buffer size in seconds.
- videoByteCount: Number (read-only): Specifies the total number of video bytes that have arrived in the queue, regardless of how many have been played or flushed.
- videoBytesPerSecond: Number (read-only): Specifies the rate at which the NetStream video buffer is filled in bytes per second.
Adobe is providing some powerful resources to implement dynamic streaming into your own applications. Most notable are the new DynamicStream and DynamicStreamItem classes. You can download these from the new Flash Media Server Tools page. The classes provide a powerful, clean, and quick way to implement the dynamic streaming functionality without all the heavy lifting of building a custom solution from the ground up. The DynamicStream and DynamicStreamItem classes leverage the key QoS data available from the NetStream.info property and provide a direct and easy-to-use API to stream with multiple bit rates to a video player with all the control a developer woulddesire.
The DynamicStream class extends the NetStream class and is used as an alternative to the NetStream in custom video applications. It also offers powerful benefits, such as integrated quick-start playback, easy implementation, and enhanced control methods. To use the DynamicStream class, you generate an instance of it, passing in a reference to the NetConnection instance you want to communicate through, just as you would with a normal NetStream. The DynamicStream instance has a method called startPlay() to which you can pass an Array of streams encoded at different bit rates in the form of a DynamicStreamItem instance.
Dynamic streaming video players
The versatile FLVPlayback component has been updated in Flash CS4 Professional to FLVPlayback 2.5, which encapsulates the DynamicStream classes and offers ease of use and additional API enhancements for dynamic streaming. You can download this now from the Flash Media Server Tools page. Full backwards compatibility has been maintained, but now there is a play2() method available for passing a DynamicStreamItem containing the list of streams with different bit rates as well as a uri property to indicate the connection point.
Dynamic streaming is a powerful new functionality powered by Flash Media Server 3.5 and Flash Player 10. With this new and enhanced capability, there is now an effective, simple way to improve the user experience with video streaming. The problems of continual rebuffering and interrupted viewing experience now have a solid solution that can handle adverse situations and improve the overall standards for the user experience.
Dynamic streaming also opens the door for the continual progression towards HD video on the web and maintains solid position of Flash as the leader of video on the web. The ease of implementation and deep level of control that the new API and dynamic stream classes that Adobe offers have made this a quick and cost-effective tool set to implement.