Brandon.ToString()

It’s been a little while since my last blog entry, mostly because of being incredibly busy, and busy on a lot of projects that I can’t share. But today, online streaming is a topic of discussion that’s been burning our ears on a number of fronts, and this post is dedicated to taking an honest look at where we’re at, where things are going, and what has to change along the way. Just in this past week alone, I’ve heard of and participated in some successes, failures, and setbacks for a number of clients.

Some Background

For years now, the team at X9 has supported a reasonably-sized media delivery infrastructure comprised of Windows Media, Flash, and most recently, IIS Smooth Streaming. Last year alone we served up hundreds of thousands of hours of content, and a little over 10,000 video assets passed through our “doors” to be delivered across the internet. We’ve been around this block a few times, and a new/major twist in the technology has been coming around every 2 to 3 years.

There are 2 common things I’ve observed with nearly every technology available today:

• Generally designed for on-demand delivery, with live broadcasts as a second-rate citizen
• Critically lacking in the area of encoding, distribution, or client-side playback

If only there were a way to clearly see which technologies in the “big 3” were performing well, like a…

Streaming Technology Report Card

Bam! Done.

Now, mind you: This is the official report card from me and only me, using my experience in these technologies as it relates to the small and mid-sized streaming market, using features that are only important to me as we have experienced them as a company. These scores also do not take specialized hardware into account; this is streaming on hardware that normal people and normal companies have and can afford, so there are no $30,000-per-channel encoding devices or massive render farms in this equation.

You may also notice that there is no “summary” or “winning technology” – that’s not the point here; rather, each technology suffers different shortcomings across its usage path through the encoding / distribution / client-playback process – only winners in each feature category.

I have excluded some other technologies that we do support (for example, Shoutcast, QuickTime) in favor of focusing on the big 3 for video delivery.

Now, let’s go over some observations on the overall media delivery path.

Encoding: Nothing Exciting Here.

Overall, the encoding process across technologies is…. meh, okay. Classic Windows Media has been around the longest and, by far, has the largest market penetration. You can encode WMV in any number of tools and nearly every video editing package on Earth. It’s performance is okay. The Achilles heel for Windows Media is the quality. It takes a large, large amount of bandwidth and video size to get to the smooth, professional look that Flash and Silverlight/IIS Smooth Streaming offer with H.264. Nevertheless, the majority of our assets are still Windows Media, because of the complexities and shortcomings of Silverlight/IIS Smooth Streaming that we’re still working through.

Flash’s tooling and integration with video editors and automated encoding libraries is hit and miss. There isn’t really a ubiquitous “go-to” solution for encoding and publishing Flash video. Moreover, the market is littered with third-party utilities of poor quality that makes you want to just use YouTube instead.

One of the best – if a bit complicated – workflows comes from Silverlight Smooth Streaming with encoding done by Expression Encoder. However, since Adaptive Bitrate streaming requires encoding a stream up to 8 times, it’s extremely resource-intensive to pull off this format.

There have also been some great advancements here recently from the Smooth Streaming team (batch encoding Transform Manager, for example).

By Next Year (Encoding):

There’s not too much going horribly wrong in the Encoding areas of all 3 technologies, but there is some room for improvement. Windows Media is still a stronghold because of the encoding and distribution problems of Silverlight/IIS Smooth Streaming. That’s a sad place for a technology to be; however, the worst shortcomings in Windows Media (quality and client experience) have already been addressed in Smooth Streaming, so it’s time for WM to fade off into the sunset… if Smooth Streaming can fill in some gaps in Distribution to get us there.

Flash – we need some better tooling for on-demand.

Distribution: Joy and Pain, all By Design

If there’s one area to improve upon in the streaming industry, distribution is it.

The scores here really reflect the different personalities of each technology, and their makers. I’d like to hit on each point in depth here:

Encoder Upstream Bandwidth Required (applies to Live streams only)

Both Windows Media and Flash are pretty much the same here. You want to deliver a 350kbps stream; you’ll need 350kbps. However, what makes Smooth Streaming great really kills it for live remote broadcasts. By the time you encode a stream a few times, you have to have a good, solid high-speed upstream connection at minimum to even be in the game. This is a problem for a lot of locations here in the U.S., with upstream bandwidth being prohibitively expensive outside of datacenters.

By Next Year: The model of intense encoding resources and high bandwidth at the encoder doesn’t make sense for Smooth Streaming. In a year, I’d like to see Smooth Streaming at a point where we can send one high-quality broadcast to the datacenter level, and encode it at multiple-bitrates there; from an architecture standpoint, this reduces the cost at the point of encoding and puts the more expensive processing and bandwidth load at the datacenter, where it should be.

Encoder Software Variety/Options (applies to Live Streams only)

Windows Media definitely gets an A here, with nearly every “webcasting” software supporting it. Flash has a couple options (since it’s an RTSP based stream), but Flash Media Encoder is free and works pretty well; perhaps 3rd party vendors gave up?

Smooth Streaming gets a D here, with Expression Encoder 4 being the only software package available.

By Next Year: More options for Smooth Streaming support, please.

Encoding Connection / Interruption Recovery (applies to Live Streams only)

Ahhh, this one is near and dear to my heart – and one of the biggest pain points.

One of the best measures of a technology is how it fails.

Flash nails this, and Microsoft still has it backwards in both Windows Media and Smooth Streaming. Specifically, I’m talking about how the encoders behave in a Push scenario (the encoder connects to the server and pushes content to it, as is the case with most remote live broadcasts, where you don’t have the luxury of network and firewall control).

Once you start pushing a Flash stream (with Flash Media Encoder), it’s an unstoppable force. If the connection is interrupted, it reconnects automatically. Simple concept, right? Flash, you get to sit this one out… we’ll call you back into the hot seat in a minute.

Not according to MSFT, and I’m not sure why – perhaps I’ll try to contact some MSFT folks on The Twitter to find out. Windows Media Encoder (legacy) and the Expression Encoder family (new & shiny) both blow up when there’s a hiccup, and require a manual encoder restart. We’ve even written a prototype encoder shell using the Expression API’s just to get around this issue.

In Smooth Streaming, there is sort of a reason for this; each 2-second video “chunk” is timestamped in metadata, so things get a little complicated when you start timestamping over your timestamps, but there’s no reason why the encoder can’t keep a timeline counter running and try reconnecting.

To be fair, again, this is specific to Push streams. Pull streams are very reliable with both Windows Media and Smooth Streaming, and it’s the style we use whenever possible. I do maintain that the design is backwards, though – when you have enough network control to open up ports and facilitate a Pull stream, you most likely also have control over the QoS quality which will yield a far lower interruption rate in the first place. But I digress.

By Next Year: Microsoft has a seriously great platform with Smooth Streaming, but this is a dealbreaker in our market. Encoders must tolerate interruptions better than they do now, and Smooth Streaming restarts are a big pain point, exponentially so with CDN’s.

Server-Side Administration

As far as administration goes, Windows Media is a clear winner. Flash Media Server is okay, with some sorta-works-but-not-sure-how third-party solutions in the mix, and while Smooth Streaming is somewhat simple in concept (made clearer with some ever-improving modules), it’s still somewhat difficult to get a good picture of what’s going on, especially with live streams. (So much so, that we made a product out of it.)

By Next Year: I’d like to see the Smooth Streaming administration modules in IIS7 become clearer with greater visibility of realtime statistics and reporting, or perhaps – architecturally speaking, a web-based dashboard viewable remotely (similar to WM’s remote administration). Additionally, starting and stopping live Smooth Streams needs to be rethought. It has to become automatic.

CDN/WAN Caching and Scalability

The fundamental shift from constant TCP streams of data or progressive downloads to HTTP adaptive chunks puts Smooth Streaming in the lead here. When your video is no different than images on a web site, caching and scalability become a lot easier with WAN caching and CDN tools that were designed just for that. (If you have no idea what I’m talking about, just skip this.)

On the other side of the fence, Windows Media and Flash live have some hurdles when scaling out to external CDNs, and requires platform-specific tools on the CDN side.

By Next Year: All hail HTTP adaptive. Flash, get on this train (or at least get your people to use the train you already have). Like, by default.

Client-Side Playback: Buffering = Not Good

I think the grades speak for themselves here, and most people reading this are already familiar with the different experiences. IIS Smooth Streaming, by far, provides an extremely compelling experience on the desktop through Silverlight, Windows Phone, and iOS devices. While it’s clear we won’t see the Flash runtime itself on iOS anytime soon, Smooth Streaming offers a compelling mobile strategy as well for iOS and Windows Phone. None of the platforms here really target Droid.

The client side experience really comes down to the quality of video, and buffering behavior. Until Flash really embraces adaptive HTTP, it will, along with Windows Media, trail Silverlight Smooth Streaming as a client experience.

By Next Year: I’d like to see Flash become a better adaptive streaming platform, with better tooling to support it. The only other thing that could create compelling client experiences would be mobile in-browser runtimes, so that on iOS and Windows Phone you could build out RIA players.

OK, I’m done.

I think this brain dump is complete for now. If you’ve stumbled upon this post and you’ve seen real, working solutions for some of the shortcomings of these streaming platforms discussed, or why some design decisions were made in each regard, I’d love to know about them. Leave a comment – happy streaming!