The results are in – when it’s going well, it’s surprisingly great, and when it’s not going well, it’s terrible.

Total Chunks: 2691
Chunks Tested: 2691
Manifest Download Time: 310ms
Avg Video Chunk Response Time: 357ms
Avg Audio Chunk Response Time: 61ms

Excellent Chunks: 590 (21.92%)
Good Chunks: 1821 (67.67%)
Warning Chunks: 68 (2.53%)
Degraded Chunks: 6 (0.22%)
Poor Chunks: 206 (7.66%)

Stream 0: video, 2962kbps, 2056kbps, 1427kbps, 991kbps, 688kbps, 477kbps, 331kbps, 230kbps
Stream 1: audio, 128kbps

Things go OK around 89% of the time. But when they don’t, it crashes and burns…

Yikes. You can almost feel the Silverlight players freaking out.

However, it’s can be difficult to get a good benchmark on how your infrastructure is doing at serving up chunks, especially when your Silverlight clients are seeing random buffering errors or you run into scaling problems.

First off, there is a lot of information available from the Smooth Streaming Health Monitor app – In a couple seconds you can have a trace of what decisions the Silverlight Adaptive Streaming Media Element is making and export that out to Excel.

But when you just need comprehensive chunk data on all bitrates to diagnose how your origin/CDN is doing, I made this app (almost called it “Chunker”):

Enter the manifest URL of the on-demand smooth stream you want to test (note that this does not currently support live or composite manifests), for example: http://server.com/streams/BigBuckBunny.ism/Manifest

Once you click Begin Test, a new test tab will open and start requesting chunks based on the manifest information. The results will tell you if you may have a problem with your disk IO on your origin or some other problem preventing chunks being delivered in a timely manner.

Note: Only the first 1000 bytes (almost 1K) of each chunk is downloaded. The point here is not to test bandwidth, but rather test your infrastructure’s performance as it relates to reading/seeking fragments and assembling chunks.

Hopefully based on the assessment of each chunk, you can get an idea of how your CDN / origin / standalone box is doing at delivering chunks.

Run Smooth Stream Performance Testing Tool (ClickOnce)

I’m not crazy. Problem is, Live Smooth Streaming is the Cinderella of content delivery, and she’s stuck at home.

A little background – This “smooth streaming” concept kicked off somewhere around 2009 (MSFT’s first Smooth Streaming event I believe was the French Open). To get in the game, you had to have some crazy hardware, most likely a couple dedicated encoders running $20K a piece. Add to that 10mbps of upstream bandwidth to support it, and you’ve got a behemoth (yep, I said behemoth) of flying bits to deal with. Cinderella rides in a really, really expensive carriage.

Then, in 2010, Microsoft released Expression Encoder 4. The gates of heaven opened, and no longer did you have to have a $20K encoder – you could now do it with a piece of software that costs in between “free” to $200. This is where we came in. We provided the publishing points, origin services, and CDN – everything you needed to deliver your live event to the world.

Folks started using it, and streaming events from corporate webinars in the USA to operas in Russia. While it was okay for the most part, we found in a consistent stream of support issues:

• While most folks had hardware good enough for single-bitrate streams (good enough for Flash, Windows Media, UStream), it was by and large insufficient for Smooth Streaming. Trying to overtax the hardware produced weird results with audio/video sync and reliability issues.
• Connection recovery wasn’t pretty.

After a number of months with these issues, I pulled it. That begs the question, what now? I think the answer lies in encoding.

There is no question that adaptive streaming is the future. There is and will continue to be a big difference between the bandwidth you have at work, at home, on your mobile device, and soon, in your car. There’s a big difference between the size of your iPhone screen and your 60” living room smart TV. Media must adapt to your device and network before we can move toward phasing out television and toward interactive experiences on a wide scale.

This is what makes Smooth Streaming the Cinderella of live video on the web; the most beautiful solution, sitting at home mopping the floors while stepsister Flash is off to the ball. I’m not knocking services like UStream and Livestream.com – they’re good services; but just using older technology.

Smooth Streaming will remain an option reserved only for those with huge budgets for as long as 100% of the encoding burden is on the client. The bar is too high for this technology.

At MIX ‘11 in Vegas, the Channel 9 team did an incredible job of producing and streaming the event. They rented fiber at $500/hr and a bank of encoders. These are not normal people. This is a huge corporation with a high-profile event. It looks great, but the bar is 5 miles high in the sky.

Non-mainstream sports. Concerts. User groups. Weddings. Church. Webinars. Corporate training. Just a few examples of live events that have huge potential for higher-quality engagement that will keep deferring to other technologies until the bar is lowered. In other words, normal people are going to be happy with Cinderella’s stepsisters until the carriage is affordable.

So that’s what we’re going to do next. We’re moving encoding “to the cloud”. In a couple weeks we’ll push out the first beta. With the burden of encoding moved to the datacenter, we can push as much quality as your connection can handle, with a high quality stream on normal hardware. This is the first step in making a superior adaptive-streaming technology accessible to the events that can benefit from it.

Come on Cinderella, hop in.

This goes on from time to time. This weekend it’s .NET vs. whatever language(s) that Expensify prefers; other times it’s network equipment like Cisco vs. HP vs. Netgear vs. Dell, followed by Mac vs. PC and, let us not forget iPhone vs. Droid. Technology racism, if you will.

No one is ever correct in generalized, vague discussions, especially when talking technology platforms. Like driving on a highway, you can cover a lot of distance in a short time, but you really haven’t spent any time in the towns you just blew through.

The technology of choice depends on the work at hand.

Case in point: This past week, I went to work. I sat down on my desk, where I have a PC and a Mac.

I connected to our Cisco wireless access points, connected to our HP and Dell switches.

I used this network connection to code in C# on my Windows PC, PuTTy’d into a few of our Linux servers, edited a couple PHP scripts in nano, dabbled a bit in C++ DirectShow filters, wrote Javascript-style sketches for an Arduino, integrated systems with Microsoft CRM, created a SQL database and integrated with Oracle, streamed 2 live events through Silverlight and Windows Media Services, talked on my VoIP phone, looked up information on a SharePoint intranet, and FaceTime’d on my iPad.

Mixed bowl, huh.

I love swimming in such a soupy mix, because it makes me a better solution provider when I have more than one tool on my belt – and it gives me respect for each vastly different approach that these tools take.

The CEO of Expensify ultimately compared .NET to “cooking in a McDonalds kitchen” – trying to make the point that folks who use .NET are sheltered from low-level details (a “real kitchen”).

“None of this makes you a “bad programmer”. All these differences are perfectly irrelevant if you just want to make 1.6 oz burgers as fast as possible, and commit the rest of your career to an endless series of McDonalds menus. But every day spent in that kitchen is a day NOT spent in a real kitchen, learning how to cook real food, and write real code.”

I guess the C# I wrote last week was fake code. I guess I wrote it for a fake customer. I suppose the check they’ll send me is fake. Does the PHP count?

You see, a good cook can make a great meal – er, real food – in a McDonalds kitchen. Or over a campfire. Or at a steakhouse.

The point I wanted to blog about so that I don’t forget is this: There is a huge difference between a vague and specific technology criticism. The question we’ve got to ask, that changes the game is:

“In what context, and in what example?”

“Macs are better than PCs.” In what context? (Certainly not ERP. But certainly print graphics.)

“iPhone is better than Droid.” In what context? (Certainly not turn-by-turn navigation. But certainly videoconferencing.)

… and back to Expensify:

“Again, this isn’t a rant against .NET — it’s fine. It’s not even a rant against .NET developers being incapable of learning — they can learn as well as anyone. It’s a rant against .NET teaching the wrong things for startups.”

In what context, and in what example? What wrong things? And whose startup?

If you’re unable to answer that question, you may as well be arguing Why You Don’t Hang Out With People Named David.

Make your choice for the job at hand. Deliver great solutions, in whatever kitchen you cook.

The situation is this: As a simple “remote broadcast” setup, we want to connect FaceTime to a regular TV (with composite inputs – you know, the Yellow / Red / White jacks), so that we can see the remote person on the TV. Seems pretty simple, with iPad 2’s display mirroring features, right? Well, not exactly.

(If you’re a nerd, the “TV” in this equation is actually a video capture card; we’re piping FaceTime into a live encoder and broadcasting it to a live stream. Think CNN Skype interview-style. A reasonably-priced capture card is no different than an older model TV with no HDMI input, with S-Video and Composite inputs.)

So, on to the adventure. Apple has a few different connectors for the iPad/iPhone/iPod Touch that mirror the display or provide a TV out in some form or fashion. Let’s bring out our contestants:

Apple 30-pin to Composite AV Cable (the yellow / red / white thingies)

This would naturally be the most direct approach, but unfortunately, doesn’t work. This doesn’t mirror, but rather functions as a TV Out display. You can play a video over it, but does not provide display mirroring. When using the FaceTime app, there is no signal.

Goodbye, Composite AV Cable.

The direct approach doesn’t work, so if you’re like me, you’ll just take the next best option and try to convert it to Composite:

Apple Digital AV Adapter (the HDMI cable)

This works reasonably well as a display mirroring solution. I connected it to a Samsung LCD TV, and the picture seems a little overexposed, but it does the job and mirrors FaceTime on TV via HDMI. One major drawback to our eventual old composite TV destination:

Your HDMI display/conversion device MUST support HDCP HDMI. So in this case, if you’re thinking all you need to do is convert HDMI to composite, most HDMI converters don’t support HDCP. This one just doesn’t output any signal.

Apple VGA Adapter

Alas, we have a solution.

VGA adapter does, indeed, mirror FaceTime on iPad 2. And, you can scan convert it to Composite.

All things considered, the “mirroring” functionality in iPad 2 works great in HDMI (mostly as advertised except when Hollywood’s legal people get in the way), but this was a fun little trip through the iPad Accessories aisle.

I imagine this will become a popular solution as FaceTime reaches more devices and more iPads get turned into presentation platforms, so I hope this post will save a little time and a little headache if you’re on the same journey. Happy Mirroring!

Some Background

For years now, the team at X9 has supported a reasonably-sized media delivery infrastructure comprised of Windows Media, Flash, and most recently, IIS Smooth Streaming. Last year alone we served up hundreds of thousands of hours of content, and a little over 10,000 video assets passed through our “doors” to be delivered across the internet. We’ve been around this block a few times, and a new/major twist in the technology has been coming around every 2 to 3 years.

There are 2 common things I’ve observed with nearly every technology available today:

• Generally designed for on-demand delivery, with live broadcasts as a second-rate citizen
• Critically lacking in the area of encoding, distribution, or client-side playback

If only there were a way to clearly see which technologies in the “big 3” were performing well, like a…

Streaming Technology Report Card

Bam! Done.

Now, mind you: This is the official report card from me and only me, using my experience in these technologies as it relates to the small and mid-sized streaming market, using features that are only important to me as we have experienced them as a company. These scores also do not take specialized hardware into account; this is streaming on hardware that normal people and normal companies have and can afford, so there are no $30,000-per-channel encoding devices or massive render farms in this equation.

You may also notice that there is no “summary” or “winning technology” – that’s not the point here; rather, each technology suffers different shortcomings across its usage path through the encoding / distribution / client-playback process – only winners in each feature category.

I have excluded some other technologies that we do support (for example, Shoutcast, QuickTime) in favor of focusing on the big 3 for video delivery.

Now, let’s go over some observations on the overall media delivery path.

Encoding: Nothing Exciting Here.

Overall, the encoding process across technologies is…. meh, okay. Classic Windows Media has been around the longest and, by far, has the largest market penetration. You can encode WMV in any number of tools and nearly every video editing package on Earth. It’s performance is okay. The Achilles heel for Windows Media is the quality. It takes a large, large amount of bandwidth and video size to get to the smooth, professional look that Flash and Silverlight/IIS Smooth Streaming offer with H.264. Nevertheless, the majority of our assets are still Windows Media, because of the complexities and shortcomings of Silverlight/IIS Smooth Streaming that we’re still working through.

Flash’s tooling and integration with video editors and automated encoding libraries is hit and miss. There isn’t really a ubiquitous “go-to” solution for encoding and publishing Flash video. Moreover, the market is littered with third-party utilities of poor quality that makes you want to just use YouTube instead.

One of the best – if a bit complicated – workflows comes from Silverlight Smooth Streaming with encoding done by Expression Encoder. However, since Adaptive Bitrate streaming requires encoding a stream up to 8 times, it’s extremely resource-intensive to pull off this format.

There have also been some great advancements here recently from the Smooth Streaming team (batch encoding Transform Manager, for example).

By Next Year (Encoding):

There’s not too much going horribly wrong in the Encoding areas of all 3 technologies, but there is some room for improvement. Windows Media is still a stronghold because of the encoding and distribution problems of Silverlight/IIS Smooth Streaming. That’s a sad place for a technology to be; however, the worst shortcomings in Windows Media (quality and client experience) have already been addressed in Smooth Streaming, so it’s time for WM to fade off into the sunset… if Smooth Streaming can fill in some gaps in Distribution to get us there.

Flash – we need some better tooling for on-demand.

Distribution: Joy and Pain, all By Design

If there’s one area to improve upon in the streaming industry, distribution is it.

The scores here really reflect the different personalities of each technology, and their makers. I’d like to hit on each point in depth here:

Encoder Upstream Bandwidth Required (applies to Live streams only)

Both Windows Media and Flash are pretty much the same here. You want to deliver a 350kbps stream; you’ll need 350kbps. However, what makes Smooth Streaming great really kills it for live remote broadcasts. By the time you encode a stream a few times, you have to have a good, solid high-speed upstream connection at minimum to even be in the game. This is a problem for a lot of locations here in the U.S., with upstream bandwidth being prohibitively expensive outside of datacenters.

By Next Year: The model of intense encoding resources and high bandwidth at the encoder doesn’t make sense for Smooth Streaming. In a year, I’d like to see Smooth Streaming at a point where we can send one high-quality broadcast to the datacenter level, and encode it at multiple-bitrates there; from an architecture standpoint, this reduces the cost at the point of encoding and puts the more expensive processing and bandwidth load at the datacenter, where it should be.

Encoder Software Variety/Options (applies to Live Streams only)

Windows Media definitely gets an A here, with nearly every “webcasting” software supporting it. Flash has a couple options (since it’s an RTSP based stream), but Flash Media Encoder is free and works pretty well; perhaps 3rd party vendors gave up?

Smooth Streaming gets a D here, with Expression Encoder 4 being the only software package available.

By Next Year: More options for Smooth Streaming support, please.

Encoding Connection / Interruption Recovery (applies to Live Streams only)

Ahhh, this one is near and dear to my heart – and one of the biggest pain points.

One of the best measures of a technology is how it fails.

Flash nails this, and Microsoft still has it backwards in both Windows Media and Smooth Streaming. Specifically, I’m talking about how the encoders behave in a Push scenario (the encoder connects to the server and pushes content to it, as is the case with most remote live broadcasts, where you don’t have the luxury of network and firewall control).

Once you start pushing a Flash stream (with Flash Media Encoder), it’s an unstoppable force. If the connection is interrupted, it reconnects automatically. Simple concept, right? Flash, you get to sit this one out… we’ll call you back into the hot seat in a minute.

Not according to MSFT, and I’m not sure why – perhaps I’ll try to contact some MSFT folks on The Twitter to find out. Windows Media Encoder (legacy) and the Expression Encoder family (new & shiny) both blow up when there’s a hiccup, and require a manual encoder restart. We’ve even written a prototype encoder shell using the Expression API’s just to get around this issue.

In Smooth Streaming, there is sort of a reason for this; each 2-second video “chunk” is timestamped in metadata, so things get a little complicated when you start timestamping over your timestamps, but there’s no reason why the encoder can’t keep a timeline counter running and try reconnecting.

To be fair, again, this is specific to Push streams. Pull streams are very reliable with both Windows Media and Smooth Streaming, and it’s the style we use whenever possible. I do maintain that the design is backwards, though – when you have enough network control to open up ports and facilitate a Pull stream, you most likely also have control over the QoS quality which will yield a far lower interruption rate in the first place. But I digress.

By Next Year: Microsoft has a seriously great platform with Smooth Streaming, but this is a dealbreaker in our market. Encoders must tolerate interruptions better than they do now, and Smooth Streaming restarts are a big pain point, exponentially so with CDN’s.

Server-Side Administration

As far as administration goes, Windows Media is a clear winner. Flash Media Server is okay, with some sorta-works-but-not-sure-how third-party solutions in the mix, and while Smooth Streaming is somewhat simple in concept (made clearer with some ever-improving modules), it’s still somewhat difficult to get a good picture of what’s going on, especially with live streams. (So much so, that we made a product out of it.)

By Next Year: I’d like to see the Smooth Streaming administration modules in IIS7 become clearer with greater visibility of realtime statistics and reporting, or perhaps – architecturally speaking, a web-based dashboard viewable remotely (similar to WM’s remote administration). Additionally, starting and stopping live Smooth Streams needs to be rethought. It has to become automatic.

CDN/WAN Caching and Scalability

The fundamental shift from constant TCP streams of data or progressive downloads to HTTP adaptive chunks puts Smooth Streaming in the lead here. When your video is no different than images on a web site, caching and scalability become a lot easier with WAN caching and CDN tools that were designed just for that. (If you have no idea what I’m talking about, just skip this.)

On the other side of the fence, Windows Media and Flash live have some hurdles when scaling out to external CDNs, and requires platform-specific tools on the CDN side.

By Next Year: All hail HTTP adaptive. Flash, get on this train (or at least get your people to use the train you already have). Like, by default.

Client-Side Playback: Buffering = Not Good

I think the grades speak for themselves here, and most people reading this are already familiar with the different experiences. IIS Smooth Streaming, by far, provides an extremely compelling experience on the desktop through Silverlight, Windows Phone, and iOS devices. While it’s clear we won’t see the Flash runtime itself on iOS anytime soon, Smooth Streaming offers a compelling mobile strategy as well for iOS and Windows Phone. None of the platforms here really target Droid.

The client side experience really comes down to the quality of video, and buffering behavior. Until Flash really embraces adaptive HTTP, it will, along with Windows Media, trail Silverlight Smooth Streaming as a client experience.

By Next Year: I’d like to see Flash become a better adaptive streaming platform, with better tooling to support it. The only other thing that could create compelling client experiences would be mobile in-browser runtimes, so that on iOS and Windows Phone you could build out RIA players.

OK, I’m done.

I think this brain dump is complete for now. If you’ve stumbled upon this post and you’ve seen real, working solutions for some of the shortcomings of these streaming platforms discussed, or why some design decisions were made in each regard, I’d love to know about them. Leave a comment – happy streaming!

As such, it’s really annoying to me when implementing an interface and VS generates nice #region tags all the time.

Props to this StackOverflow question/answer for the solution.

Tools > Options > Text Editor > C# > Advanced > uncheck “Surround generated code with #region”.

I guess this is a new technique… Punish with Patience.

That’s why I was astonished at the most recent Flash update to 10.1. This morning I was prompted to update Flash because of a security issue via the latest update of Firefox. No problem, however, this was the first clue that Adobe was starting to abuse their place as a popular browser plugin:

See that? By default, “Include in your download” – McAfee Security Scan Plus. I’ve used McAfee. I’m not fond of their software. I don’t want it. I just want Flash. No problem… I unchecked “Include in your download” and downloaded Flash player.

So then I downloaded the Adobe Download Manager (let’s not get into the pet peeve of downloading a downloader), and to my surprise:

Flash Player at least asked me to accept the license terms. Not only did McAfee Security Scan Plus download and install automatically when I specifically unchecked that option – it installs quickly and automatically with no confirmation. Almost like they knew I would say no if given the chance.

So Adobe, here’s my list of grievances:

1. Attempting to include third-party software by default, especially when it isn’t related to what I actually want (and you know that), is unethical.
2. Browser plugin runtimes are to be treated as sacred. We as developers use them to bring great experiences to the web, and in turn, get users to install your product. It is imperative that our users trust your runtime. You have disrespected us, disrespected the users we serve, and destroyed that trust by making such a move.
3. Waiting until version 10 and 96% browser plugin market penetration to do this is unacceptable. Being the #1 rich content plugin comes with an industry-wide responsibility.
4. Including the download anyway, when I specifically unchecked the option, turns a previously respected Flash runtime into malware.

Please reconsider this delivery model. This is not about Flash in particular, but it is about users being able to trust the applications we deliver to them. The more we leave users with a bad taste in their mouths after updates, the less they’ll update… and that ultimately hurts us all.

If you need to retrieve the ID (GUID) of a Feature, for running STSADM commands, etc, there’s a quick and easy way to do this for features that are currently activated:

1. Go to your Site Settings, where your features are listed and you see the Activate/Deactivate buttons.

2. Press the Deactivate button (don’t worry, this goes to a confirmation page, so you’re not causing anything to happen yet)

3. While you’re on the confirmation page, take a look at your browser address bar – voila, your feature ID:

http://server/_layouts/DeactivateFeature.aspx?FeatureId=f6924d36-2fa8-4f0b-b16d-06b7250180fa