by Oliver Lietz, CEO nanocosmos
How can I create true interactive live streaming applications with ultra-low latency?
Live video streaming is getting more and interactive. Webcasters and broadcasters want to be able to communicate directly in real-time or engage with their audience by getting live feedback during presentations or events.
Live streaming of user-generated content on mobile devices, or live communications tools for auctions or e-sports require bidirectional connection with ultra-low latency (below 2 seconds). Our nanoStream H5Live playback technology lets you build interactive applications with ultra-low latency, plugin-free on any browser and on any device.
ENCODING ———–STREAMING ———–PLAYBACK
What is latency?
Latency is the time between the capturing of a video frame with a camera and the playback of this same frame on another device (“Glass-to-glass”). For internet streaming applications, latency is potentially added by all components in the streaming workflow, the camera/encoder, upstream network, streaming server, downstream network, or video player on the viewer side.
You might already have noticed this effect: when watching a live event like a football match over IPTV or a live stream over the internet, and listen to the same match on the radio, you will notice that the video stream is several seconds behind the radio.
What latency values can I expect?
Standard video distribution protocols for live playback in web browsers are based on HTTP, for example the protocols HLS and DASH. Latency values can get very high, up to 30-60 seconds. When people talk about low latency, they usually mean turning latency down to 5-10 seconds.
However, this is still far too high for interactive live video applications, which need ultra-low latency below 2 seconds.
What is the reason for long latency times?
Every network transport creates latency. Sending packages around the world can take up to 1 second. But why is video latency still much higher?
Latencies are not only dependent on network infrastructure, but also on the video compression and packaging and delivery method used. Current playback technologies use HTTP segments pulled by the player from the streaming server. HLS segments are several seconds based on the concept of “key frames”.
All compression methods like H.264 use the concept of “grouping” pictures (GOP) into chunks with keyframes of every 2-10 seconds or more. With HLS, you usually combine 3 segments together to deliver to the player, this results in at least 6-30 seconds end-to-end latency. In the end, the player will see a playlist and file chunks which it loads over HTTP. Other technologies like MPEG DASH basically work with the same concept and have the same issues.
Additionally, latency will increase the risk of network dropouts and instabilities. Interactive applications can not work under these conditions.
Live Encoding with RTMP; Segmented delivery / playback with HTTP/HLS/DASH
How can ultra-low latency be achieved for interactive applications?
Interactive applications, like live streamed auctions or bets, depend on and require ultra-low latency below 1 or 2 seconds for an approved buyer experience.
In Flash-based environments on desktop browsers, the RTMP protocol was able to achieve latencies around two seconds in good network conditions. RTMP is still a viable protocol for in-app applications if you are using a latency-tuned software like nanoStream; but as browsers are disabling their support for Flash-based players, only HTTP-based protocols remain available. These protocols like HLS and DASH cannot deliver a realtime performance for live streams.
How can you bring latency down to ultra-low levels?
A more “intelligent” delivery format is required to keep the latency low independent from constraints like GOP lengths.
As there is no viable solution and wide-spread technology available, we decided to develop a new technology in-house, called H5Live. H5Live is compatible with all HTML5 browsers and runs plugin-free on any device.
What is H5Live and how can we guarantee ultra-low latency live streaming and playback?
To respond to our customers’ needs for ultra-low latency use cases, the nanocosmos team invented the plugin-free nanoStream H5Live stream delivery and playback technology.
nanoStream H5Live is a client-server delivery and playback solution based on HTML5 technologies, enabling ultra-low latency browser-based playout, plugin-free on all web browsers.
What about WebRTC – is it used in H5Live?
WebRTC, the real-time communication technology promoted by Google was designed for peer-to-peer communication in small groups only.
Our nanoStream WebRTC.live solution can be used for live encoding and broadcast, but on the delivery side it is missing CDN and vendor support (e.g., it is not supported by Apple). H5Live is much better scalable and compatible with all browsers on all devices.
An end-to-end live streaming workflow: the H5Live Player and nanoStream WebRTC.live
Together, nanoStream WebRTC.live and H5Live form a powerful end-to-end live streaming workflow. With no plugins needed, the workflow enables ultra-low latency live streaming from encoding, through streaming and playback – directly within web browsers.
Why did we at nanocosmos decide to create H5Live?
Availability: H5Live is based on standard HTML5 technologies that are available in all browsers, and can be deployed on desktop and mobile platforms (iOS and Android).
Large-scale Distribution: WebRTC is not very well suited for large scale distribution and delivery. Neither is it available on all browsers and OS: it can primarily only be used on Chrome and Firefox.
Cross-platform Integration: The streaming protocol is similar to HLS and DASH, but with a modified workflow to enable true ultra-low latency live streaming applications for the web.
H5Live works on all browsers. If the “Media Source Extension” is available, we are using a low-latency delivery approach based on segmented MP4. For iOS, H5Live is using a modified version of HLS to allow playback with standard Safari setups.
Cloud-based Application: Our nanoStream Cloud end-to-end streaming solution makes live streaming applications adaptable to different streaming environments.
Live Encoding with WebRTC and Live Playback with H5Live: For live encoding, we created a WebRTC-Live Streaming Bridge, which can stream to any RTMP destination, CDN or nanoStream Cloud. In combination with the H5Live playback solution, you can create a full plugin-free, browser-based end-to-end live streaming application.
nanocosmos has been providing customized video solutions since 1998, with their nanoStream products for live encoding, streaming and playback now available cross-platform end-to-end from the camera to the viewer, for your brand.
based on Oliver Lietz’ Executive Vision for Streaming Media Magazine, Winter 2016:
Further reading on Low Latency with nanoStream: