Hybrid Storage – A Smarter Way of Balancing Cost and Performance

While the arrival of high-resolution video formats has helped deliver a more dynamic and engaging experience for TV audiences around the world, it has also created a problem for content producers and broadcasters in terms of storage and archiving.

Many video professionals have decided to store material in the cloud, and this enables them to collaborate and exchange media files more easily. However, while being flexible and very accessible, the recurring cost of cloud-based storage for high-volume producers rapidly becomes prohibitive. And if we look at the macroeconomics of media production, this problem is likely to get worse, as cameras will shoot more pixels and video file sizes will grow accordingly.

At NAB Show Las Vegas in April 2024, BlackMagic Design announced the launch of their URSA Cine 17K digital film camera which can produce 5GBs of video data per second. This is a growing trend. As the amount of content being produced annually keeps on accelerating and data volumes grow in parallel, the average producer and post-production facility now manages hundreds of Terabytes of content every week; content that needs to be stored and, ultimately, archived. Initially, broadcasters managed this material by investing in on-premise storage with long-term archiving being managed by specialist third-party companies. However, the advent of cloud computing opened up the tantalising prospect of broadcasters being able to store, archive, interrogate and share multiple copies of hi-resolution video content quickly, easily and at an accessible price point.

While this certainly sounds attractive, the coal-face reality is unfortunately a little different. Cloud storage currently remains a stubbornly expensive exercise – costs range from US $5 to $20 per Terabyte per month (depending on the performance) – and so most producers will store the majority of their footage on premise at a fully burdened cost of €100 per Terabyte (all in), and hold additional copies of content on tape for safety and security reasons. The utopian vision of a seamless cloud-based editing, sharing and storage workflow still therefore feels like a long way off.

That said, there are innovative solutions available to help work around this challenge, often referred to as Hybrid Storage, and the most notable of these is Limecraft’s Cloud Connector.

What is it?

In essence, Cloud Connector, announced at IBC 2018 and fully introduced in 2019, is a component of the Limecraft platform that enables content producers and broadcasters to maintain (convenient and cost-effective) local storage but still use the Limecraft suite to locate, manage and share content. It is an agent that sits on a local server or machine in the producer or broadcaster’s facility (provided by them) and talks to the Limecraft platform. Security and access permissions are set at the time of onboarding the customer and are completely configurable thereafter.

What’s the benefit?

• Connects local storage to the online production workspace, thereby establishing a so-called Hybrid Storage model.
• In cases of real-time collaboration, users don’t have to upload hi-res material to the cloud, thereby avoiding expensive recurring storage costs.
• Integrates with pre-existing on premise MAM solutions such as Editshare or Avid MediaCentral.
• Cloud Connector makes cold storage accessible, which then starts behaving like near-line storage in combination with the workspace on top. This means you can execute partial retrieves from said tapes for editing purposes, thereby saving massively on allocated network and online storage capacity.
• Cloud Connector can execute a variety of other tasks as part of workflows initiated from and controlled by your production workspace.
• Cloud Connector reports back to the production workspace, so you can keep track of workflows in the central workflow overview.

How does it work practically?

Cloud connector is a local software agent that connects local storage (and/or S3 and Azure cloud storage) to the central Limecraft platform, and executes tasks running in the docker container. It is orchestrated through the main Limecraft workflow engine and can execute a variety of different tasks.

Cloud Connector includes a number of very useful features and integrations:

• Full two-way communication enables upload and download to the central platform, and manipulation of files on local storage.
• Cloud Connector subscribes to worker queues, and multiple cloud-connectors can be subscribed to specific queues, resulting in a highly efficient and scalable system to manage your content data workload.
• Watch folder functionality enables Cloud Connector to monitor local storage for changes and then execute predefined activities, e.g. upload a proxy of new content to the central platform and notify relevant users.
• Cloud Connector can perform transcoding activities as part of a defined workflow and can automate certain activities like content grouping and checkout functions.
• Integration with Archiware middleware enables Cloud Connector to manage archive and backup functions.
• Integrates with popular Production Asset Management solutions, such as Avid Interplay.

A reassuring note for IT and Network Security professionals 🙂

The use of Cloud Connector and the setup of a Hybrid Storage architecture using it is 100% secure. It does not affect your security perimeter and does not make you more susceptible to hacking, Denial of Service (DoS) or similar types of attacks.

Logically, the Limecraft Workspace controls the Cloud Connector, but in fact it is Cloud Connector that polls for tasks. Limecraft will never request access through the firewall from the outside in.

It uses port 80 – a standard network port for web servers using HTTP – to communicate with the Limecraft platform, so does not involve any complex or esoteric forms of installation or operation. It is based on familiar and well-understood protocols.

Real world examples

De Mensen/LISA: As part of the ingest process, Cloud Connector receives rushes which it stores on near-line storage and subsequently backs up on LTO tape (via Archiware). As a result, De Mensen has fully automated the ingest of camera cards, end to end.

Hotel Hungaria/Onze Natuur: Cloud Connector is used to swap material between Scale Logic, Archiware, etc. Hotel Hungaria manages several Petabytes on tape. Hotel Hungaria was an early adopter of Hybrid Storage through Limecraft Cloud Connector.

💡In this webinar organised by Archiware, Olivier Struyven (CTO and Head of Postproduction at Hotel Hungaria), explains the setup.

Arrow Media: Cloud Connector is instrumental in implementing systematic archiving processes, and providing impressive storage economies. Rushes are ingested to the Limecraft platform (via offload storage) which then uses Cloud Connector to orchestrate archive storage on a disc-based archiving system (Alto Disk Archive), with secondary archiving in the cloud (Azure). Content can also be uploaded directly to archive storage in the same manner.

SVT/Transcription: Cloud Connector provides the link between Avid’s MediaCentral platform and what happens in Limecraft. Users have no need to leave the native Avid environment when creating transcriptions and subtitles, so Cloud Connector provides a smooth and seamless user experience.

Eclair: Multiple Cloud Connector instances connect to +/- 10 different storage technologies. Eclair executes 55000 workflows per year, mostly involving file transfer operations through Cloud Connector.