All about... how we archive television

The first time the BFI acquired a television programme, the BFI National Archive was known as the National Film Library and it was a mere two years old. It was 1937 and the BBC had launched television in Britain the previous year at Alexandra Palace. Television Comes to London, a film produced by the BBC showing the inner workings of television production, was shown on the opening night of the television service on 2 November 1936. A copy was acquired by the BFI in 1937 and so our television collections began with a single title. 

Television was considered an ephemeral medium at this time, with programmes being broadcast live without any recording to keep them for posterity. Consequently, little pre-war television survives and it would be many years before the archive began to consider collecting television on a regular basis.

TV culture in Britain boomed in the 1950s, with the launch of ITV in 1955 breaking the BBC’s monopoly and providing viewers with a choice of channels for the first time. The BFI took note of the burgeoning medium and appointed a television acquisitions officer in 1959 before officially adopting television as part of the institute’s remit in 1961.

In this period the archive was reliant on donations for programmes and there was also a technical reluctance to accept videotape as a preservation medium. These impediments restricted television collecting for years until, in 1969, the Independent Television Companies Association provided the archive with an annual grant to begin to purchase copies of programmes from TV companies for acquisition. The programmes were selected by a series of committees, consisting of broadcasters, producers, journalists and archivists. The committees considered all manner of programmes – dramas, documentaries, current affairs, children’s programming, entertainment, all of the inexhaustible variety of television.  

But the grant could only go so far, and television was going much further, with millions of people tuning in to a vast variety of programmes each and every day. And television was more than the programmes that composed the schedules – the full experience of watching it had a whole range of rhythms, with advertisements, trailers, idents and more.

Plans began to emerge for a much more ambitious way of archiving television in the 1980s. Below, my colleagues explain how we started to keep up with public service broadcasters, how this is still maintained on a daily basis, and how we’ve expanded our reach in the age of streaming. 

– Lisa Kerrigan, Senior Curator of Television

We’ve been recording programmes directly ‘off-air’ from public transmission services as a means for preservation and access for around 40 years. Through the early 1980s, the BFI forged close relationships with the ITV franchise companies and also the fledgling Channel 4 with the aim of establishing a formalised national arrangement for the systematic selection and permanent preservation of independent television.

A new arrangement would emerge, whereby through the assistance of voluntary funding from the television companies, a state of the art technical facility at the BFI National Archive’s Conservation Centre at Berkhamsted would be built to capture selectively, ITV and Channel 4 off-air signals at optimum transmission quality and record them onto the broadcast industry standard 1 inch C-Format for permanent preservation in the archive’s vaults. For every master preservation recording created, a viewing copy was simultaneously created on VHS cassette to provide an easily accessible surrogate for research purposes at other BFI premises.

The new process was officially launched on 1 January 1985 with a video unit at the Conservation Centre operating every day of the year (including public holidays), resourced by a small team of newly recruited broadcast engineers on a 12-hour shift basis, with a programmable timer system later added to automatically make recordings between 01:00 and 13:00 while the unit was unstaffed.

In 1990 the arrangements with ITV and Channel 4 became further consolidated through legislation within the 1990 Broadcasting Act to give the BFI National Archive the role of the national television archive by statute. Another function was added in September 1990, when it was agreed with the BBC that all BBC1 and BBC2 programmes should be captured continuously off-air for the BFI to provide a comprehensive public access viewing service on their behalf. These recordings were not intended as a means of preservation, so for simplicity – and to minimise cost – the Super VHS videotape format was used, with a VHS copy being made at the point of access for viewing at other BFI premises.   

These two recording processes, based on mainstream analogue terrestrial television broadcasts continued steadily throughout the 1990s, with all technical maintenance, repairs and developments performed by engineers within the shift teams to ensure smooth running and high preservation standards were met. Long days shift working enabled efficient use and care of expensive equipment and provided the added advantage that recording schedules could be changed and increased at very short notice to accommodate live coverage of emerging world-situations.

Occasionally special technical projects were run in order to accommodate the introduction of NICAM stereo sound, and the gradual replacement of analogue 1 inch videotape systems with Panasonic’s new digital format D3, and then a few years later Sony’s more ubiquitous Digital Betacam.

The latter part of the 1990s saw the addition of a further independent public service broadcaster, Channel Five. The video unit operation was extended and programmes were selected for preservation from the outset. The BBC Access operation was also extended to incorporate the rolling News 24 service and then subsequently BBC3 and BBC4. As Super VHS approached the point of obsolescence in the mid 2000s, it was agreed that it should be replaced as a storage medium by the far more cost-effective and space efficient recordable DVD format. 

The advent of digital terrestrial television in 1998 brought both technological challenges and perceived opportunities for preservation and access. Having obtained one of the first ‘On Digital’ set top boxes in advance of the digital switch on, the video unit carried out experimental work to see if it might be possible to capture and preserve this new mode of public television delivery. This in-depth research formed the basis for a later evolution whereby the entire ‘data’ of digital terrestrial television (DTT) transport streams would be captured and stored temporarily for 21 days to allow greater curatorial selection focus than had previously been possible in advance of transmission.

Through the 2000s, momentum was maintained within research and development to enable the transition from analogue technologies to an ever more digital world. Moore’s law was proving a reality and signalled the continued decline and eventual demise of videotape as a storage medium, with the high commercial yield of digital storage rapidly becoming a far more appealing technological and economic proposition. 

Having already developed a solution to capture and temporarily store digital terrestrial TV data from off-air sources, during the latter part of the 2000s the video unit investigated digital preservation models to enable the transition from proprietary, obsolescent and costly videotape formats to a digital alternative which would provide the best chance for future migration to a fully integrated digital preservation infrastructure (DPI). 

After much review, which involved many peer site visits, seminars and conferences, LTO tape using the emerging and later to become ubiquitous LTFS file structure was chosen as a cost-effective and standards-based solution, completely achievable within the constraints of existing BFI operations and technical skills.

From 2010, programmes received off-air from digital television and curator-selected from the 21-day hard disk RAID arrays were committed to the new LTO 5 tape format for preservation. It was understood that while this method would remove proprietary videotape from the off-air archiving process, still being a human intensive tape process, it would only herald the beginning of a longer-term strategy where automation and integrated with metadata within a maximum of five years would become essential to mitigate the threat of digital obsolescence.

Coincident with the early development of the BFI National Archive’s digital preservation infrastructure around 2011, discussions were held with the BBC to determine the viability of integrating their own digital capture and access technology, Redux. Despite being a far more complex and involved endeavour than originally anticipated, the ambition was accomplished with the result that a specially modified and independent iteration of the BBC Redux off-air digital capture and data preservation system was installed at our Conservation Centre.

With digital transport streams now captured direct from free-to-air satellite transmissions, it was possible to commit the data representing the entire programming schedules and EPGs from selected channels, directly to the preservation storage within DPI, thus rendering the previous manual and unsustainable methods of capture and storage to LTO5 and DVD redundant.

In 2015 we also began acquiring a selected number of programmes from independent public service broadcasters in broadcast quality master file formats. These programmes are selected annually by curators and reflect a range of genres from the channels preserved in the national television archive.

Our television legacy has over 350,000 curated titles, one of the largest collections of television in the world. The collection is unique – it not only contains the programme content but also a huge wealth of advertising, sponsorships, announcements, breaking news items, including the live content where it all went wrong. It’s the whole visual experience just like the audience at home experienced it.

It’s a huge social document and while we now watch anything, anytime and anywhere (quite a revolution), it all started with videotape. Now that our off-air recording is taken care of automatically by STORA (System for Television Off-air Recording and Archiving), our team of video conservation specialists can focus their attention on the digital preservation of off-air television titles that exist only on video tape.

– Charles Fairall, Videotape and Engineering Advisor, and Stephanie Perrin, Technical Operations Manager

We recently built a new system to record off-air television content from 18 UK channels. STORA is custom-built in-house for BFI use, replacing the existing off-air recording system, BBC’s Redux, which became defunct in May 2022.

It was our aim to build a replacement using open-source software and tools and in turn to openly share our code base via our repository at GitHub. This was launched in October 2022 and contains all the code needed to receive the streams from satellite feeds, cut them into their programmes and store them with their electronic programme guide (EPG) metadata in date and channel folders ready for ingest into our digital preservation infrastructure.

The FreeSat streams received from UK broadcasters generally contain one video stream, two audio streams, a DVB subtitle stream and a Teletext subtitle stream. In the DVB stream are event information tables (EIT) with ‘now and next’ programme information. This includes programme title and descriptions, broadcast channel, start time, duration and running status of a programme (running or not running). Every element in this stream is critical to our preservation requirements.

The STORA code uses this ‘now and next’ data to stop and start each programme recording, live as the stream is updated throughout the day. To capture the data we use one of the open-source project VLC’s libraries, which writes every piece of data in the MPEG Transport Stream direct to disk. These programmes are saved into a folder structure ordered by date of broadcast, channel and start time of broadcast. One stream recording is added to each folder, named ‘stream.mpeg.ts’.

Alongside the stream video file, the STORA code extracts the subtitles from the video and writes these into a file, and the EIT data is written to a CSV file. Finally, the EPG metadata for a day’s programming is downloaded for each channel and cut up into individual items and matched to the programme folder it represents, where it is also saved alongside the stream video file.

From the video, metadata and subtitles we build a database record which represents the programme with description of the stream content and the programme title and descriptions. Where this might be a repeat or part of a series the code links to existing records. Finally, the video is ingested into the BFI’s DPI where an access proxy video and thumbnails are generated that become available to our BFI teams for curatorial review.

The BFI have two STORA servers, with the newest about to be deployed to our BFI National Archive site in Warwickshire, the first remaining at our Conservation Centre, Berkhamsted. This will help aid robust capture of stream signals where local weather or power events may impact recordings.

This research project has involved many colleagues including John Daniel (digital preservation engineer) and Brian Fattorini (digital capabilities engineer), and we must thank the international developers and archivists who write and maintain the open source software that made this, and many BFI National Archive workflows, possible.

The latest addition to our exacting set of born-digital audiovisual workflow is digital media from Netflix, the first of the streaming platforms to enter into an agreement empowering the BFI National Archive to collect and preserve Netflix original programming that the archive’s television curators select for the national collection. With Netflix, that means IMF (interoperable master format), a recent technology standard for the management of complex versions in the digital audiovisual supply chain. As Netflix themselves describe it: 

“The interoperable master format (IMF) is a componentized media format and an IMF package (IMP) is a collection of media files with corresponding assembly instructions. In technical terms, an IMP – either complete or supplemental – consists of one IMF packing list and all the assets it references.”

The IMPs that we acquire from Netflix have created new digital preservation challenges, and we have built an IMF infrastructure to let our digital media specialists validate the packages against the SMPTE standard, to undertake quality checks for all video and audio versions as well as subtitles and audio description. Once checked, the packages flow into an automated documentation and digital preservation workflows, written in Python by our knowledge and collections developer. The result is IMPs documented in full in the Collections Information Database, and the validated files placed under preservation in our digital preservation infrastructure, safe and retrievable for decades to come.

– Joanna White, Knowledge and Collections Developer, and Stephen McConnachie, Head of Data and Digital Preservation

• After Life series 1 (Netflix)
• Anne Boleyn (Channel 5)
• The Bay series 2 (ITV)
• Bridgerton series 1 (Netflix)
• I’m a Celebrity… Get Me out of Here! (episode from 7 December 2021) (ITV)
• It’s a Sin (Channel 4)
• 999 Critical Condition (Channel 5)
• This Way Up series 2 (Channel 4)
• Top Boy series 1 (Netflix)
• We Are Lady Parts (Channel 4)

The BFI’s Collections Information Database is available to search online. Here you will find records of our television collections by searching by title, transmission date or broadcaster.

The BFI Reuben Library holds a large collection of materials about the history of television, television production, and television audiences.

Thousands of titles from our television collections are available to see in the BFI Southbank Mediatheque.

Thousands of newly digitised television programmes from the BFI National Archive and partner UK regions and nations archive are available to access in local libraries through BFI Replay.

The BFI Data and Digital Preservation GitHub for the STORA project.

The Special Collections of the BFI National Archive are home to personal collections from television writers and producers including Troy Kennedy Martin and Irene Shubik, find out more through our catalogue.

A select number of television titles from the archive are available for free on BFI Player.

BFI Southbank hosts regular programmes of TV previews, seasons of archive television and rare screenings of recovered programmes at Missing Believed Wiped.

Television titles are available to view for study through the BFI National Archive’s Research viewing service.