Artificial intelligence and archive film: how The Light Surgeons put a new spin on Japan’s earliest moving images

Tokinokawa is a six-screen and multichannel audio-visual installation produced by UK-based media collective The Light Surgeons. Commissioned by the BFI, it’s part of a larger performing music and cinema collaboration, Tracing the Circle, inspired by new restorations of some of the earliest moving images of Japan, preserved at the BFI National Archive and dating back to 1902.

The installation is currently on display at BFI Southbank, accompanied by new music by Japanese composer Midori Takada, a pioneer in experimental and ambient music. The result is a mesmerising work that charts the evolution of architecture, the natural environment and the social landscape through the dual lens of early and contemporary filmmakers, working 120 years apart.

The images were then analysed by bespoke AI software and presented with an animated infographic layer alongside audio-engineered compositions. At first glance the result is an odyssey of infinite algorithms, as the footage flows through Japan’s traditions and landscapes. But the combination of the images with Takada’s euphoric percussion also suggests a warning about the rise of AI tools and how they influence our view of the world.

The creators of Tokinokawa, The Light Surgeons’ Christopher Thomas Allen and Tim Cowie, joined me for a conversation about the project.

Christopher Thomas Allen: Back in 2018, the BFI National Archive were looking into a lot of early archive films in Japan to coincide with the Olympics. Initially the plan was to do screenings of these films with live music. We were invited to develop a live cinema show and given about five hours of early black-and-white film material.

Christopher: A lot of the footage was disintegrating and, to be honest, I loved it. When they started to restore it, it almost lost some of its beauty. I love stuff that’s falling apart; it’s evocative of memory. We went through the black-and-white material – we put an order to it, we structured it, and we edited about 90 minutes of thematic narrative.

Working with a Japanese producer, we tried to work out where the locations in the archive footage were, and then finally we got the budget to go there.

The window for doing this became quite intense because it was at the end of 2019 that we finally got the go ahead. I made it over there just before the lockdown kicked off in 2020. Tim was meant to join me to work with Midori on the music, but he couldn’t make it because they changed the travel restrictions.

Tim Cowie: Between Chris going and my flight leaving, which was just a week apart, the travel laws had changed. It was crazy.

Christopher: I managed two weeks of filming and gathering all the material. There were certain places we couldn’t get into because COVID had also impacted Japan, although less so at the time.

When I came back to the UK, the project went into an indefinite pause because everything was closed, and the live show couldn’t be done. It went into this prolonged state of not knowing where it would go. We then started thinking about how it might function as a film rather than a performance. We could go back to the performance later. The concepts and ideas evolved a bit, and we started going through a lot of the footage that was shot while I was in Japan.

Christopher: I flew over to Paris to see her perform. Musicians like her and Steve Reich are influences on the stuff we’ve made. It was amazing to go and see her live performance and get a sense of the key theatrical element in her work. She had lots of interesting things to say about the archive. We discussed how the archive exists in this relationship between the past, the present and the future. And so, from there, a three-act structure started to emerge. The archive has no fixed position. It’s this constantly moving thing that we’re reassessing from the present and projecting into the future. So those ideas of circular narratives and themes were central in the piece.

Midori went away and started writing and thinking about music based on the sections and chapters that we had edited from the archive footage.

Tim: The idea was that when we got together in London for rehearsals, we would have some time to develop the music and bring the new video footage that Chris shot with the extra sound design layers to it. Unfortunately, we couldn’t collaborate closely with Midori and develop the installation, so we just worked with those pieces of music that she created. I was looking at ways that we could use the music and have the footage projected on a wall. The idea was this metaphor of time as a river running through the piece.

We’ve got six speakers matching the video screens and the music travelling with them. Everything’s flowing left to right as if it’s a stream. As one little excerpt drifts off, another one drifts back in. There was talk of time all the way through the development of the piece – a lot of Midori’s pieces are at 60 BPM and they’re all on the second. It’s like this ticking clock with the drum pounding, so it made it easier to mix the pieces.

Tim: I wanted to bring the sound of spaces and environment into the piece, so the sound design layer is made of a lot of film recordings and recordings from the video footage itself. That helps immerse you in the environment. There’s a drone layer that’s being driven by the actual sound in the footage, that’s been processed.

Christopher: There’s a funny story about a dead body, which was affecting and made me think about the material a lot afterwards. One location in the archival footage was a waterfall, Kegon Falls, in a place called Nikko that has a very large Shinto shrine. We arrived there very late in the day, and it was very cold. I had a long 600mm zoom lens on the camera, and I was shooting stuff very quickly.

When I got back, I gave all the footage to Tim and said to him, “Can you look through the rushes and give me some feedback?” and Tim said, “Yeah, I’ve watched it, but there’s a dead person in one of the frames.” I was unaware what he was talking about. In one of the shots there was this poor guy looking directly into the lens of the camera. It was a chilling and freaky moment. It made me think about the detachment of filming, the gaze of the camera and the juxtaposition of something so horrific in such beautiful scenery. This is a famous place for suicide, apparently.

That sent me on a bit of a philosophical trip. In this interim period, when we couldn’t work on the project, I read Jacques Derrida’s book Archive Fever. In that, he talks about this idea of the death drive, and the need to film and to record. For me, a lot of those ideas are implicit in the piece, which became much more than just thinking about a single culture. That’s what led to the idea of the AI.

Christopher: It’s scary when you think about how quickly this stuff is becoming used in all sorts of ways, consciously and unconsciously, and how AI models are trained, and who’s training them. In cinema, we have the same problem: whose gaze is this?

Going back to the meaning of the archive, it was western production companies going into Japan, looking at it from an orientalist perspective. In the same sense, we’ve got the same thing, neo-colonialism with AI. It’s rich, white technicians building models based on our own biases. All that stuff’s there, but it’s got the potential to be even more dangerous in a way. For example, we used this model called YOLO, which is used a lot for driverless cars. There’s a whole bunch of these models that you can explore and experiment with.

There’s an interesting website called Runway, which is accessible for artists and creative people. We worked with a company called Artists & Engineers to write some software that would utilise some existing models. These models are mad, and you think oh God, I’m never going in a driverless car. It’s spookily accurate and crazily not accurate at the same time. That’s the situation now, but who knows what it will be like in 50 years?

Tim: These models are old as well, aren’t they? I mean these are the models that are available free and to people like us, whereas I’m sure that the technology has moved on a lot since these were created.

It’s both sinister and exciting, because the technology could have dark uses and implications. At the same time, it can open lots of possibilities for freeing up some of this material. I use it on my phone, looking at photos and searching. It’s really nice when I’m going to look at some pictures of my daughter when she was small, and it just throws up all the photos. Or looking for a family member in your photo stream, which is quite a nice and positive thing. It’s about treading carefully around who’s going to write the rules.

Tokinokawa is open daily from midday to 10pm at BFI Southbank until 30 January.

The original films that were used for the Tokinokawa installation are newly restored films from the BFI’s Japan on Film archive and are available to watch for free on BFI Player.