Redefining Creativity

Fresh off the back of their headline talk at Semi Permanent Sydney, Patrick Clair and Raoul Marks talk with John Gibson about a new way of creating.

Patrick Clair and Raoul Marks are Emmy Award-winning directors and title designers who have created titles for Westworld, True Detective, Call of Duty, Wired, Watch Dogs, Xbox and MTV.

John Gibson is a director at Faculty, a leading artificial intelligence consultancy based in the UK. Prior to this, he worked in strategic projects for 10 Downing Street.

In conversation with Christopher Barker, co-curator of Semi Permanent.

“The tracking software we use to infer movement in camera footage was the same software and algorithms used in missile targeting systems.”

Patrick Clair

Why don’t we start with how you became interested in the topics we’ll be talking about today?

Patrick Clair: I’ve worked on developing longer format stories that explore the ramifications of big data and AI, and consulted with television/film/video game companies in how they they portray future-conflict scenarios as accurately and interesting as possible. That led me to work with different showrunners in entertainment to create title sequences for shows like True Detective, The Night Manager and The Crown. One of these higher profile titles was Westworld, which has brought us around the topic of AI, the jumping off point for today.

John Gibson: I started out in various policy roles including a few stints in the Prime Ministers office here in Downing Street, eventually being asked to look after David Cameron’s interest in the impact technology was having on jobs. This emanated from conversations he used to enjoy having with then-President Obama; I’m told it was a favourite topic of downtime conversation.

At Faculty we build machine learning models for a very wide range of different clients from transport to health and counter-terrorism. Arguably the most salient work we’re doing is with the Transatlantic Commission on Electoral Integrity which was set up by Anders Fogh Rasmussen (former Prime Minister of Denmark and former Secretary-General of NATO) to protect the integrity of elections in the West from the threat posed by synthetic media (the kind produced by artificial intelligence).

Redefining Creativity-2 — Raoul Marks: True Detective was the first TV titles I’d worked on. Patrick had designed some beautiful boards using photography from Richard Misrach’s Petrochemical America; the tricky task was translating the still into the world of motion. We resolved on a form of photogrammetry camera mapping, where the compositions were projected onto simplified digital geometry. This enabled us to create limited scope camera moves through the photographs. The resulting effect kept a beautiful dreamlike quality half-way between the classicism of a photograph and the immediacy of realtime motion.

I’m interested in the crossover between your work and how creativity trickles from the policy level, and how policy makers consume and process fiction as an input.

Patrick: A show like Westworld is a really great example of science-fiction exploring this century old question of what’s going to happen when general artificial intelligence emerges. There are more creative ideas (like synthetic brains) attached but ultimately it’s asking what happens when extremely sophisticated narrow AI becomes general AI.

John: Sci-fi always gets there first and I’m grateful for how it influences the perception of technology. It’s much better at painting a picture of technology than a research paper, and until someone has the gadget in their hand the best impression will be something they saw on the screen or read in a book. Sometimes you will be explaining an idea to someone and they will say ‘oh yeah, like Black Mirror?’ and you know they’ve got it. Sci-fi indirectly frames the way our technology is received in other domains, including but not limited to politics and business.

Patrick: We’re also talking about deepfake videos becoming more relevant to political discourse. And that strikes me as strange from a VFX angle because this is the stuff we were trying to do from the beginning; the nuts and bolts of ‘faking’ an image. Suddenly, we see algorithms recreating the work that we do. This wasn’t supposed to happen. It wasn’t supposed to be the high level creative jobs that were getting taken by algorithms and yet we’re in this conversation.

Redefining Creativity-4 — Raoul: The digital veil was a particularly enjoyable scene to create. It references heavily the Catholic imagery of the Madonna; head tilted back looking up to the heavens. Except what would be a fine lace veil has been replaced with a digital shroud. The complex filaments of circulatory mimics the fine embroidery of lace.

John, I’m curious how you think narrow AI is going to change the future in ways shows like Westworld aren’t able to interrogate?

John: I often describe to clients that it’s unrealistic to expect a machine learning model to achieve superhuman levels of capability in terms of performing a task. But it may be able to perform the task as well as a good human a million times a day (rather than 10 times a day). In a system where you have 50 people doing a job, some of them are experienced and some of them are bored and you see those variations in productivity. You can train an algorithm to be as good as the best person, and it will always be that good.

To what extent can AI be genuinely creative then? A lot of what we’re talking about deals with impressions of an environment or voice. But what about the theoretical possibility of generating new personas and environments?

John: The limitation of this debate is that you end up defining or abstracting creativity. Human creativity interprets everything you know within bounds, but so do algorithms. In that respect I’m skeptical.

Patrick: This is what I’m trying to step around. I’m not the least bit concerned the computer is going to take my job; it’s an important debate to have. But I’m more fascinated by the question of ‘is this real creativity or not?’ And I wonder if there is this middle phase in the process we’re ignoring which is narrow AI. I was impressed by the NVIDIA project ‘This is Not a Face’, which automatically generated a fairly realistic looking face every time you clicked. And John, I’m wondering that if you can already create a photo-real face, what are the other bizarre things narrow AI will develop in the next decade?

John: That algorithm is taking all the components that constitute a human face and figuring out how it can be mapped together, then combining them in lots of different ways. But that’s what a human does when they draw a face, right? If I tell you to draw a face, you’ll pull together components that defines a human face from memory, then draw it. You’re not born with the ability to draw it, you learn; so does the machine. So the difference between human and algorithmic productivity just becomes a matter of definition.

Patrick: A colleague of mine created an AI that wrote horror stories. It was fed 25,000 horror stories, then you could feed it a sentence and it would complete it using a probability engine. And while it might credit the perspective of an author, it’s not conceptualising a horror story. What’s the power of that when you replace ‘horror stories’ with ‘Obama speeches’?

John: The key difference is that Obama understands the context and uses it to sell a story instead of building a series of probabilities that makes something coherent. If I were to use a deepfake to cause trouble, what would I do? Well, I’d have to find an opportunity. I’m not going to go for Donald Trump because whatever you put out there he probably said something worse that day. Maybe I choose Theresa May because she’s in an incredibly complex situation with different factions. So if we were to choose the right messaging that really pissed off her Brexiteer flank but also alienated her remainers and we are able to push the message out on the right forum, her government wouldn’t survive. The deepfake is just the execution of a cunning plan, but the genuinely creative part is knowing that it’s this person, in this place, with this message, at this time, attacking this vulnerability. That’s the human part and the application of narrow AI is to create a video that looks like Theresa May bitching about Brexit. So it’s providing new capability to pre-existing human ingenuity.

Redefining Creativity-5 — Raoul: The piano player’s hand was a very tricky task from a few perspectives. In season one their movement is more rigid and machine like, partly due to the limitations of the software at the time but also a stylistic decision. In season two we finessed the movement to be more fluid and lifelike. Partly to reflect the evolving character but also just because we had the time to make it better. Interestingly, message boards started discussing the evolution of the shot and what it meant for the show. The lesson here is that the fans pay very close attention to details.

Redefining Creativity-6 — Raoul: The lovers scene from Westworld was a key moment in those titles. Patrick’s direction was that the light should reveal a more resolved clean flesh, and as that light falls off into darkness we see a glimpse of the more anatomical forms beneath. It’s a nice play on the beauty of love vs. the grotesque flesh and sinew beneath the surface. It seemed fitting for a show about dualities and false impressions. This core concept was carried through to the second season’s titles, but this time with a mother and her newborn infant. With the depiction of a new human life we wanted to be particularly careful to not go too far. There’s actually a large amount of rather unsettling detail in the models: bone, muscle, brain tissue etc. are all exposed but have been almost completely hidden in darkness.

Which leads us to this post-truth era we live in, largely driven by technology that gradually erodes trust in traditional institutions. How do we educate the general population to trust (or distrust) what they’re looking at?

John: The public needs to become more aware, and sci-fi has a role to play in that. We are developing a lot of synthetic content while making sure the public consumes it in the right context. Our hope is that the first time they hear a fake Trump audio snippet it’s via the ABC or BBC rather than something placed by the Internet Research Agency on Reddit three days before an election.

There has (to some extent) been a mass democratisation away from established media to social media. Now someone share’s a video and, even if you can see it with your own eyes and hear it with your own ears, you need to ask ‘what is my source here?’ When you see something on the BBC and it’s got their branding and commentary and a news anchor you can trust, that’s what you need to look for rather than the fact that something is just video. Video has been associated with trust for 50 years, and that’s about to end because of AI.

“Video has been associated with trust for 50 years, and that’s about to end because of AI.”

John Gibson

And then Patrick, on the culture side of that question do you think there’s a lot we can do to defer that influence?

Patrick: Narrative is about giving us a space to think about the choices we make. The more entertainment engages with real world concerns, the more interesting and relevant it is. Sometimes that can be extremely gritty like True Detective or fantastical like Westworld, but it’s important for program-makers to ask ‘why are we telling this story?’

Westworld asked some big questions around the nature of consciousness and intelligence. Having worked with Jonathan [Nolan], Lisa [Joy] and the broader team, I know them to be incredibly interested in the relevance of these questions and the cutting edge science behind them. There was no cynicism, just a genuine hang-up for knowledge of how to translate that
into a story.

Redefining Creativity-8 — Raoul: For The Crown we had the challenging task of revealing the crown but never actually showing the fully formed structure. It’s a simple concept but actually has a lot of hidden connotation within the images. For this section, we try to evoke a prison like structure — a gilded set of bars encroaching around the viewer. It references the real restrictions and heavy expectations placed on the young Queen.

Redefining Creativity-9 — Raoul: For the opening few shots we were referencing the elemental origins of the Crown. The position of British monarch could be argued as being almost naturally evolved from that particular landscape. The somewhat threatening natural form of this golden arc slowly develops into more rigid and manmade structures as the sequence continues.

There was also a great deal of humanity attached to these machines. John, are there conversations in your work that deal with AI on an emotional level?

John: A lot of emotions, specifically. In order to make sure you don’t inadvertently build systems that have catastrophic consequences, you need to think very deeply about what an automated system looks like and how it behaves, what bias it delves in and so on. You don’t need to be in the realm of futuristic possibility for that to be real; it’s technology people are building today.

Take the Trolley Dilemma and apply it to autonomous vehicles: Do you kill a group of people or one person? The car would take a course of action recommended to it by its code which was probably written by some young coders in Mountain View who didn’t even know they were writing code that would ultimately decide who lives and who dies. This is is why we approach things not just from a systems perspective but a societal one; it requires the public to debate. Frankly, the best way for people to contemplate is by reading books or watching movies. It links back to the role of hope, for the propagating of these ideas and seeding the right conversation.

Patrick: That’s exactly the kind of narrative inquisition that I’ve been trying to get my head around. From a selfish point of view there are all sorts of really compelling narrative questions to be asked and social dividends that can be paid by asking them.

John: Patrick, I’ve got a question for you. I feel the most compelling content is the stuff based on the now rather than the distant future, because if it catches me right it just sends a shiver down my neck. I mentioned Black Mirror earlier as an example. What are your tips for who is doing that well?

Patrick: I’m also a huge fan of Charlie Brooker’s work, not just across Black Mirror, but going back to his more absurdist stuff [which is] really fascinating and culturally aware. Working with Jonathan Nolan was fascinating, we would have meetings to discuss certain frames in the titles which would turn into two hour ramblings on AI philosophy where you could see the producers freaking out about the damage to the budget as we tried to incorporate those themes.

I think the sci-fi I most enjoy comes from a curiosity for the world around it. The show runners I’ve worked with have these broad ethical human concerns and interest in what’s going on. That’s not limited to science fiction: I had a great time working on The Night Manager for a team who had interesting influences in geopolitical terms, defining the forces shaping American business and the arms trade. And one of my favourite films is Gareth Edwards’ Monsters, which was a really reflective, engaging film that explored the situation at the US southern border back in 2011. Whatever happens to VFX, ultimately having smart insight and good storytelling makes a film.

“Heavenly bodies would tilt and rotate glacially with the scale and weight of the world.”

Raoul Marks

I feel like we should finish on a positive note. What are some net gains we have to make from everything we’ve spoken about and some amazing things that are actually going to happen?

John: The class of data science or machine learning is expanding the range of questions that computers can provide you answers. You can draw a comparison to the electrification of the economy. Electrification didn’t make economies grow because people traded candles for power lines, it just meant everyone got a little more efficient, discovering new ways of doing things and new product categories and new types of human possibility. AI will do the same thing, and a lot of it will be of value.

Robots don’t make the content, it’s the human factor. Some will use it to cause trouble and spread lies that ruin elections, some will use it to create beautiful artwork and powerful video games. And so I think that there’s every reason to be optimistic about it. It’s just about making sure that as a society we enjoy the benefits and spread them out as widely as we can and make sure that we’re keenly aware of the downsides, as with any other technology.

And then Patrick, from a design point of view are there many gains to make?

Sometimes technology we use in everyday life comes from unexpected places. One example is that the tracking software we use to infer movement in camera footage is the same software and algorithms used in missile targeting systems. I’m not scared or concerned by these technologies, but from a storytelling perspective I’m aware we’re citizens of a society we don’t understand. It’s always been stories that help us understand, whether it’s Vietnam war films that brought those atrocities to light, or today it’s films and television that help us understand big data, narrow AI, general AI, social media and how it shapes the world around us; contemplation in the form of fun-to-watch film and TV. The more we can do to help people consider the world around them, the better the creative industry will be at serving its purpose.