Paul: Welcome to The Data Movement. I’m Paul from Seagate, and in this episode, I’m talking to Eric Weaver from the Entertainment Technology Center, and we’re digging into AI filmmaking and the future of emerging media. Let’s get into it.
Here live at NAB. This is The Data Movement. This is my good friend Eric, principal of AI and Emergent Media at the Entertainment Technology Center at USC. I know you have the most rich, interesting background, and you dabble in so many different areas, and I want to dive into it, but maybe let’s just start there.
What does one do with their days as principal of AI and Emergent Media?
Erik: So, we get to explore what’s new, what’s going on, what’s the cutting edge. And I’ve been there for a little while now, and my kind of title or what I looked at has really changed over the years. I originally came in looking at cloud, and how does cloud workflows work?
And then got into HDR and other things. Virtual production — I was very big on that for a long time. But I give away an annual grant, and we do a production, and we push the boundaries of all the technology out there. So we’re just finishing one up called The Bends, and it’s a purely generative film.
And then we’re also finishing one that’s called Pathways, which is a hybrid film.
Erik: And last, we had a project before that that just got into Tribeca called ChickaBOOM, which is an animation one with Asteria Studios. So I get to play with the tools and have fun.
Paul: Yeah, so right on the edge of what’s emerging in terms of tools and technologies and how you can integrate those into visual storytelling.
Erik: Absolutely.
Paul: Sounds like the best job.
Erik: It’s a ton of fun.
Erik: Yeah, it’s kinda curious how it came about though, because originally I did a lot of working groups and best practices and standards, and Universal came to me and said, “Let’s just try this in a film.” Because now all of a sudden we have context to what we’re doing. So, getting that kind of opportunity to dip your toe in, see how it really works. It’s a great place.
Paul: Generative film. Do you find that when you are talking about this in filmmaking, videography settings, that it’s controversial?
Erik: When we do our projects, we are filmmaker creative first, period.
I have a DP who’s analyzing every frame. I’m not doing prompt and pray. This is not a slot machine. Everything is storyboarded and animatic-ed out that we’re doing. And we also do projects that are all impossible or near impossible for a traditional filmmaker to do. So The Bends is about a blob fish, which is a deep-sea fish, and how it changes as it goes up the depths and becomes a very ugly fish as it comes to the surface.
But it’s chasing something beautiful and shiny, and it changes them. So that’s the basic gist of this deal.
Paul: Yeah. It’s like any generative experience seems to be based on the context, the data that it’s been trained on, but it has access to generate the output. So how, how are you thinking about that?
Erik: In our situation, we did a couple steps here.
So the writer director actually had an animation background, so she did just some of her own sketches. There are plenty of tools out there where it can create things for you or can take very rudimentary sketches and make things, or just look at your script and come up with things. But I think in AI you got to figure out where your boundaries are.
And I think the first boundary for us is we don’t really affect story or writing. It’s an individual writing. And we’re creating the first frames or designing them in a more traditional manner. That’s a little unique to some people who might want to go a little more AI in. Not quite our flavor yet.
And from there, we go and we create things like LoRA models and series of prompts. A LoRA is a low-rank adaptation. So in this situation, a blob fish is not something you can just kind of find on, say, VO3 or any normal model, because it’s a very unique fish. So, we had to go to oceanic institutes that had images, and we took those images and then basically created three-dimensional models and cleaned them up. oceanic institutes that had images, and we took those images and then basically created three-dimensional models and cleaned them up. oceanic institutes that had images, and we took those images and then basically created three-dimensional models and cleaned them up.
And then we wrote a script that turned it around and lit it in different ways. And then we had to burst to mass compute, because we didn’t have a large enough VRAM footprint to create the models, and so we created the character. When people would log in — because they all had a virtual machine — they would see things like their models or the tools they could use, like VO3 and some of their base prompts. So they had everything ready for them.
In this situation we’re also running ShotGrid, so we have it completely broken down. We had 86 shots across eight different scenes, and a woman named Verena Puhm actually managed the creative team, which is five people. She said, “You get these shots and these scenes.” and handed them out. Then they would log into the virtual machine. They’d have access to everything, and they would work on those shots. They would look at the animatic, and they would be designing to that.
Paul: Interesting.
Erik: And then when they thought they had a shot, they’d raise their hand.
And basically, the director and the DP would come over and look at the shot and say, “You know, I really like this,” or, “I don’t like this,” and, “Can you change this or change that?” What you’ll find is usually you can get about 80% of the way. Once in a while you get a little higher on a finished frame, but sometimes you still have to go in and do in-painting, out-painting, a bunch of manipulation to the shot to get it exactly as the writer/director wanted.
Paul: How different is the workflow that you’re describing from a traditional workflow?
Erik: So, it’s really interesting. If you take something like Chicka Chicka BOOM , which is going to actually premiere at Tribeca, it is very unique. Because it’s not this concept of prompt and pray. A lot of things make it look very easy.
But it is not as simple and easy as you might think. This is layers of AI being used in different things, from style transfers to procedural generation, through other components of the pipeline, and then building it together, which saves a lot of time in some of the animation pieces. But it’s a very sophisticated hybrid use. So Asteria finished that film. It’s now theirs with C. Greg Patterson.
When you get into purely generative, it’s a lot like animation. You know, you’re designing things. Even though it might look live action, you’re designing a little bit more of an animation workflow. When it comes to the hybrid, that’s more traditional production. You’re just trying to capture your performances and what’s critical there.
Paul: You mentioned earlier that you’re associated with the university. The university has associations with the big studios. They’re very curious and interested about how these tools and technologies can… I guess they’re not doing it for fun. Right? They’re doing it because they see opportunity and they’re curious about how these things, these projects that you’re working on can be integrated into bigger projects, right?
Erik: Yeah. When we originally started, they’d really put a pretty hardcore list of, “I want to test this, I want to test this.” Now they’ve kind of lightened up and said, “Go do what you do and show us what you come up with.” So, we’ll go and do a roadshow afterwards to the studios and kinda give them a breakdown of how do we execute this? Or what was tricky here? Or what are we actually finding that doesn’t work? We break a lot of systems. It’s pretty tricky. Or we really push some of these tools very hard.
Jetset’s a really good example of this. Jetset basically allows you to map people and things into the environment. And it requires a level of precision that is very, very articulate and hard to do. Unlike traditional tools that map to virtual production stages like a Mo-Sys or Stype or anything like that, those have a little more flexibility in exactly what’s being played. But this situation when you’re marrying the worlds requires a lot of nuanced detail.
Paul: Yeah. How do you feel about this as a direction for art and creativity in general?
Erik: I absolutely think that this is going to empower a lot of more creativity. I think that it is going to bring up the value of what more towards independent and independent productions can do. A lot of people like to say, “Well, this is time and cost savings.”
I wouldn’t ever phrase it in cost savings. I would say a little bit more like your budget can stretch further with this. If you learn how to do this and do this appropriately, you can get a lot further with the budget we have.
You might be able to do a $30 million film on basically a $10 million budget. Matter of fact, we were working with one particular client who’s actually looked at it and they had an indie budget, and they had a scene where it was going to cost them a million dollars, and all they had was a $6 million budget.
Now we could show them how to execute this scene for $100,000 … $200,000 at the most on that side. And probably savings overall on their production were about 16%, more B-roll, second unit, kind of other things, and it was about a $1.5 million savings on a $6 million film, and that makes an impact, you know?
Paul: Yeah.
Erik: And so the number one thing is creativity. It does push the boundaries of what you can do creative-wise.
First thing you do normally when you get a script is you walk through it and you say, “We can’t do that shot. We can’t do that shot.” They’re too complex. They’re too technical or too expensive. Now you can do a lot of these shots. So that’s what’s kind of exciting to me, being able to write unbound.
Paul: So, you’re firmly in the camp of this is augmentation to the creative process rather than replacement.
Erik: I absolutely am, but I think that you really still have to keep those solid people in there as department heads. You still need that structure and that organization.
Erik: You need a production designer. You need a DP. You need all these people still. They work with new or different tools but their job is fundamentally their job. They have a sophisticated eye. They know what they’re doing. They know what they’re looking for. They have a reason they put a plant here and a that there and that costume’s on them and this costume isn’t. They are experts at their job, and they still need to be involved in the process.
Paul: When you think about the infrastructure underneath the workflow, what are your initial observations and maybe even predictions about how disruptive this stuff that you’re doing is going to have an impact on the infrastructure that supports it?
Erik: That’s actually an interesting point because creatives are not computer scientists. And so there’s an odd balance here coming. For example, when we were doing The Bends, we ended up — depending on how good somebody was — they’d have 50, 60, 100 shots. Trying to get it right.
Then we’d have to save so many. There was a huge amount of data that starts generating, and you’re like, “Well, I want this. Do I want that?” And you’ve got to sort through all of this, so that’s a very sophisticated thing. The other thing you’re seeing is a lot more metadata across your different tools and pieces, which is sophisticated. And then going to be able to use things like cloud or localized storage in appropriate ways. That becomes very tricky because you have a lot of mediums and you’ve got to balance it all, and make sure you don’t lose content … you’re not doing things wrong. All this stuff.
Paul: Do you see it creating new challenges?
Erik: I do, but I think with this whole generation of AI, there’s a lot of tools that are going to allow you to simplify this process.
Erik: You know, vibe coding and other things. The ability to build these pipelines is going to just all off a sudden spin up much faster. And it probably won’t be individuals doing it from their own production. There’ll just be a couple more booths here at NAB next year that help solve these complexities or make them simpler.
Paul: Well, you mentioned storage and data, right? Do the emerging workflows and things that you’re working on, do they have kind of a compounding impact on storage requirements?
Erik: I absolutely think so, because you get more and more shots, and it’s more complex. And where have you stored this and what’s that going on?
For example, when we’re shooting hybrid we’ve got to marry the scenes and understand the lighting, and keep all of these different camera components and files and make sure they’re connected in some way. So things like the ASC MHL. ASC did a media hash list for creating manifest when you’re on set that makes sure that the people who get it later are all the same things and that it could be correlated to other files or a whole day of production or something like that.
Erik: What’s interesting is this is actually a correlation to SAG. If you are familiar with the latest SAG rules, it says that you’re allowed to use a digital double or whatever you want to call it for an actor, but there’s different, A, pay rates, and B, at the end of the production, you must delete all that data. It can’t be used for anything else, and it can’t go to a public model. This is really important. So private cloud, private storage devices, or private compute to run some of these things are critical.
Paul: So that’s written into the license agreement?
Erik: That’s written into SAG contracts for actors, but in our agreement they said the same thing.
Paul: That whole thing is just so fascinating, the digital IP and the future of licensing and how that gets governed.
Erik: In the future, everyone’s going to have a digi-double that understands exactly what they look like, their persona, their nuances. And how licensing works is going to completely change. How CAA or UTA deal with a client.
Erik: It’s going to totally change. You know, you used to have to … Say, for example, you’re a major actor. You might have to go to Japan to shoot the latest whiskey or cigarette commercial. Now they just license an avatar.
Paul: How far out is that?
Erik: Deals are being done today.
Paul: Yeah.
Erik: How it becomes mainstream? Probably two years. Year and a half ... two years before it becomes really mainstream.
Paul: Is that something that they have you thinking about?
Erik: In our next production, we’re going to have partners that are playing with some of this stuff. We might use it in the use case of B-roll footage or dangerous scenes. Things that I can’t get insurance or other things on. We might play with it in these ways. We’ll see how it goes.
Paul: What are you most excited about looking forward?
Erik: I think it’s not what I’m most excited about, it’s what I kind of want to tell people. Have you ever seen the movie Interstellar?
Paul: Yeah.
Erik: Do you remember where they’re on the water planet and the wave, they realize it’s a wave coming at them. They see this thing in the distance, and they’re like, “Ha, ha,” and then they realize it is a tsunami coming at them.
And a lot of people have talked to me and they say, “Oh, AI has messed up Hollywood.” And I’m like, “AI hasn’t hit Hollywood.”
AI –Yes, there might be some utility based tools that are changing, but on the creative side, they’re very, very few and far between actual productions that have implemented this technology. But that wave is coming at you. It is. It’s coming at you right now, and if you don’t understand that, you’re going to get hit real hard.
It’s pretty crazy. It’s okay if, for example, you were a physical artist and you drew things, and you didn’t want to go to digital systems. That’s okay. But you need to know what’s coming. You need to understand how it can either benefit you, or if you’re just going to be out and put your head in a hole like an ostrich.
That is your prerogative. But it’s coming.
Paul: What’s your advice. They see the tsunami, what do you do about that? What do you do about that?
Erik: I think you should just go out and just do a survey of the tools out there. I originally used to like to suggest to people that they take one of these 31-day classes where you play with a different tool every day and just see if any of them are appealing to you.
But I might actually just go to ChatGPT and ask it to put together a program of testing for you. And say, “These are my interests. I want to look at tools that do these kinds of things,” and ask it to put together a program of, “Let’s play with this. Let’s try this. Let’s see what this is,” and so forth. And then go explore from there.
Again, you’ll find things that you really like and things where you’re like, “I’m not interested,” or “I don’t want to touch that at all.”
Paul: How far off are we from the first generative movie … Like silver screen Hollywood movie?
Erik: I think I think we’re probably nine months to a year from the first one. What a lot of people don’t understand are the complexities in making these things.
There’s still a whole lot of process. There is a film right now that is being shot that where they’re replacing the backgrounds. It’s live action. I can’t remember the name of the person doing it, but Casey Affleck is in it, and I think they’ve done most of the shooting already.
Paul: It’s’ a hybrid. It’s not pure generative?
Erik: It’s hybrid. I’m honestly not going to be a proponent of pure generative. It’s not really my thing, unless you’re doing animation or something that’s non-actor. You really still want to value the performance and the nuance of that in my personal opinion.
But a purely generative one is going to be a little longer. A few of the people who promised AI shorts or animation pieces are realizing how hard some of the components are. I wouldn’t suspect a genuine one, unless it’s something light and easy, to be out for another six or seven months at least.
There’s a lot of nuanced problems that we run into when you’re building home these pipelines, like locked weights or closed environments. You might use the same prompts and you’re getting a totally different answer the next day because the model changing or, again, a lot of different issues there.
So, you got to solve a bunch of problems behind the scenes.
Paul: But there’s funded projects today that are being worked on that will hit the screen within the next year.
Erik: Yeah.
Paul: Interesting. The comment that you made around the human performance being important.
Erik: It’s critical.
Paul: I guess the emotion is not replaceable.
Erik: Don’t get me wrong. I think that there’s going to be some really good versions of things coming out. But I personally like actors, you know.
Paul: Think we’ll have pure digital actors?
Erik: Absolutely, but in my opinion, that’s simply a puppet. We connect with Muppets or puppets sometimes, and sometimes we don’t.
It’s about how does that impact you or make you feel human?
Paul: Right. Which is, I guess that’s the purpose of arts and media is to connect and to resonate.
Erik: And that’s what I think a lot of people get lost is. This technology is not just for technology’s sake, it’s to serve a story. Right? If you are not serving story then I’m not sure why you’re doing this.
Paul: I like that. Start with a story, and then I think blending the technology and the creativity for greater output, and then lowering the barrier to entry for more people to be able to create.
Erik: Because again, there’s a lot of communities who have traditionally very hard times telling their stories. Younger generations, unique regional groups or people… There’s just a lot of those things where it requires so much resources to make some of these films that the story got lost in storytelling. Now some of these things can be shown, can be understood, can connect with people.
Paul: What’s next?
Erik: I’m actually in the middle of winnowing down to our next script. We do our final pitches on Friday of this week.
Paul: Okay.
Erik: And so, I create this award committee. It’s a lot of senior people. And I’ve got four directors, and they’re going to just pitch. They’re going to pitch their stories and say, “What do we want to make?”
Paul: Is it student directors?
Erik: They’re all people who’ve graduated with a master’s from USC, with diversity, and they get to just, you know, share these things. This year is different because we normally have them write the stories and direct. But instead we went to Mary Sweeney, who’s the head of the writing department at USC and David Lynch’s former partner. And she found wonderful scripts out there with recent graduates. Now they’re going to each pitch one of the stories. And we’ll pick one of them, and then we’ll get to making it.
Paul: Very cool. Eric, what question should I have asked you that I didn’t ask you?
Erik: I don’t know. I think that you were correct in the idea of data and metadata and complexity and storage. These worlds are just going to continue to get more complex. And it is critical to be able to figure some kind of layer over that manages that complexity. So paying attention to that is absolutely important as we move forward.
Paul: Yeah, it seems like just a lot of computing power, lot of storage capacity in order to render the files, edit the files, archive the files...
Erik: Understand what files you have.
Paul: Right.
Erik: It’s just going to get bigger and bigger. It’s a bigger data problem.
Paul: Kind of like a multiplying effect on the data problem. Which maybe AI will be able to help with too, right?
Erik: I think it eventually will. Absolutely. We’re in a world where AI, vibe coding and other things, can sort this out. It’s not completely sorted out yet, but we expect tools to show up very soon.
Paul: Cool. Eric, it’s been such a fun conversation. Love your perspective, love that you’re pushing things right at the bleeding edge of what’s possible. I’m going to be following closely to see what you guys are building over there. And I’m going to watch The Bends.
Erik: Awesome. We also have ETC official on Instagram. We run a lot of videos there. We publish stuff on LinkedIn as well. We’ve traditionally done a large white paper at the end, but we’re trying to do a lot more real-time media out there so that people can see things, because stuff ages really quick out here.
Paul: People can find you at ETC Instagram?
Erik: Yeah. @usc_etc_official. Try that out. It’s somewhere along that line.
Paul: Very cool. Thank you, sir. Appreciate you. Thanks for joining us.
Erik: No, thank you.
Paul: That’s it for this episode of The Data Movement. Thanks for listening. Find us wherever you listen to your podcasts for more conversations about how data is moving the world forward.
