Paul: Hi, I’m Paul from Seagate. Welcome to The Data Movement. Today, I’m joined by Richard Cullen, technical director from Blink, a company that focuses on live production for the biggest musicians and artists on the planet. Let’s get into it.

Welcome to the Data Movement. We’re here today with Richard Cullen from Blink. Super great to have you on the show.

Richard: Great to be here.

Paul: We’re going to be diving into some of the most leading edge, cutting edge, I would say, media workflows probably on the planet.

Richard: Well, we are NAB and like where else would we have that kind of conversation, right?

This is probably the perfect place to talk about that stuff.

Paul: Absolutely. Before we dive into what you have been up to, Richard, I’m super excited, can you just tell me a little bit about Blink — some of the background, some of the story — and what brings us to this point today?

Richard: Blink is a multifaceted post-production, multimedia company. We’re based out of Burbank, California. We have heritage from 15, 20 years ago, I would say now, from London, where we started in screen advertising, and then we moved into multi-cam editing of music festivals.

Started getting into visual production for the tours, working with a lot of huge UK artists before moving over to Burbank about 10 years ago now. And ever since then, we’ve just exploded. We’re doing bigger and bigger things. We do Coachella headliners. We’re very invested in visuals of the Sphere at the moment. Every day we just seem to get more and more things going on.

We keep growing and growing, and we’re very excited for the future and what we’re doing.

Paul: Yeah. So, you mentioned the Sphere.

For people who may not know what you’re referring to that context, explain what that is.

Richard: It’s this incredible dome arena where the screen of about 180 degrees wraps around the audience, and it’s a huge immersive visual experience where some of the largest bands in the world are currently in residency.

They have films and stuff going in there, and it’s a phenomenal space to see music and art come together, I think.

Paul: Describe the experience of being inside the Sphere. I’ve only seen it from the outside.

Richard: The first time you walk in, it is incredibly disorientating. I think, because you just walk in and it’s this huge screen just like up and over you. And depending on what you’re looking at, right? It can, it can just really discombobulate you if ... I went to go see U2.

Paul: Oh, amazing.

Richard: Right? And I walked in there, and their walk-in look was like a concrete wall.

And I sat down, and I was looking at it, and I was like, “I don’t know if I can actually be in here,” because I felt incredibly compressed. It felt like it was a sphere that had just been shrunk, pushed in half. It was a very odd experience. But now, it’s day to day. I’m there all the time.

It doesn’t bother me anymore. But it’s definitely a venue where if you’re walking around while something is playing, like the camera moves and tilts ... you better hold onto your beer. Because you could very easily just go, “Oi.” You know?

Paul: You guys at Blink get brought in to manage the full end-to-end production workflow. So, you’re collaborating with the venue, the technologists at the venue and also the artist to bring the vision to life.

Richard: Yes. You know, we work very closely with the people at Sphere Studios. We work very closely with artist management and the band creatively.

We then also then manage all of our team of artists, just to bring this vision to life. Everything about that venue is incredibly hard. You need to surround yourself with good people and good teams to be able to do it. You know, and good technology as well. This is a very professional venue.

It’s a very professional environment, so you need professional people to, to do it.

Paul: I imagine it’s not a plug-and-play workflow or setup, right? For something, a venue that unique. What are some of the considerations as you’re going through that?

Richard: Any venue or tour that we do, there is no plug and play on any of them, right?

Every single one of them has its own unique properties, its own unique challenges. So, with something like Sphere, it’s very much making sure that we have vast amounts of storage available to hold all this data. You know, we have incredibly fast networking. It’s just a lot of power that you need to be able to handle things of that size to be able to work efficiently, right?

Making sure you’re all set up for that is setting yourself up for success. And even then, it’s still a bit of a struggle at times.

Paul: But what was the path that brought you to develop those capabilities?

Richard: Well, you know, we have a history of producing great content for bands, right?

We do Coachella headliners, we do Las Vegas residencies or stadium tours, and things like that. So, Sphere is the next logical step. If you think about it, it’s just a screen, right? It’s the same thing, it’s just bigger. And so, you know, we had to really think about our own pipelines, our own workflows of how to translate what we do day to day on anything else to something that is, you know, exponentially like 10 times bigger, right?

And so, we partnered with Scott Miller from MDLR Technologies to help us build a render farm to be able to do that. And then also build all of our storage pipelines and our networking so that we can work incredibly efficiently, you know, across that canvas without too much of a pain point for everyone, right?

And because a lot of our teams are scattered around the world, right? We’re quite a good company for our artists to be wherever they want to be. So having all of those guys work to a single source of truth, to be able to do their work, you know, whether they’re in Mexico, Spain, England, so they can just bring everything together and to make it look, look as cool as what it does.

Paul: Yeah. Nice. As you were just describing everything that goes into a production of this scale, the thought popped into my head this is the epitome of the data movement, right? Walk us through the journey of the data, the files that you’re handling in this workflow that you’ve been describing.

Richard: It’s just needing vast amounts of storage and the speed because cause a frame for Sphere is a gigabyte, if not more, versus a regular tour that we may do.

Paul: One frame.

Richard: One single frame is one gigabyte. About one gigabyte.

Paul: Okay.

Richard: If you think about how big is a song at that point, you’re in terabytes just for a final render of the song. And then obviously all the pre-comps and the projects, and all the edits and progress, it starts down here. And then when you get into full production, it just shoots, explodes to that. So, you know, huge, massive RAID arrays of drives. We use Suite Studios, they’re a very good partner and friend of ours, to help us move files from our remote teams when they’re just noodling on their own at home, to send files to and from the farm.

They’re our single point of truth. And then we just have a load of sync software for all of our bits and pieces so that everything is running constantly. There’s never a point where you’re just working and they’re like, “Oh, we’re going to hit sync now.” The whole thing from farm to venue to the remote artist is always constantly moving back and forth.

And, you know, that runs every couple of minutes just to keep up, but there will still be points where you’re just like, “I have 20 terabytes to move in two hours.” That’s when it gets a little bit hairy because you’re like … it can only move as fast as my read/write speeds are going to allow.

Paul: Right.

Richard: But the days of shuttling drives around, long gone. It just doesn’t work like that anymore. You just got to have huge networks set up to be able to send things down the pipeline.

Paul: What’s the choke point there for using a drive shuttle in a workflow like this?

Richard: For using a drive? It just wouldn’t be possible. A drive would just never be able to hold a single song. You’d be carting huge RAIDs around if you wanted to. Because everything is in frames.

Paul: Right.

Richard: Moving all of that around is always going to be your bottleneck. You need to do as many files as you can in parallel. That’s why File Explorer and Finder are not tools to even consider when you’re working in this way, right? Because 60 frames a second, a song could be 25,000 frames, right? And you’ve got to move all of that from one place to another to be handled and ingested and set up.

Paul: At Seagate it’s so interesting to see this play out in a real-world example. But at a macro level we study data creation. Data growth. Naturally. We’re a storage company, so that is something that we track really, really closely. And one of the fascinating trends where you look at longitudinally, is the size of the files are just growing and growing and growing. I think you said you’ve been doing this for close to a couple of decades. Can you speak to that?

Richard: You remember buying your first SLR camera when you were a teenager or whatever? And you would get a 512 megabyte SanDisk card. You know, 300 bucks and you’re like, “Oh my God, this is incredible.”

Paul: So much space.

Richard: So much space. It’s 512 megabytes. Right? And now we’re dealing with shows in the hundreds of terabytes. A Coachella headliner could be five to 10 terabytes.

Things have just gotten exponentially bigger because the technology to power screens has gotten better. Right. So, we’ve gone from SD to HD to UHD to 6K, 8K, and it just keeps growing and growing. And the more these things become ­— I don’t want to say affordable, because none of this stuff really is affordable, but probably more achievable.

People are going to build bigger, crazier venues, bigger, crazier stage sets to give the audience a show. And we have to accommodate that and figure that out, and everything just accumulates over time. And data retention is, I think, a big problem in our industry.

Especially in this touring world. Even with our multi-cam stuff that we do, right? It’s like edits, edits, edits. Changes, go back. Changes, go back. You’re just building all this stuff over time, and no one ever wants to get rid of anything these days either, because you never know.

Paul: Why?

Richard: I think people are just like, you never know when something’s going to happen. Oh, wait, I’ve got to go back to that thing. Or six months down the line, an artist is releasing a deluxe version of a record, or they want to have a special re-release for an anniversary or something like that. Like, “Oh, hey, can we open the edit back up? I just want to go in a noodle and fix a few things.” It’s like, well, we got rid of all of that stuff because we didn’t need it anymore, but now we’re kind of expecting. It’s almost like as a post-production company, we’ve almost had to also become a data storage as well to hold all of this stuff because things will come up all the time.

Someone will have a pitch for a tour, or a band will be doing something, they’ll be doing a special show somewhere, like, “Oh, hey, can we get that piece of media that we did six years ago?” Got to go find it, grab it, figure out what you need it for now, remake it, re-render it. You just can’t get rid of anything, otherwise you’re going to shoot yourself in the foot. As much as I would love to hit delete on so much stuff, I really, I can’t. Because I know the moment I do it, even if I think, "There’s no way they’re going to need this version 17 edit," the moment I get rid of it someone will be on the phone, “Hey, you remember when we did that cut? With that version in it? Can we get that?”

Paul: Would you say you have a data retention strategy? Do you talk about it in those terms? Or is it just part of the overall storage workload?

Richard: No, we should, and it’s something that especially this year I want to get a bit of a better handle on just internally, right? Where are we, where do we store all of our stuff? Because at the moment for our internal stuff, we have all of our QNAP systems, you know, with our Seagate drives obviously, in-house. We’ve got two 60-bay racks. One is our live storage, one’s semi-cold, as it were.

The live stuff, the most pressing stuff is also then backed up on our Google Drive account. So, it’s always in a couple of places. We’re always going to be hitting storage limits, you know?

And I would love to probably archive things to LTO tape, but even now I don’t know if I want to do that. Because once I archive it to LTO tape, I’ve got to keep that drive and that database around if I ever want to go back to it, and that changes every couple of years. So where do we put it, you know?

We’re going to have to start really thinking about, I think soon, long-term archive storage for everything. You know, because like I say, we’re just accumulating data across the board, and especially with the camera records from our multi-cam stuff, you know. Got to hold onto that.

Paul: AI.

Richard: Oh, no.

Paul: What’s your perspective? It’s so funny, when I ask people about it, it’s …

Richard: A visceral reaction to it.

Paul: Or actually, it’s going to augment the workflow, and it’s going to make life easier for us, and ... what’s your take?

Richard: I think there are two paths to AI. One is the creative use of it and having it generate ideas, which I’m not a fan of. I think there are people in this world that are incredibly talented people, have their own thoughts, their own views, that can produce way better work than an AI ever could, right?

And I think having AI do that stuff is taking away work from this industry, which is not great. On the other side of that, there are uses where you can have it do things that maybe take a long time. They’re a bit more menial tasks, maybe on the engineering side, like figuring out some code and stuff like that. Generally, I’m just not a fan of it at all, and I just think it should always have that human touch to it.

Paul: The role in the creative process?

Richard: Having it in the creative, I’m just really not keen on it, right? And I think there’s a danger, I feel like if style frames are generated with AI, you show that to a client, they can latch onto that.

Because the AI is never going to be able to do it in a way that’s going to work for a tour, right? Because artist notes get particular, edits are particular, timing changes constantly. The AI can never deal with that. But they can latch onto that style frame and be like, "Well, why doesn’t it look like this?"

Well, that’s because a computer did that, and it can only go up to 1080p, and your screen is 12,000 pixels wide. I don’t think there’s any AI engine in the world that could make a screen for that. Right? Although it does also dishearten me when people share their work on tours, things like that, and you look at comments on Instagram and Twitter, wherever, it’s just like, “This is AI slop, right?”

And I think people have seen so much AI-generated stuff, they have forgotten that CG work is a real thing, right? They just see 3D stuff now and just assume a computer has done it rather than some incredibly talented individual.

Paul: I was thinking about your data wrangling. Tagging content and archiving it, knowing where it should live, the level of accessibility it needs based on how you’ve labeled it and dated.

Richard: I think that use, it feels a little bit dirty still. But those are the things I feel that might be fine. I know Blackmagic are doing some really interesting stuff in Resolve with it, and so is Adobe and in Multicams and if you’re doing commercials and films, you could just search, “red dress,” and it will find everything that has the red dress in because you may not have had time.

I mean, who has time these days to go through and log all those clips and fill out all the metadata tags, right? That timesaving is potentially a good use of it, so it gets you into the creative part of it quicker.

Paul: Going back to the file size growth curve, what is your outlook for the future? Do we just see this continued growth, and do you see that on the horizon? And then, follow-up question to that is, what is the potential impact on these pipelines and compute power and network bandwidth? Do you see those things coming to a head at any point, just based on the speed and the scale of the file size increase?

Richard: I feel file size I would hope is hitting a plateau. A PNG’s never going to be bigger than what it is, right? I’m hoping that screens are going to sort of cap out at resolutions at this point. Do we really need anything bigger at this point, you know?

Paul: Is it 8K? Is that the target deliverable for most of what you guys do?

Richard: I mean, a lot of post-production is still UHD, right? And I don’t know how true this is, but the human eye cannot tell the difference past 8K. Something like that.

Even 4 to 8K is hard to see the difference. But once it’s past 8K, it’s like, what’s the point now?

And I think we need to stop. You know, there’s a point where it’s just like, what is it in Jurassic Park, where it’s like “your scientist was so preoccupied with whether or not they could, they just didn’t stop to think if they should?”

Paul: Right. Right.

Richard: That. Can all agree this is good? This is perfect. Let’s stay here, because if we keep making larger things, everything keeps getting bigger, things keep getting longer, demand keeps getting more, budgets get tighter, timelines get tighter, how are we going to turn it around, you know?

There’s Got to be that consideration of how much data something is, this is the timeframe we have to do it in, how, these two things have to meet in the middle. And someone kicking and screaming, “Hey, I gave a note. I need it in the run in two hours,” isn’t going to make it go faster.

People have to understand that. Yeah, I would like it to just plateau now, please.

Paul: Yeah. The dairy.

Richard: Our render farm.

Paul: Tell me about the dairy.

Richard: The Burbank Dairy is the render farm that we built two years ago with Scott Miller to help us service the sphere. Because you would not be able to render sphere content on a local machine And so we originally started with 50 4090 graphics cards that we lovingly referred to as the cows because, you know, we have this British humor. It’s the farm, it’s the cows, they’re producing cheese for us. We’re going to milk them for all they’re worth.

Paul: So, this is on premise?

Richard: Uh, no, it’s not on premise. It’s in a data center now down in El Segundo, near LAX.

Paul: Okay. You just rent a space?

Richard: So, we have partnered somewhat with this company called RFX who are already in there. And they help us sort of manage that, manage our storage. We share things. They help us spin up some artist machines for our guys to remote so they can access the farm itself directly.

And that’s really cool. It’s like walking into the scene at the end of Raiders of the Lost Ark with the Ark of the Covenant, you see rows and rows of data banks. It’s very impressive. We built that. We had 50 4090s that was able to handle a sphere show. We used it on a few other things and it was good.

And now we have expanded. So now we have 100 GPUs. We have a bunch of 5090s, we have a bunch of 4090s in there. We have a whole bunch of Ceph cluster storage. Got a bunch of QNAPs. We’ve got all the 100 gig networking and all that stuff. And I think it’s really useful for our industry to have this because it’s a very personable render farm for us.

We’re very hands-on with it. All of our friends in our studios, they use it. Everyone has access to it. We can monitor and deal with it, right? And it’s C4D, Redshift, Octane, After Effects, a whole bunch of that. And it’s fantastic and I don’t think we would’ be where we are now with a lot of our content if we hadn’t have invested, I want to say, somewhat of a million dollars, I think into doing that.

Paul: Yeah. It’s not cheap.

Richard: It’s not cheap gear at all. And the power it consumes is crazy. Yeah, we’ve got that now. We’re doing a bunch of other Sphere shows on it. We rendered some of the Coachella stuff on it.

Paul: So, you use those resources specifically or exclusively for Sphere content?

Richard: No. We built it because of that, but it is usable everywhere else. We did a Coachella headliner this year. We did Sabrina Carpenter. And we rendered some of the content on that. And what would’ve taken our guy 10 hours on his computer, which, you know, in the grand scheme of things, 10 hours to render a song, not the worst in the world. But we sent it through our farm, and he had it back in 10 minutes.

Paul: Okay.

Richard: It’s serious productivity gains. It’s serious efficiency. So, you know, yes, it does a lot of our Sphere stuff. And because of the nature of Sphere, it gets locked up rendering those. We could have frames up of two and a half, four hours a frame.

Paul: You’d use that when you’re really up against a tight deadline? To do a quick turn?

Richard: Yeah, quick turnarounds on everything else, but when it’s a sphere show, it’s up and it’s running, and I think it’s been rendering for about two months at this point. You know, it’s just constantly going. And the more people that use it and do these things, they see the benefit and the ease of this versus — and I love them, but going to Fox Render and the render farm over in China, those guys are fantastic. But sending them files via their pro- then downloading it, it’s just that time spent doing that stuff versus being connected to it, submitting it, and then it’s rendered, it’s already on a drive that you have mounted. You don’t have to push and pull it at all. It just saves so much time and headache for everyone. I think people really love that.

Paul: Are more and more of your client workflows going to start using?

Richard: Yes.

Paul: Okay.

Richard: The more shows we do, the more things we’ll just run through it. And I think it going to give people time back. And I’m hoping that that time saved on the render farm isn’t then spent on doing more notes and changes.

Paul: Right.

Richard: But then people can go outside and go to the pub. That’s my goal. The Burbank Dairy Render Farm should help people render their things faster so, they can go down to the pub. That’s going to be the website tagline, I think.

Paul: I like it. Is this just a natural evolution of infrastructure for media workflows, like where there’s big data volumes that needed to be processed and managed?

Richard: Yeah, I think so. It was something that we’d considered doing for a long time, and I think it was when we got our first Sphere job, which was the UFC fight.

That kind of made it, “This has to be a necessity now, right?” And I wish we’d built it sooner. I remembered long nights just watching renders go, trying to deliver it, and then have them fail, and it’s just, it’s a nightmare. It was just something that needed to happen, and I’m glad that we did it, you know, and I’m excited to give it up to all my friends to use to help them out.

Paul: So, you virtualize it.

Richard: It’s a hyper-virtualized environment. I don’t really know too much about it. That’s very much Scott Miller’s bag. But it’s a hyper-virtualized environment. We have virtual artist machines that the guys can connect and work on, which is just like working at a computer at home.

And there are Ceph Storage clusters everywhere, like the 100-gig network. It’s a very impressive thing that Scott Miller likes to say is built on hopes and dreams, you know? Because it is just like we’ve built this thing and now it’s a little bit unwieldy, but we manage it and it’s fantastic.

Watching something go, ah. Watching Deadline, we like to call it Deadline TV, it’s like some sort of Spanish novella. Just watching a project load up and just go, it’s very satisfying in a really nerdy way to see something render really fast. It just gives me a small amount of joy.

Paul: Well, it sounds like that kind of innovation is the future, right? People want to spend more time in the pub and want to get their deadlines expedited. So, what about, you mentioned how much power these things consume. What’s the cost-to-time saved ratio?

Richard: Very expensive. It just saves a ton of time, and the value of the money spent on the electricity, and the power, and the cooling and all this stuff just lets us work faster, right? And deliver on time, right? So, it’s worth its weight in gold, I think.

Paul: All right. Lightning round. Richard Cullen, of all the shows and productions you’ve worked on, which one are you most proud of and why?

Richard: J Balvin at Coachella, 2019. Two reasons. One, they moved the set up during the day so it would happen at sunset, so he could perform in the dark when the screens had the most visual impact, and two, because Variety called it the best set that Coachella had ever seen, and that included Beyonce.

Paul: Very cool. Compute, storage, or networking, which one should technical teams optimize for first?

Richard: Obviously, it’s got to be storage, right? Because without storage we can’t do any of what we do.

Paul: Without storage there’s no data.

Richard: Yeah. Right.

Paul: No data, there’s no show.

Richard: And without data, yeah, there’s no data movement.

Paul: There you go. Okay. Biggest technical mistake you’ve learned from, without incriminating yourself.

Richard: Never delete anything without checking.

Paul: Okay. There’s a backstory to that that I’m going to ask you off camera.

Richard: Yeah. We’ll take that off camera.

Paul: What’s the single biggest technical bottleneck you’ve already solved, and what’s the one that you know is coming?

Richard: Well, render times I think was a bottleneck, but obviously we built the Farm to help that. And the one that’s coming I think is just storage. We’re going to run out at some point, and then it’s always more. It’s always more. You’ve got to keep adding, keep adding, and cost of scale.

Paul: What’s your best advice for creatives who want to become more technical, or for technologists who want to serve creatives better?

Richard: Get a good understanding of the software you use, right? So, Premiere editors who are happy cutting, understand the underlying fundamentals of that program, how it works, why things work, why setting up something in one way is better than the other.

Just really get to know what you use rather than just being like, “Well, I just cut video all day. I can just go cut, cut,” and not have to think about it. A better understanding of it will make your life a lot easier.

Paul: Well, Richard, I appreciate your time this morning. Thank you. It’s been such an interesting conversation.

I learned a ton from you. It’s such an iconic feature, the Sphere. Being able to talk about what goes on in terms of bringing something like that to life, very, very cool.

Richard: No, thank you.

Thank you very much for having me. It’s my first podcast. And my first NAV, so it’s nice to do it here. You see a bunch of familiar faces and see a bunch of friends around here. It’s been ... It’s really cool.

Paul: Great to have you. Appreciate you.

Richard: Appreciate it.

Paul: That’s it for this episode of The Data Movement. Thanks, Richard, for joining us, and you for listening. Subscribe for more episodes about how data is moving the world forward.