1005: Programatic and Skill Based Video Creation With Remotion  🔗 **Source:** [1005: Programatic and Skill Based Video Creation With Remotion](https://share.snipd.com/episode/d03871ef-11cb-4a73-af88-3251f2d2fbff) ## Highlights - **Render Videos By Treating Time As UI Input** - Remotion uses React to make programmatic videos by treating UI state plus time as a render input. - Jonny explains you code a UI and change it over time, then render deterministic frames via Puppeteer and FFmpeg. Transcript: Jonny Burger How it works? Sure. ReMotion is a way to create videos programmatically. So you use React to essentially code a website, but then you kind of also have this dimension of time and you change the website over time and through that you can render a video. And I've been doing that since 2021. Found a couple of people who like to use it. And for a couple of years, we were all coding the videos manually. And now the latest craze is that the AI writes the code. And now it has become a lot more easier. Scott Tolinski Yeah. Yeah. And it seems like, you know, longevity of the project is really great. ([Time 0:01:40](https://share.snipd.com/snip/21ceaaa4-bd46-489a-ba29-c3ee5ea1e8c8)) - **Sustain Open Source With A Simple Commercial Boundary** - Protect open-source sustainability by defining a clear licensing boundary for commercial users. - Jonny set a threshold (organizations with four+ people pay) so hobbyists stay free while funding maintenance. Transcript: Jonny Burger So essentially, the idea has not changed since the very beginning, since 2021. I was in the situation where I already had a couple of open source projects that I maintained, like some indie hacker projects, and I was also working part time. And I really felt like taking on ownership of another project could stress me out. And I kind of sensed that this one could become more popular. And then I felt like if it becomes successful and all I get out of it is like issues and bug reports and stress, then I would not want that. And so I kind of put a clause in the license, which requires the bigger companies to pay for that and for everybody else. It's free and it's been going great. We've been able to sustain ourselves from that. We are now a team of three people. Yeah, a decision I do not regret. ([Time 0:02:40](https://share.snipd.com/snip/79177c98-abd9-4a1f-9f03-303f72616167)) - **Agent Skills Drove Explosive Adoption** - Agent skills (Claude skill + npx skills) massively broadened Remotion's user base by letting non-coders prompt video generation. - Jonny credits a tweet and the npx skills tool for the jump from ~125k to ~800k installs/day. Transcript: Jonny Burger Yeah, I would say so. So I guess this pivotal moment was when we just made a tweet where he said that now you can steer re-motion through agent skills. So there's this npx skills command line, which is kind of like a package manager for skills. And that one blew up, yeah. ([Time 0:05:07](https://share.snipd.com/snip/16939b79-7d36-4163-a1aa-de333918a22a)) - **Nontechnical Users Now Power Many Remotion Videos** - The user base shifted toward non-technical people using broad prompts to generate product-launch videos and YouTube graphics. - Jonny highlights even large companies building motion design systems with Remotion components. Transcript: Jonny Burger Yeah, I think the audience has changed towards more non-technical people. They are now willing to give it a try and just giving Claude a very broad, non-specific prompt about the video that they would like to have. I think because they are just like so, because it's so accessible, many people are now just doing that and doing videos like small like product launch videos or like explainer videos Or like graphics for their YouTube videos. There have also been like real companies, big companies building up like design systems, like reusable graphics, kind of like a motion design system. Yeah. Wes Bos Like ([Time 0:06:03](https://share.snipd.com/snip/4d6b912f-b839-4b8e-aea5-00b57032f7f6)) - **Visual Editing Must Stay Source-Of-Truth In Code** - A fast, iterative visual editor that writes back to code is crucial but technically hard due to bundlers and fast refresh. - Jonny experiments with proxies, source maps and AST transforms to map UI edits into code without slow refresh. Transcript: Jonny Burger Like a super nice, fast, iterative workflow. I mean, I'm using After Effects myself. I'm fascinated by the program and I'm trying to learn from it. And like, gonna be honest, I want to copy some of its ideas now. Like, for example, they have like this, this effects system, this effects system where you have like blur, wiggle, vignettes, you know, just like grab them from a panel and drag them On top of a layer. And then you have these controls which you can adjust and you get instant feedback. And this does not make sense to do agentically because you have to wait. I think this could be the next step and I'm actively working on this. I'm kind of like trying to figure out how this can make sense because I do think one other important aspect is that the code should like remain the source of truth. So if you like visually edit the values in a UI, how does this save back to the code? What I've kind of figured out so far is that, you know, like with some black magic with JavaScript proxies and source maps, you can kind of like figure out where a React component is located In the original code. And then we can use some abstract syntax tree processing to write the props back. But then if you write values back to the code, then that triggers a fast refresh, and there's like 200 milliseconds latency, and what happens at the same time, it's not a nice experience. So now I'm trying to bypass the bundler if you write back to the code, but then the agent writes at the same time. I'm thinking about this. And, you know, once I figure it out, you know, we can also bring this aspect to Remotion. ([Time 0:10:30](https://share.snipd.com/snip/f0dafbbd-fd1b-49a3-b2aa-5cadaecc07f1)) - **HTML In Canvas Unlocks Pixel Effects On DOM** - Chrome's experimental HTML-in-Canvas lets DOM content be rasterized into a canvas so you can apply shaders and pixel effects to HTML. - Jonny says this removes a major web constraint and unlocks editor-style raster effects previously impossible in the DOM. Transcript: Wes Bos I think that's probably really exciting for you. It's very exciting. Jonny Burger Maybe I need to contain the hype at the beginning. It's like an experimental technology. For it to work, you have to like download Chrome Canary and enable a flag in the settings. So you cannot like ship this preview to as a website to a user. But if you are willing to put up with it, then you can use this to create some really cool videos. And if we think about it, it's like any video editor or any motion graphics editor. What they do is they essentially have like a canvas, a rasterized bitmap, and all of these effects are just being drawn onto the bitmap and then if you have effects transitions they just Manipulate the the pixels in many different creative ways and that's not been possible in the on the web so far we do a canvas element, but it is like separate from the DOM world. So in the DOM world, you can lay out items in a grid, in a row, you know, like with CSS, with one line of CSS, apply all sorts of things. But you cannot apply like a progressive blur or a vignette effect on top of that content. And this API makes that possible. And so it feels like day one again, because that's number one constraints that we had. It's now gone. ([Time 0:15:14](https://share.snipd.com/snip/cf710fa7-ffc0-4065-a872-64d55c856c1c)) - **Experiment Early But Pin Versions For Stability** - Experiment with new browser APIs early but plan for changes; pin versions if needed. - Jonny warns HTML-in-Canvas is experimental (Chrome Canary flag) and could change or be scrapped, so adapt and provide feedback. Transcript: Jonny Burger Yeah, it could happen. I mean, if we need to be realistic and they could just like scrap it. I think there's like this whole process of like stage zero, stage one, stage two that like any web technology has to go through and it's not even at stage zero yet. And also like the API could change entirely. I'm just not willing to let it dampen my mood right now because we can pin the version if we wanted to, and I'm willing to adapt to whatever. ([Time 0:19:13](https://share.snipd.com/snip/dae79bbc-2cfb-404e-ae26-9f072827a06f)) - **Deterministic Frames Via Headless Chrome And FFmpeg** - Remotion renders by launching headless Chrome, seeking frames, capturing screenshots, and encoding with FFmpeg for deterministic frame output. - Jonny notes getting HTML-in-Canvas working required compiling Chrome builds with codecs to run in AWS Lambda. Transcript: Jonny Burger Yeah, pretty much. It sounds very simple and it is on the surface that we just like essentially launch Puppeteer, seek to the right frame and capture it and encode it with FMPEG. Yeah, I don't want to make it sound more complicated than it is. To like get this HTML in Canvas thing working was like very difficult though, actually, we like wanted to compile a version of Chrome which was recent enough to have this thing but also Was like small enough so that it can fit in an AWS Lambda function and also that it has like the H.264 and H.265 codecs enabled, which by default, like the headless Chrome does not have. And this is also very important for us to play videos. And so for like all of the operating systems, we like compiled Chrome, like for five different platforms. And that's like a huge project. ([Time 0:21:16](https://share.snipd.com/snip/2dee9143-4331-40e0-972d-3fd205f1863b)) - **Use Time-Deterministic Rendering Not CSS Animations** - Make render logic time-deterministic and avoid native CSS animations to prevent flicker. - Use Remotion hooks like useCurrentFrame and pure rendering functions so frames are reproducible and parallelizable across Lambdas. Transcript: Jonny Burger CSS animations as well? You can animate the CSS properties in JavaScript, but you cannot have the CSS native animations. These do not work and lead to flickering. Our model is essentially that using a hook called use current frame, you get the current time and then it's like up to you to like render the right image. It's like a pure function. The time is the argument and your job is to return an image. And because like the CSS animation is not like tied to the time, well actually you can make it, but it's a bit tacky. That does not work by default. But it unlocks some really nice things. Like you can have like a huge timeline and only need to mount and render the content that you really actually need at the moment. So if you have 100 clips, then, you know, you don't have a bottleneck and you can multitread the render because like every frame is so deterministic. You can parallelize it as much as you want. ([Time 0:23:30](https://share.snipd.com/snip/429c6b1d-fef7-4cfa-97da-dd249c8571c7)) - **Pure Frames Let You Parallelize Video Rendering** - Deterministic per-frame rendering enables massive parallelism because each frame is a pure function of time. - Jonny says users can split renders across many Lambdas (Remotion allows up to 200) to speed up long renders. Transcript: Wes Bos You could theoretically spin up, like let's say you had like 100,000 frames. Could you theoretically spin up 100,000 lambdas and render it all in like one second? Jonny Burger Yeah, let me think about it. So 100,000 frames, let's divide it by 30. That's like 3,000 something seconds. So it's like an hour long video. And with like ReMotion Lambda, you can, I mean, this is like a hard coded limit by us because we think too much concurrency also doesn't make sense, but you can like spin up to 200 Lambda Functions and actually split up this hour of video into 200 parts and render it pretty quickly. The longer the video, like the more X real time it is because the parallelism outweighs the overhead. So, absolutely possible and our users actually want to do that. ([Time 0:24:50](https://share.snipd.com/snip/13ab8bb9-7811-4e5d-a616-4fba9fef2158)) - **Metadata Can Reveal Programmatic Video Origins** - Some users noticed Remotion metadata embedded in FFmpeg output and worried platforms might detect programmatic 'slop' videos. - Jonny apologizes and clarifies Remotion wasn't built for that use, but users reported platforms using metadata to flag content. Transcript: Jonny Burger Yeah interesting i i don't have like super secret info okay um you know like some some people have been like whenever a video is being made with free motionotion, we like using FFMPEG, Like embed like metadata. Yeah, I FF probed a video the other day and it says made by ReMotion. And some users complain that, you know, they want to be able to remove this because apparently platforms use this to detect their slop yeah i made a slop video the other day um and i removed Wes Bos It i think i removed it but or no i didn't and tiktok just tanked it immediately and then instagram did well but i think that they were i'm not sure if they detected the remotion tag or they Detected that it was like an AI music kind of meme reading GitHub mean GitHub comments it was really funny by the way, Shawadja but I was curious what they used to detect that I mean if that Jonny Burger Is really true then I apologize to the world really for all of this love. I was never part of the plan. Started this when AI was not a thing. And I'm trying to not amplify it or compliment our users on it who are doing this. ([Time 0:27:30](https://share.snipd.com/snip/942ef2b8-2518-4d31-82de-0181ab5bbe6a)) - **WebCodecs Enables Low Level Browser Video Workflows** - WebCodecs lets browsers access and re-encode H.264/H.265 chunks, enabling low-level video manipulation without the video tag. - Jonny joined Media Bunny instead of rebuilding codecs because the library handles tricky MP4/WebM edge cases and accelerated codec support. Transcript: Jonny Burger Yes, I'd love to. So traditionally in the browser, if we are dealing with video, then we are just like interfacing with the video tag. Or we wanna extract a frame from a video, we mount a video tag, we seek to the correct time and then like capture a screenshot, but this is not frame accurate. There's a limit, you cannot like mount too many video tags. At some point, like the browser will just start trodling you and it's not efficient at all. And there's another really interesting technology that is a bit more mature than HTML and Canvas that is now available in all browsers in 2026 called WebCodex. And that allows you to interface with video on a very low level. So this allows you to take encoded H.264 chunks, H.265 chunks, allows you to decompress them and also to encode them again. These H.264 chunks, they are like embedded. This is what you will find in an MP4 video. And to read them out, you actually need to read the binary data of an MP4 and extract and interpret these encoded packets. And that is, as it turns out, a really difficult task because MP4s have so many different features. There are many edge cases and that is just like one container. There's also WebM and like six other containers if you include the audio containers. I saw this opportunity that we can work around limitations of the video tag by just fetching the MP4 via the web fetch API, reading the binary data, putting this into web codecs, and then Doing things and then even doing things like converting an MP4 to a webm and have like a different codec which was not possible before um i mean it was possible like software only but this Was also just like the accelerated codecs and so sorry only now i'm getting to media bunny when i saw this then i immediately realized that the guy behind it was like much smarter at it In every aspect. And now we talk often and it's like amazing how brilliant he is. He like for every topic always knows the right answer. And like I immediately realized that, you know, maybe I should give up on this rabbit hole and, you know, just use this awesome library instead. ([Time 0:29:30](https://share.snipd.com/snip/1ddc229f-cd40-4f2f-ac5a-2c85de2a896e)) - **Parametrize Motion Graphics With React Props** - Parametrize motion-graphics components in React using props and provide a small UI form for non-technical editors. - Jonny points to remotion.dev/brand demo where users pick branded lower thirds and copy a CLI to render transparent videos for DaVinci. Transcript: Jonny Burger Yeah, I guess the reMotion way is that you would first figure out how to build a scaffold in web technology using React, CSS, make it look like you want it to, then you animate it, you add Keyframes in reMotion. In the end, we have a way to like parametrize these motion graphics. We simply use React props for that. You have a component, you give it props. And I think you've also like alluded to like these little drop downs that we have in the studio. So we allow you to like interactively edit and preview them using a thought schema, kind of create a form where you can try different values and see how... So good. Just makes so much sense in my brain that way of doing it. And I would say, if you're good with this workflow, then that is the way to go. But reality is, and a lot of users ask us this, is that they have an After Effects template and now they want to parametrize it. This is where you have to use Lottie. Well, it works well. Use the regular Lottie React component. Well, we've kind of made our own integration, but it's like a very thin layer on top of it. And you parametrizeize the JSON essentially. But it's also one big limitation of the Lottie file that it is just a JSON which describes the whole video. You do not have the crazy cascading of CSS. You do not have the crazy if-else data fetching stuff that you can do in React. It's not code. It's just data, which defines a Lottie file. So you are limited in how you can parameterize a Lottie file. And it's very pragmatic. You design something in After Effects, and then you it over to re-motion with Lottie. That's cool. Wes Bos And does Lottie have like, so they wouldn't have the ability to like say measure a name. You know, if my name West Boss is short, maybe you want to increase the size of it until the height of it gets to a certain point. Whereas like somebody's Dermon last name is really long. Jonny Burger You want to like you maybe want to shorten it the Lottie doesn't have that logic like that does it I do think it gets pretty hairy if you want to do that because as far as I know the like a Lottie Animation has like fixed dimensions So if you clip outside of it, then you have to maybe with code also detect, like measure the text and also decrease the font size, which is also in the JSON. Yeah. I think that's how it works, but I'm not a lot of the expert. Maybe they do have like an authorizing feature. This would be cool. Wes Bos Yeah. I think like honestly, I'm talking, like, Randy, our editor as well, and, like, essentially, I think what we want to end up with is just, like, a website we can visit, and we have got all Of our components, and you can just type in, and there's all logic in those, you know, if there's three people, render it this way, and then you obviously could just click download. It'll render it out, download it, we can throw it into DaVinci. But, like, I think the dream would be like a DaVinci plugin or something like that where you can type in the different props and then click re-render or something, and that would replace That piece somewhere in there. Jonny Burger Yeah. So we do not have a plugin yet, but what you have just described before, you can totally do it, and we actually have a demo if somebody wants to check it out. It's on the remotion.dev slash brand. And you will essentially end up on a website where you can click through our logos, like lower turds that we use in our videos. You can even like with this sword schema parameterize like the lower turds that we have. And then not yet possible to render directly on the website because this is a static website, but you can copy a CLI command and then you can render out that lower turd as a transparent Progress video and drag it into DaVinci. ([Time 0:35:01](https://share.snipd.com/snip/173850d4-c6c0-458d-bfc5-12d6d68f9717)) - **Using ENTTEC DMX To Vibe Code Light Shows** - Jonny's sick pick was the ENTTEC DMX USB Pro, which lets him control professional stage lights from the browser via Web Serial. - He used it with Remotion and AI tools to create vibe-coded lyrics videos and a birthday DJ performance. Transcript: Jonny Burger And it is right in my hand. It is the Entek DMX USB Pro. This is a device that's not so cheap, actually, maybe like $100. And what it allows you to do is it's an interface that allows you to plug in like actual disco lights, like this one, into your MacBook, and then interface with this disco lights through The DMX protocol. This is like the protocol that is actually being used for like big concerts, big festival productions. It all runs over the DMX protocol. I figured out with one prompt, I was really surprised that it worked, that through the web serial API, we can program these slides in the browser. And then we can wipe code slides. You can and and it's like honestly so cool so um i bought like a couple of these lights um made some uh vibe code that made essentially a vibe coded light show um use remotion to and 11 labs To like make some karaoke videos some lyrics videos, composed of After Effects, and essentially used all of the AI tools. ([Time 0:39:57](https://share.snipd.com/snip/a5ccf263-a076-4d20-b553-9bdcf8ffc2e6))