Animate images for film with RunwayML
Learn how to use RunwayML to animate static images into short clips for use in a short film.
2024-12-06
This is the 3rd tutorial of our course, How to create an AI-generated short film.
Now that we have our storyboard images, we can animate them into short clips with RunwayML. In this tutorial, we'll walk you through the process of transforming your static scene images into dynamic video animations.
Steps we’ll follow in this tutorial:
- Define video settings with RunwayML
- Upload a scene image and add a prompt
- Generate the first video animation
- Generate all video animations
Define video settings with RunwayML
Now that we have our scene reference images, we can head over to RunwayML to animate these images. To do this, click on the “Text/Image to Video” tab in the left-side navigation.
This will take us to the Gen-2 video generator. The first thing we’ll do is select a video-style preset. We’re going to go with “Anime” to match our image generation style but feel free to pick the style that matches your story.
Upload a scene image and add a prompt
Next, we’ll upload our first scene image and add a companion prompt. To do this, click on the “Prompt” tab in the left-side navigation. Then, select the image upload button and upload your first scene image.
In addition to uploading a reference image, we can also add a prompt to describe our scene to further enhance the animation. The components of this prompt can include the following parts:
[Camera Movement]: [Establishing Scene]. [Subject Description] [Action].[Lighting and Atmosphere]. [Additional Details].
To create the full prompt, we’ll go back to Claude and ask it to create an animation prompt for every scene in this recommended structure.
Sample prompt:
For each of the scenes, we now want to animate them using RunwayML. Can you generate a prompt for each scene for this? Please follow the prompt structure below. The prompt can be 320 characters max.
[Camera Movement]: [Establishing Scene]. [Subject Description] [Action].[Lighting and Atmosphere]. [Additional Details].
Claude will generate the prompts for RunwayML for each scene. Copy the first prompt, go back to RunwayML, and paste it below your uploaded scene image.
Generate the first video animation
Finally, we’ll generate the animation by clicking the “Generate 4s” button, which will generate a 4-second animation of our image.
You can view the animation on the right side of the screen.
Generate all video animations
Similar to our process in Leonardo AI for generating similar style images with a “Fixed Seed”, we’ll do the same for our animations in RunwayML. To get the “Fixed Seed”, click on the Settings icon in the bottom left corner of the page and enable the “Fixed Seed” setting.
Now, we can go back to Claude and copy the RunwayML prompt for the second scene.
We’ll paste it into RunwayML and upload our second scene image in tandem to generate our second scene animation.
Similarly to the first scene, you can watch the second scene animation on the right side of the page.
Repeat this process for all of your scenes. Make sure to download each of the videos from RunwayML. We’ll need these later when we put everything together into a compiled film.
This tutorial was created by Garrett.