小举人 Manuals

↑↓ to choose Enter to open Esc to close

Dify Now Supports Video Generation, With a Tutorial for Building AI Video Apps

Build text-to-video and image-to-video apps in Dify with the Doubao video plugin: install, authorize, import a DSL, or build from scratch with Chatflow.

4 min read
On this page

TL;DR: if you just want to try it out, jump straight to the demo links at the end of the post.

No more chit-chat. Here's the result first:

Text-to-video chat example: typing "Thousands of troops charging" produces an expanded prompt, an image and a video
Text-to-video chat example: typing "Thousands of troops charging" produces an expanded prompt, an image and a video

Type some text into the app's chat box and you get a video. Even a single simple sentence is automatically expanded into a detailed video prompt before the video is generated.

This post walks through how to try text-to-video yourself, and how to build your own AI video generation app.

Sample results (these videos were embedded in the original WeChat post; here we keep the cover frames. For the full animated effect, click the demo links):

  1. Girl and fox (text-to-video)
Cover frame of the girl and fox text-to-video
Cover frame of the girl and fox text-to-video
  1. Thousands of troops charging (text-to-video)
Cover frame of the thousands of troops charging text-to-video
Cover frame of the thousands of troops charging text-to-video
  1. King and airplane (image-to-video)
Cover frame of the king and airplane image-to-video
Cover frame of the king and airplane image-to-video
  1. Ancient warhorse (image-to-video)
Cover frame of the ancient warhorse image-to-video
Cover frame of the ancient warhorse image-to-video

A sketch of the image-to-video workflow interface:

Image-to-video workflow: upload an image, run it, and get an MP4
Image-to-video workflow: upload an image, run it, and get an MP4

No complicated sign-up or authentication is needed. Just let your imagination run.

#How it works

All three apps were built in Dify. The core of it is Dify's video generation plugin.

If you don't know Dify yet, think of it as a playground for AI apps, where you assemble an app like playing with building blocks. Official site: https://dify.ai/

Dify homepage and workflow preview
Dify homepage and workflow preview

Here are the full steps to reproduce it. It takes about three minutes.

#1. Sign up and log in to Dify

Open https://cloud.dify.ai/ and sign up with your email. You can switch the language in the top-right corner. You can also install the open-source community edition.

Switching the language in the top-right corner of cloud.dify.ai
Switching the language in the top-right corner of cloud.dify.ai

#2. Install the video plugin and complete authorization

Dify itself is not a video model. Video generation relies on an external plugin.

Top-right: "Plugins" → "Explore Marketplace". Search for "video" and find Doubao Image and Video Generator. Its capabilities are based on Doubao.

Searching "video" in the Marketplace and finding Doubao Image and Video Generator
Searching "video" in the Marketplace and finding Doubao Image and Video Generator

After installing, enter your own API key to complete authorization. Don't write the key into the docs or commit it to a repository.

Plugin authorization panel: enter your Volcengine API key
Plugin authorization panel: enter your Volcengine API key

#3. Create an app

After you've signed up for the platform, you still need to create an app and then add video generation to it.

#3.1 Copy a ready-made app

Under the hood, a Dify app is a DSL. On the web, use "Import DSL file" and paste in a link:

Import DSL file dialog
Import DSL file dialog

Once the import succeeds, you should see the app interface below:

App interface after a successful import
App interface after a successful import

#3.2 Build from scratch

Create a blank app, choose Chatflow, and fill in a name and description.

Creating a blank Chatflow app
Creating a blank Chatflow app

Add a tool node and choose Doubao Generator. Depending on the effect you want, pick text-to-image, text-to-video or image-to-video.

Adding a tool node and selecting Doubao Generator
Adding a tool node and selecting Doubao Generator

Each tool node has default inputs and outputs. For text-to-video, the input is the prompt and the output is the video the plugin generates.

Input and output parameters of the text-to-video tool node
Input and output parameters of the text-to-video tool node

I connect the input to the output of the previous LLM step. A user's prompt is often not detailed enough, so first let the model expand it into a detailed video prompt, then send it to the generation node.

Connecting the LLM-expanded prompt to the video generation node
Connecting the LLM-expanded prompt to the video generation node

This plugin can also be used inside an Agent node.

Using the video plugin in an Agent node
Using the video plugin in an Agent node

If you don't understand workflows yet, start here: Dify Workflow.

#Summary

In the past, generating video in Dify usually meant using an HTTP node to call an external API. It works, but it isn't elegant, and everyone who wants to make videos has to set it up again. This is a common need, so it's better to make it a plugin where you just enter an API key and go.

Plugin source code and Marketplace card preview
Plugin source code and Marketplace card preview

Plugin source code: AllenWriter/doubao-image-and-video-generator

Developer docs: Getting started with Dify plugins

If you also want to try video generation in Dify, or want to develop your own plugin but don't know where to start, feel free to reach out on WeChat:

Author's WeChat QR code
Author's WeChat QR code

Original article: https://mp.weixin.qq.com/s/obqH8fliSfFIJC0yH7dqvg