There are already neural networks that create original video on a textual description. Although they can not completely replace operators, progress in this area is obvious.
Disney Research and Rutgers have developed a neural network that can create a rough scenario and text-based video. The system works with a natural language. That will allow it to be used in a number of areas, such as creating video tutorials.
In addition, the AI system will help writers visualize their ideas. Its creators assert that the goal is not artificial intelligence to replace writers and artists. Its goal is to make their work more effective and less boring.
Translating text into animation
Developers say that translating text into animation is not an easy task, as input and output data do not have a fixed structure. Most of the operating systems can not cope with complex sentences. To overcome the limitations of previous similar programs, developers have built a neural network of several components. They include a natural language module, a script analysis module, and an animation-generating module.
First, the system analyzes the text and translates complex sentences into simple ones. Then it creates 3D animation. A library of 52 animated blocks has been used to operate the system. Now the list contains 92 animated blocks. Creation of the animation is done using Unreal Engine. It relies on pre-installed objects and models – the system selects the appropriate elements and forms the video.
For system training, researchers have compiled a set of descriptions of 996 elements taken from more than 1,000 scenarios with IMSDb, SimplyScripts, and ScriptORama5. Then qualitative tests were performed in which 22 participants rated 20 animations. At the same time, 68% of the participants say that the system creates quite decent animation based on the input texts.
The team, however, admits that the system is not perfect. Its list of actions and objects is not exhaustive and sometimes lexical simplification leads to inappropriate animations. Researchers intend to eliminate these shortcomings in their further work.