KR101306221B1 - Method and apparatus for providing moving picture using 3d user avatar - Google Patents

Abstract

The present invention relates to a video production apparatus and method using a three-dimensional user avatar, the video production method of the present invention, generating a three-dimensional user avatar using a two-dimensional user image; Recognizing the facial expression of the user in a user image input in real time and presenting operations corresponding to the recognized facial expression; Generating a dynamic avatar by applying the action selected by the user and the recognized facial expression to the 3D user avatar; And generating a user animation by synthesizing the dynamic avatar of the user with a pre-stored background video. This creates a 3D user avatar that reflects the user's face, and reflects the user's facial expressions captured by the camera to the user avatar to create a dynamic avatar with moving facial expressions and actions, and combines the user's voice with the image of the user's avatar. By allowing recording and editing, a video using a user avatar can be produced.

Description

Video production apparatus and method using 3D user avatar {METHOD AND APPARATUS FOR PROVIDING MOVING PICTURE USING 3D USER AVATAR}

The present invention relates to a video production apparatus and method using a three-dimensional user avatar, and more particularly, to generate a three-dimensional user avatar reflecting the user's face and to reflect the facial expressions of the user taken by the camera to the user avatar The present invention relates to a method and method for producing a video using a 3D user avatar, which generates a dynamic avatar in which a motion and a motion move, and to record and edit a user's voice and an image of the user's avatar together. will be.

Recently, with the development of content production technology, applications that allow individuals to produce various multimedia contents are being serviced. However, these conventional multimedia content production applications are not merely for producing music videos using photos or for producing video contents that are simple to edit videos, and do not satisfy users who want to produce more convenient and high-value personal multimedia contents. have.

In particular, in the case of the conventional multimedia content production application, the source required for the production is limited to a small amount of content provided by the actual service, or limited to a small amount of personal data owned by the individual on his computer. There is a problem in that it is difficult to produce a personal multimedia content in a true sense reflecting the changing personal tastes.

Meanwhile, with the development of graphic processing technology, 3D imaging technology is developing differently every day. However, the production of 3D video still requires professional skills, and it is difficult for a general person who is not familiar with computer 3D to produce high quality 3D video.

Korean Patent Application Publication No. 2006-0082738 Title: Personal character service method of mobile communication terminal and mobile communication terminal

The present invention has been made to solve the above-described problems, generating a three-dimensional user avatar reflecting the user's face, and generates a dynamic avatar moving the facial expression and motion by reflecting the user's facial expressions photographed with the camera to the user avatar In addition, there is a technical problem to provide a video production apparatus and method using a three-dimensional user avatar to be able to record and edit the user's voice and the image of the user avatar together, to produce a video using the user avatar.

According to an aspect of the present invention for achieving the above object, generating a three-dimensional user avatar using a two-dimensional user image; Recognizing the facial expression of the user in a user image input in real time and presenting operations corresponding to the recognized facial expression; Generating a dynamic avatar by applying the action selected by the user and the recognized facial expression to the 3D user avatar; And generating a user animation by synthesizing the dynamic avatar of the user with a pre-stored background video.

Here, in the generating of the 3D user avatar using the 2D user image, a ratio of the part of the user's face is calculated by applying AAM to the 2D user image, and the 3D user avatar is calculated according to the calculated value. Modeling a facial shape of the face; And synthesizing the skin texture of the user face to the avatar face by applying AAM to the two-dimensional user image.

The generating of the 3D user avatar by using the 2D user image may include: synthesizing a pre-stored avatar's body type and clothes according to the user's selection to the face of the 3D user avatar; And generating a 3D user avatar of the user by editing the body shape and clothes of the avatar according to the user's selection.

In addition, the step of recognizing the expression of the user from the user image input in real time and presenting the operations corresponding to the recognized facial expression, extracting a feature point of the user face by applying AAM to the user image input in real time step; And BPNN processing the feature point to recognize the facial expression of the user.

The feature point of the user face may include at least one of a face shape and a skin texture.

The step of recognizing the facial expression of the user from the user image input in real time and presenting operations corresponding to the recognized facial expression may include: receiving and storing motion data of an avatar corresponding to the defined facial expression in advance; Retrieving motion data corresponding to the recognized expression of the user from the stored motion data; And presenting the retrieved motion data to a user.

Meanwhile, generating a user animation by synthesizing the dynamic avatar of the user with a pre-stored background video may include receiving a voice of the user and storing the same with the user animation.

According to another aspect of the present invention for achieving the above object, when driven in a mobile terminal, generating a three-dimensional user avatar using a two-dimensional user image; Recognizing the facial expression of the user in a user image input in real time and presenting operations corresponding to the recognized facial expression; Generating a dynamic avatar by applying the action selected by the user and the recognized facial expression to the 3D user avatar; And a computer-readable storage medium storing a program for performing a video production method using a three-dimensional user avatar comprising the step of generating a user animation by synthesizing the dynamic avatar of the user with a pre-stored background video.

According to another aspect of the present invention for achieving the above object, modeling the face of the three-dimensional user avatar using a two-dimensional user image and modeling the body shape and costume of the avatar matching the avatar face according to the user's selection A user avatar generator to generate a 3D user avatar; Recognize the user's facial expressions from the user image input in real time, present the motions corresponding to the recognized facial expressions, and apply the motions selected by the user and the recognized facial expressions to the 3D user avatar to generate a dynamic avatar. An avatar generator; A basic animation DB storing a background video; And a user animation storage unit for generating a user animation by synthesizing the dynamic avatar of the user with the background video.

The user avatar generator may include: a 3D avatar face generator configured to apply an AAM to a 2D user image to calculate a proportion of each part of the user face and synthesize a skin texture to model a face of a 3D user avatar; A database in which the body shape and costume data of the avatar are stored; And a three-dimensional avatar body for synthesizing the avatar's body shape and clothing stored in the database to the face of the three-dimensional user avatar, and editing and modeling the avatar's body shape and clothing according to the user's selection. The clothes generating unit may be included.

The dynamic avatar generator may include: an avatar motion database storing motion data of an avatar corresponding to a defined facial expression; A user facial expression recognizing unit which extracts a feature point by applying AAM to the user image input in real time and recognizes the expression of the user by BPNN processing the feature point; A facial expression and motion matching unit for searching for and presenting motion data of an avatar corresponding to the recognized facial expression to a user; And an avatar facial expression and motion synthesizer configured to generate a dynamic avatar by applying the motion data of the avatar selected by the user and the user facial expression recognized by the user facial expression recognizer to the 3D user avatar.

The feature point of the user face may include at least one of a face shape and a skin texture.

As described above, the apparatus and method for producing a video using the 3D user avatar of the present invention generates a 3D user avatar reflecting the user's face, and reflects the user's facial expression photographed by the camera to the user avatar. This moving dynamic avatar is generated and the user's voice and the video of the user avatar can be recorded and edited together, thereby producing a video using the 3D user avatar.

1 is a control block diagram of a user terminal to which a video production apparatus using a 3D user avatar according to an embodiment of the present invention is applied;
2 is a control block diagram of a user avatar generator according to an embodiment of the present invention;
3 is a control block diagram of a dynamic avatar generator according to an embodiment of the present invention;
4 is a state diagram of storing reference facial expression data of a video production apparatus using a 3D user avatar according to an embodiment of the present invention;
5 is a hierarchical structure diagram of a neural network of a BPNN (Back Propagation Neural Network) for user facial expression recognition according to an embodiment of the present invention;
6 is a flowchart of an avatar face generation process of a video production method using a 3D user avatar according to an embodiment of the present invention;
7 is a flowchart illustrating a process of recognizing a user's facial expression of a video production method using a 3D user avatar according to an embodiment of the present invention;
8 is a flowchart of a video production method using a 3D user avatar according to a first embodiment of the present invention;
9 is a flowchart of a video production method using a 3D user avatar according to a second embodiment of the present invention;
10 is a flowchart illustrating a video production method using a 3D user avatar according to a third embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description with reference to the accompanying drawings, the same or corresponding components will be denoted by the same reference numerals, and redundant description thereof will be omitted.

1 is a control block of a user terminal 10 to which a video production apparatus 100 using a 3D user avatar according to an exemplary embodiment of the present invention is applied.

The user terminal 10 is a device supporting a camera function and provided with a user interface. The user terminal 10 may include various devices such as a mobile phone, a netbook, a PMP, a tablet, a smartphone, and the embodiment of FIG. 1 is applied to a smartphone. To illustrate.

The user terminal 10 includes a communication unit 12, a camera 14, a display unit 16, a user input unit 18, a video production apparatus 100, and a terminal control unit 20 controlling them.

The communication unit 12 may process a communication function of the user terminal 10 to transmit and receive data. The user may receive a face photograph through the communication unit 12 or may download data for making a video. In addition, when the video production apparatus 100 using the three-dimensional user avatar of the present invention is implemented and distributed in the form of one application, the communication unit 12 is a video production apparatus 100 through a wired or wireless communication unit 12 You can download it.

The camera 14 generates a 2D image by capturing an image of an object, for example, a face of a specific person.

The display unit 16 displays a video production process using the 3D user avatar to the user. The user may check and modify the shape, motion, and facial expression of the avatar by checking the 3D avatar synthesized through the display unit 16.

The user input unit 18 provides an input interface for allowing a user to set and modify each area of the avatar, an operation, and the like. For example, when the user input unit 18 is provided as a touch panel, the user may input a touch signal, a pinch signal, a drag and drop signal, or the like to adjust the size of each area of the face of the 3D avatar or the texture color. .

The video production apparatus 100 generates a 3D avatar of the user by using the user's face photograph in the 2D format, analyzes the expression of the user input to the camera 14, and reflects the expression and motion of the 3D avatar to the user. Create an animation using your 3D avatar. The video production apparatus 100 may provide a function of recording, editing, and playing a user's voice on the expression and motion of the 3D avatar.

The terminal controller 20 controls the overall function of the user terminal 10 by transmitting and receiving data signals and control signals with respective components connected by a bus. When the user executes the video production function, the terminal controller 20 activates the function of the video production apparatus 100, and the information input through the camera 14 and the user input unit 18 is input to the video production apparatus 100. While controlling to be transmitted to, the video data output from the video production device 100 is controlled to be transmitted to the display unit 16.

The video production apparatus 100 may include an animation production controller 110, a user avatar generator 200, a dynamic avatar generator 300, a user animation generator 400, a user animation storage 450, and a basic animation DB ( 500).

The animation production control unit 110 controls each component in the video production apparatus 100 so that a video using a 3D user avatar is generated.

The user avatar generator 200 generates a 3D user avatar by expressing the user's face as a 3D object using a user's face photograph and modeling the user's body shape and clothes according to the user's selection.

The dynamic avatar generator 300 reflects a user's facial expression on the avatar by using real-time facial expression recognition technology for the facial image of the user input to the camera 14. In addition, by allowing the user to set actions that can correspond to the defined emotions, a dynamic avatar representing various expressions and actions is generated.

The basic animation DB 500 stores an animation background for synthesizing a dynamic avatar and text information to be read by the user. The basic animation DB 500 may store a plurality of animation backgrounds and text information having different themes. For example, when the video production apparatus 100 is provided to produce a storytelling story, the basic animation DB 500 may store a background animation corresponding to the development of the storybook and the content of the storybook to be talked about.

The user animation generator 400 synthesizes a background animation stored in the basic animation DB 500 and a dynamic avatar created by the user to generate an animation in which a 3D user avatar appears. In addition, by inputting and synthesizing the contents of the animation read out by the user, that is, the contents of the moving picture, the user animation generated by the user's voice and the user's three-dimensional avatar can be generated.

The user animation storage unit 450 stores a user animation generated by the user.

2 is a control block diagram of the 3D user avatar generator 200 according to an exemplary embodiment of the present invention.

The user avatar generator 200 includes a 3D avatar face generator 210, a 3D avatar body and costume generator 220, and databases 230, 232, and 234 storing various data for generating an avatar. .

The database may include an avatar face DB 230, an avatar costume DB 232, and an avatar body DB 234. The avatar face DB 230 stores basic avatar face information used to model a 3D avatar face. The avatar body DB 234 stores basic avatar body information used to model the body of the 3D avatar. The avatar costume DB 232 stores information such as clothes, accessories, and props to be applied to the 3D avatar.

The 3D avatar face generator 210 generates a 3D user avatar face by recognizing a face image of the user in the 2D image. An AAM (Active Appearance Model) may be applied to generate the 3D avatar reflecting the user's face. AAM assigns faces in the form of designated meshes to calculate the difference from the previous mesh while the face is moving so that the face can be tracked continuously and once the mesh is precisely positioned on the face, it can separate parts of the face. It is a skill.

Thus, the 3D avatar face generator 210 generates the shape and skin of the 3D avatar using the captured image of the camera 14 and calculates the ratio between the meshes to which the AAM is applied to the input image. You can create it and change it according to your needs. Extract the parameters of appearance and shape from the 2D image where the face region is recognized, and model the area of the eyes, nose and mouth in the input face image to fit the shape of the AAM based on the extracted facial features The facial skin texture of the user avatar is synthesized using the skin of each region.

The 3D avatar body and costume generation unit 220 performs an avatar body production and a costume wearing function. The 3D avatar body and costume generation unit 220 receives body and costume data from the avatar body DB 234 and the avatar costume DB 232 to model the body and costume of the user avatar.

3 is a control block diagram of the dynamic avatar generating unit 300 of the video production apparatus 100 using the 3D user avatar according to an embodiment of the present invention.

The dynamic avatar generator 300 models dynamic avatars representing various expressions and actions, and includes a user facial expression recognizer 310, an avatar facial expression and motion DB 340 and 345, a facial expression and motion matcher 320, and an avatar. Expression and motion synthesis unit 330 is included.

Avatar facial expression and motion DB (340, 345) stores the basic facial expression and motion information of the avatar. Avatar facial expression DB (340) is stored in the six basic expressions such as joy (surprise), fear (fear), anger (anger), disgust, sadness (sadness) are stored . The facial expression data stored is as shown in FIG. 4. The facial expression data shown in FIG. 4 is a Cohn-Kanade DB, which is a standard facial expression DB recognized by the International Society, and provides facial expression data (race, gender, and facial expression) which are the most basic for expression-related research. The standard facial expression may be used to recognize the user's facial expression seen through the camera 14. The avatar motion DB 345 stores motion data according to the avatar expression. For example, actions that can be taken in an emotional state such as pleasure, surprise, fear, anger, disgust, sadness, etc. may be stored.

The user facial expression recognition unit 310 generates the shape and skin of the 3D avatar using an image captured in real time by the camera 14. The user facial recognition unit 310 generates an avatar face by calculating a ratio between meshes to which the AAM is applied to the input image, and may change the image according to a user's request. The creation of the skin is also converted to the avatar's skin using each part of the applied AAM mesh.

The user facial recognition unit 310 may include a face recognition unit 312, a feature extractor 314, and a BPNN (Back Propagation Neural Network) processor 316.

The face recognition unit 312 recognizes an area corresponding to the face in the input image.

The feature extractor 314 applies an AAM (Active Appearance Model) to the face image. When AAM is applied, the area of the eyes, nose, and mouth are matched to the shape of the AAM in the input face image, and the facial skin texture of the user avatar is generated with the skin of each area.

The BPNN processor 316 may classify various facial expressions by recognizing facial expressions using a BPNN (Back Propagation Neural Network) method by using feature points of the facial image.

5 shows a neural network hierarchy of a BPNN. The BPNN has a hidden layer between the input and output layers. The hierarchical structure is composed of an input layer, a hidden layer, and an output layer. Each layer is connected by a weight, and a stable output is formed by setting the weight. The weighted neural network is generated for each expression, and the current AAM data is compared with each neural network to recognize that the probability is high as the current expression.

In the embodiment of the present invention, by applying a sigmoid function as a nonlinear active function to the reclination layer and the output layer, the crystal region is bounded by a gentle curve rather than a normal straight line, so that the BPNN learning algorithm for learning the reclination layer is performed. do. In this way, it is possible to secure an advantage of increasing the crystallization zone by increasing the recluse layer and using the nonlinear active function.

Here, for the model, the input layer I , the hermit layer H , and the output layer O are defined as the following [Formula 1].

[Equation 1]

In Equation 1, the weight W _ji for reducing the squared error E _p of the input layer I and the hermit layer H is readjusted as Equation 2 below.

[Equation 2]

The squared error E _p for all patterns P can be represented by equation (x). Here, (T _pk ) is a target vector, and (Q _pk ) is an output vector.

[Equation 3]

By minimizing the square error (E _p ) by learning each pattern P , the overall error function (E _t ) is minimized.

When applying the BPNN (Back Propagation Neural Network) facial expression recognition method using the feature points as described above, it is possible to classify various facial expressions than to apply only the basic expression. Thus, an interface role for which a user is suggested to use available expressions may be extended.

The facial expression and motion matching unit 320 searches the avatar motion DB 345 for a list of operations that can be taken in the facial expression according to the facial expression of the user recognized by the user facial expression recognition unit 310. Different emotions can be transmitted depending on the action you take with your expression. For example, if you laugh at the same scene, you must finger or grab your stomach and roll your feet. On the other hand, if you are smiling with joy, open your hands and smile with a big gesture. Accordingly, the facial expression and motion matching unit 320 extracts motion data related to a situation that can be described in the facial expression of the user recognized by the user facial expression recognition unit 310 and proposes it to the user.

The avatar facial expression and motion synthesizer 330 generates a dynamic avatar by applying the facial expression of the user recognized by the user facial expression recognizer 310 and the motion selected by the user to the user avatar.

As described above, the dynamic avatar generator 300 recognizes a user's facial expression using an image captured in real time through the camera 14 and proposes motion animations related to a situation in which the recognized facial expression can be described. Configure the actions to describe different situations. Accordingly, the user can quickly select the appropriate motion data from the motion range classified by the recognized facial expression without having to go through countless motion data one by one and find the desired motion data.

6 is a flowchart illustrating an avatar face generation process of a 3D user avatar according to an exemplary embodiment of the present invention.

The user inputs a face image by photographing his or her face with the camera 14 in the 3D avatar generation system (S210). Here, the user may receive or receive a face image from the outside or use a pre-stored face image.

The face region is recognized from the input face image (S215), and an AAM is applied to the image on which the face is recognized so that the areas of the eyes, nose, and mouth fit in the shape of the AAM (S220).

The appearance of the avatar face is generated using the AAM-applied image (S230), and the texture is synthesized using the skin of each region (S225).

Thereafter, the 3D avatar is synthesized using the appearance and texture of the avatar face, and the synthesis result is displayed to the user (S235).

Accordingly, when the user adjusts the size of each of the face parts or changes the skin texture, the avatar is modified accordingly (S240), and the modified avatar is displayed (S245).

7 is a flowchart illustrating a process of recognizing a user's facial expression in a method for producing a video using a 3D user avatar according to an exemplary embodiment of the present invention.

The user may input a face image by photographing a face with the camera 14 (S310).

Thus, the facial features are recognized by enhancing local features included in the face image (S315).

The AAM modeling unit 516 detects a feature point for modeling the face image by performing AAM fitting on the input face image (S320). In this case, the feature points used may include parameters of appearance and shape.

The back propagation neural network (BPN) processor 316 classifies facial expressions by applying the BPNN to the detected feature points (S325).

The facial expression recognition result recognized by the BPNN method is reflected and displayed on the avatar (S330).

When applying the BPNN facial expression recognition method, it is possible to classify various facial expressions than to apply only the basic expression. Thus, an interface role for which the user is suggested to use the available expression through the facial expression may be extended.

8 is a flowchart illustrating a video production method using a 3D user avatar according to a first embodiment of the present invention. This embodiment illustrates an interface provided to a user when generating a storytelling video using the video production device 100. In this case, the video production apparatus 100 may be mounted in the form of an application on a mobile device such as a user's smartphone or tablet.

The user may select an avatar to be generated by selecting a "movie sync" program, which is a video production program (S510). The video production apparatus 100 is activated according to the user's program execution selection.

The user avatar generating unit 200 generates a 3D avatar face by generating a shape and a skin of the 3D avatar using the captured image of the camera 14 (S515). The user avatar generator 200 may generate an avatar face by calculating a ratio between meshes to which an AAM is applied to the input image. Skin generation can also be converted to the avatar's skin using each part of the applied AAM mesh.

When the 3D avatar face is generated, it is edited according to the user's request (S520). The user may adjust detailed settings by changing the shape and skin of the avatar so that the generated 3D avatar face may be more similar to the face of the user.

Thereafter, the basic body shape is provided to the avatar, and the user adjusts the settings of each part of the body to be similar to himself (S525). The user may change size settings such as height, shoulders, belly, arms, thighs, muscle mass, etc. using the provided menu.

When the body setting is completed, various clothes or accessories are set on the avatar using the costume simulation technology (S530). When setting the avatar's costume, a costume creation tool based on a low polygon suitable for a mobile environment can be applied. As a result, the avatar may be produced closer to the style of the user.

When the style setting such as the clothes is completed, the generated user avatar is stored (S535).

9 is a flowchart illustrating a video production method using a 3D user avatar according to a second embodiment of the present invention, which illustrates a process of modeling a dynamic avatar expressing various expressions and actions.

In the video production apparatus 100, data such as an expression of an avatar, an operation of an avatar, and the like are constructed as databases 340 and 345. In addition, a matching operation is preset according to each expression.

Thus, the user photographs the expression of the user using the camera 14 of the smart device (S410).

Thus, the user facial expression recognition unit 310 recognizes the user's facial expression through the facial expression recognition algorithm (S415). Here, the user facial recognition unit 310 may classify various facial expressions rather than applying only basic expressions by recognizing facial expressions using a BPNN (Back Propagation Neural Network) method using feature points of the facial image.

When the facial expression is determined (S420), motion data corresponding to the facial expression is extracted from the avatar motion DB 345.

The extracted action list is provided to the user using a popup or a presentation method such as a list presentation (S430). Here, the operation on the list may be popped up as an icon according to the determined expression, and operations that cannot be suggested by other expressions may be provided as fixed icons.

As such, the facial expression recognition technology may simultaneously serve as an interface for effectively reflecting a user's facial expression on an avatar and selecting a gesture. If a user must select a gesture without using facial recognition technology, the user has to go through countless gestures and find a long time. Facial expressions allow you to quickly select from a range of motions that are shrunk in a bundle of laughter.

10 is a flowchart illustrating a video production method using a 3D user avatar according to a third embodiment of the present invention, illustrating a process of generating a user animation by synthesizing a dynamic avatar of a user with other content.

In the video production apparatus 100 for producing a fairy tale story, a basic animation including a text and a picture of a fairy tale content is stored in the basic animation DB 500.

Accordingly, when the user executes the program (S610), the background and the text are displayed when the previously produced picture is played in the story, and the screen for setting the operation of the user is displayed together (S615). The user photographs the story of a fairy tale story with the camera 14, and thus, the video production apparatus 100 recognizes the user's expression in real time and proposes an action suitable for the recognized expression.

The 3D avatar expresses the action selected by the user (S620), and synthesizes the background of the user's avatar and the fairy tale (S625). As a result, an animated moving picture animation is generated in which the user's three-dimensional avatar expresses facial expressions and actions according to the contents of the moving picture.

By such a configuration, the present invention implements a three-dimensional image differentiated from the e-book content of the conventional two-dimensional image, and also various and characteristic images are possible by implementing the captured image through the camera 14 as an avatar. .

The implementation animation animation using the user's avatar can be used in the field of early childhood education by setting the teacher of the kindergarten as a 3D avatar, or the implementation animation animation can be produced using the avatar of the parents at home. You can easily create and use animations.

Thus, those skilled in the art will appreciate that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the embodiments described above are to be considered in all respects only as illustrative and not restrictive. The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

The present invention generates a three-dimensional user avatar reflecting the user's face, and reflects the user's facial expressions photographed with the camera to the user avatar to generate a dynamic avatar moving the expression and motion, and the user's voice and the image of the user avatar By recording and editing together, the present invention can be used for a video production apparatus and method using a 3D user avatar to produce a video using the user avatar.

10: user terminal 12: communication unit
14 camera 16 display unit
18: user input unit 20: terminal control unit
100: video production device 110: animation production control
200: user avatar generation unit
210: 3D avatar face generator
220: 3D avatar body and costume generator
230: avatar face DB 232: avatar costume DB
234: Avatar body type DB 300: Dynamic avatar generator
310: user facial recognition unit 312: face recognition unit
314: feature extraction unit 316: BPNN processing unit
320: facial expression and motion matching unit 330: avatar facial expression and motion synthesis unit
340: Avatar facial expression DB 345: Avatar motion DB
400: user animation generator
450: user animation storage unit
500: basic animation DB

Claims (12)

Receiving and storing motion data of at least two avatars corresponding to one facial expression in advance;
Generating a 3D user avatar using a 2D user image; extracting feature points of a user face by applying AAM to the user image input in real time;
Recognizing an expression of the user's face by BPNN processing the feature point;
Retrieving and presenting motion data of the two or more avatars corresponding to the recognized facial expressions of the user's face to the user;
Generating a dynamic avatar by applying the motion data selected by the user and the recognized facial expression to the 3D user avatar; And
Generating a user animation by synthesizing the dynamic avatar of the user with a pre-stored background video.
The method of claim 1,
Generating a 3D user avatar using the 2D user image may include:
Calculating a proportion of each part of the user's face by applying AAM to the two-dimensional user image, and modeling a face shape of the three-dimensional user avatar according to the calculated value; And
And synthesizing the skin texture of the user's face to the avatar's face by applying AAM to the two-dimensional user's image.
The method of claim 1,
Generating a 3D user avatar using the 2D user image may include:
Synthesizing the pre-stored avatar's body type and clothes according to the user's selection to the face of the 3D user avatar; And
And generating a 3D user avatar of the user by editing the body shape and clothes of the avatar according to the user's selection.
delete The method of claim 1,
Feature points of the user's face,
A method for producing a video using a 3D user avatar, comprising at least one of a face shape and a skin texture.

delete The method of claim 1,
Generating the user animation by synthesizing the dynamic avatar of the user with a pre-stored background video,
And receiving the user's voice and storing the same together with the user's animation.
When driven from a mobile terminal,
Generating a 3D user avatar using a 2D user image; Recognizing the facial expression of the user in a user image input in real time and presenting operations corresponding to the recognized facial expression; Generating a dynamic avatar by applying the action selected by the user and the recognized facial expression to the 3D user avatar; A computer-readable storage medium having stored thereon a program for performing a video production method using a 3D user avatar including synthesizing the dynamic avatar of the user with a pre-stored background video.
A user avatar generation unit for modeling a face of a 3D user avatar using a 2D user image and generating a 3D user avatar by modeling a body shape and a costume of an avatar matching the avatar face according to a user's selection;
Recognize the user's facial expressions from the user image input in real time, present the motions corresponding to the recognized facial expressions, and apply the motions selected by the user and the recognized facial expressions to the 3D user avatar to generate a dynamic avatar. An avatar generator;
A basic animation DB storing a background video; And
A user animation storage unit configured to generate a user animation by synthesizing the dynamic avatar of the user with the background video,
The dynamic avatar generator,
An avatar motion database storing motion data of at least two avatars corresponding to the defined facial expression; A user expression recognizing unit which extracts a feature point by applying AAM to the user image input in real time and recognizes an expression of a user face by BPNN processing the feature point; An expression and motion matching unit for searching and presenting motion data of the at least two avatars corresponding to the recognized expression to a user; And an avatar facial expression and motion synthesizer configured to generate a dynamic avatar by applying the motion data of the avatar selected by the user and the user facial expression recognized by the user facial expression recognizer to the 3D user avatar. Video production device using.

10. The method of claim 9,
The user avatar generator,
A three-dimensional avatar face generator for calculating a proportion of each part of the user face by applying AAM to a two-dimensional user image and synthesizing a skin texture to model a three-dimensional user avatar face;
A database in which the body shape and costume data of the avatar are stored; And
3D avatar body shape and costume for synthesizing the body shape and costume of the avatar stored in the database according to the user's selection to the face of the 3D user avatar, and editing and modeling the body shape and costume of the avatar according to the user's selection Video production apparatus using a three-dimensional user avatar comprising a generation unit. delete 10. The method of claim 9,
Feature points of the user's face,
An apparatus for producing a video using a 3D user avatar, comprising at least one of a face shape and a skin texture.

Images