KR20200104105A - Apparatus and method for producing integrated information using image - Google Patents

Abstract

Disclosed are an apparatus and a method for producing integrated information using images. The apparatus for producing integrated information using images according to the present invention includes: an interface unit supporting user manipulation; a control unit that matches a specific target of a selected image with content information having a specified format, and generates fusion information formed by combining link information of the content information with the specific target; and a storage unit that stores images and stores the images and the fusion information. In addition, the method for producing integrated information using images according to the present invention includes: inserting the image into an image output area of an authoring application in response to the image selection; analyzing and confirming the existence of an object in the specific target when the specific target is selected from the image; providing at least one content information format corresponding to the object when the object exists; and generating fusion information by combining link information of the content information with the object when at least one content information is generated in the content information format. According to the present invention, since the user can link various types of content information wanted by the user even when using a specific SNS, the diversity of information delivery is improved by breaking away from the sense of isolation of boring platforms and standard SNS.

Description

Apparatus and method for producing integrated information using image}

The present invention relates to a content creation technology. More specifically, it relates to an apparatus and method for producing fusion information using an image that enables linking of various types of content information to a specific object of an image.

In modern society, SNS (Social Network Service) has become an important tool for forming human relationships. There are various platforms such as Facebook, Instagram, YouTube, Twitter, Google+, Pinterest, LinkedIn, Tumblr, blog, Naver Band, Kakao Story, and so on.

Initially, SNS was limited to the function of sending simple message information or the function of uploading created messages to the homepage, etc., but in recent years, it is used as a productive purpose for business and various information sharing while sharing one's interests and personality with the center of the individual. . As such, various services such as personal exchange, recording, and marketing are provided through SNS. To support these various services, dedicated applications are sometimes provided for each SNS platform.

As the transmission speed of a terminal connected to the network (eg, a smartphone) is improved and various applications are supported, the transmission of multimedia information is becoming free. Accordingly, as described above, a dedicated application for optimizing images and videos may be provided on SNS for various platforms.

However, these applications are standardized for each SNS platform, so when using a specific SNS, there is a limit to using only the application. Accordingly, there is a disadvantage in that the type of content information is standardized or limited. This is also because the sizes of images or videos are standardized for each SNS platform.

Meanwhile, the present applicant has proposed a technology for linking content information to a specific object of an image included in information transmitted and received through a network.

This technology can provide a variety of information together by fusing content information to a specific object of an image, rather than simply transmitting an image.

In addition, since this technology can link and distribute multiple links corresponding to a specific target of an image, it can be used for marketing through SNS, and the recipient who receives the message containing the linked image can receive information related to the link. The additional information related to the link can be reused by modifying or applying the target area information and link information included in the message to other images.

However, this technology uses SNS economically and easily and efficiently by providing a user interface that enables sharing and distribution by matching links to specific targets of images, but content information linked to specific targets of images A technology that can be set in an information format has not yet been proposed.

Korean Intellectual Property Office Patent Registration No. 10-1250107 (announced 2013.04.03.), "System and method for creating and managing product meta information in TV contents in a smart TV environment" Korean Intellectual Property Office Patent Registration No. 10-1688742 (announced on December 21, 2016), "How to distribute and use image links"

Accordingly, the present invention has been made to solve the problems of the prior art, and an object of the present invention is to produce fusion information using an image that allows a user to arbitrarily set various types of content information linked to a specific object of an image. It is to provide an apparatus and method.

Convergence information production apparatus using an image of the present invention for achieving the above object, preferably, an interface unit for supporting user manipulation; A controller configured to match a specific object of the selected image with content information of a specified format, and generate fusion information in which link information of the content information and the specific object are combined; And a storage unit that stores an image and stores the image and the fusion information.

The control unit may include a program support unit supporting user manipulation for linking the content information; An image insertion unit for inserting the selected image into the image output area on the program screen; An object identification unit that checks an object in response to designating a specific object of the output image; A content information format providing unit that provides a content information format to select a content information format to be matched with the object; And a fusion information generation unit for fusing link information of the object and content information of a selected format.

In the object identification unit, a method of detecting an object using region proposal extraction including a method of generating a heat map based on a convolution neural network (CNN), and properties of the object Based on the multi-tasking learning method used, an object region is extracted and a bounding box is adjusted to detect an object, a region proposal is extracted for an image, and the extracted region is proposed. One of the methods of detecting an object by extracting the features of each of the region proposals through feature extraction based on a convolutional neural network can be used.

The control unit may further include a communication unit connected to a network.

On the other hand, the method of producing fusion information using an image of the present invention preferably includes the steps of inserting the image into the image output area of the authoring application in response to image selection; If a specific object is selected from the image, analyzing and confirming whether an object exists in the specific object, and if the object exists, providing at least one content information format corresponding to the object; And generating fusion information by combining link information of the content information and the object when at least one content information is generated from among the content information types.

The fusion information may be stored or transmitted as an SNS (Social Network Service) message. In the storage and transmission, it is preferable to further include additional information including information on the object, the type of link tool applied to the object, and region information including the location of the object.

The method of analyzing and confirming the existence of the object includes detecting an object using region proposal extraction including a method of generating a heat map based on a convolution neural network (CNN). Method, a method of detecting an object by performing extraction of an object area and adjustment of a bounding box based on a multi-tasking learning method using the properties of the object, and extracting a region proposal for an image In addition, any one of the methods of detecting an object by extracting features of each of the region proposals through feature extraction based on a convolutional neural network from the extracted region proposals may be used.

When generating the content information, it is preferable to run a content information generating application capable of generating specific content information according to the content information type.

An image displaying a link tag on the object may be used. In this case, the link tag may be an action tag.

As described above, according to the apparatus and method for producing fusion information using images according to the present invention, even when using a specific SNS, it is possible to link various types of content information desired by the user, so that information is delivered by breaking away from the sense of isolation between the boring platform and the standardized SNS. Can improve its diversity.

In addition, according to the present invention, since all the content information formats desired by the user can be linked to images, it can be usefully used for promotion, introduction, catalog, and homepage. That is, since a plurality of pieces of information are linked to one image, various content information can be included together even though the screen is simplified or simply configured.

1 is a configuration diagram of a service system to which the present invention is applied.
2 is a block diagram of an apparatus for producing fusion information using an image according to an embodiment of the present invention.
3 is a block diagram of a control unit according to an embodiment of the present invention.
4 is a flowchart of a method of producing fusion information using an image according to an embodiment of the present invention.
5 is a flowchart illustrating a process of a control unit according to an embodiment of the present invention.
6 is a diagram showing types of content information formats according to an embodiment of the present invention.
7 is a diagram showing types of content information formats according to another embodiment of the present invention.
8 and 9 are diagrams according to whether views and statistics exposure are set according to an embodiment of the present invention.
10 and 11 are diagrams showing an action tag according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to a preferred embodiment of the present invention and the accompanying drawings, but it will be described on the premise that the same reference numerals refer to the same elements.

In the detailed description of the invention or in the claims, when any one component "includes" another component, it is not construed as being limited to only the component unless otherwise stated, and other components It is to be understood that it may further include.

In addition, components named as "~ means", "~ unit", "~ module", and "~ block" in the detailed description or claims of the invention mean a unit that processes at least one function or operation, Each of these may be implemented by software or hardware, or a combination thereof.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are assigned to similar parts throughout the specification.

Hereinafter, an example in which the apparatus and method for producing fusion information using an image of the present invention are implemented will be described through specific embodiments.

First, a brief description will be given of a service system for producing converged information using an image proposed in the present invention.

1 is a configuration diagram of a service system to which the present invention is applied.

Referring to FIG. 1, a service system to which the present invention is applied largely includes user terminals 1a, 1b, and 1c, and an SNS server 2 communicating with the user terminals 1a, 1b, and 1c through a network. .

The wireless Internet may be used as the network, but is not limited thereto, and various types of communication methods may be used.

Users of the user terminals 1a, 1b, and 1c will create an SNS message including an image using a user interface for executing an SNS application installed in the terminal and creating a new SNS message. As the user terminals 1a, 1b, and 1c, a desktop type computer capable of connecting to a communication network for data communication, or a portable terminal such as a notebook computer, a smart phone, or a personal digital assistant (PDA) may be used.

The SNS server 2 is an equipment that enables users to use SNS, and supports message transmission/reception between users. In the present invention, the SNS server 2 is free communication and information sharing between users such as Facebook, Instagram, YouTube, Twitter, Google+, Pinterest, LinkedIn, Tumblr, blog, Naver Band, Kakao Story, and It refers to an online platform that creates and reinforces social relationships through social networking, etc., and messages exchanged by users through the SNS server 2 will be referred to as SNS messages.

In the service system configured as described above, when creating an SNS message, a specific target of an image and content information are matched, and fusion information obtained by combining link information of the specific target (object) and the content information may be included in the message. That is, content information desired by the user may be matched with a specific target in at least a partial area of the image, and a plurality of specific targets matching the link information may be set.

Subsequently, when a command to transmit the created SNS message is input from the user, the created SNS message is transmitted to the SNS server 2 through the network, and the SNS server 2 processes the SNS message to be accessible on the SNS.

For example, it is posted to the sender's account so that followers following the sender can see it on the SNS, or if the type of SNS has a predetermined recipient, it is delivered to the corresponding recipient terminal. That is, in the present invention, the SNS message is not limited to a type of message that is delivered in one direction after a recipient is determined, and refers to a unit of content posted or delivered on the SNS.

Here, the transmission of the SNS message means processing to be viewed by users other than the sender on the SNS, and is not necessarily limited to processing in the form of being delivered to the recipient. Accordingly, the receiver terminal is interpreted as a terminal that accesses the SNS message transmitted by the sender terminal, and is not necessarily limited to a terminal that is directly delivered from the sender terminal.

Meanwhile, when transmitting an SNS message, information on a specific target to which link information is combined will be included in the message as additional information and transmitted.

2 is a block diagram of an apparatus for producing fusion information using an image according to an embodiment of the present invention.

Referring to FIG. 2, the apparatus for producing fusion information using an image of the present invention includes an interface unit 11 that supports user manipulation, and a specific object of an image selected in response to the user manipulation and content information in a format desired by the user. Matching is made, and a control unit 12 that generates fusion information that combines link information of a corresponding specific object (object) and content information, and a storage unit 13 that stores an image and stores image and fusion information. Include.

The apparatus for producing fusion information using the image of the present invention may be included in the user terminals 1a, 1b, 1c or the SNS server 2.

When the device for producing fusion information using the image of the present invention is included in the SNS server 2, the control unit 12 is configured to be connected with a communication unit (not shown).

On the other hand, the storage unit 13 stores an application for producing fusion information using an image. Applications include authoring applications and content information generating applications.

In the apparatus for producing fusion information using an image of the present invention configured as described above, it supports to select an image stored in the storage unit 13 in response to a user manipulation through the interface unit 11, and also allows a specific object and user of the image It supports to select the desired content information format. Accordingly, when content information is generated for a specific object of the image, the controller 12 combines the specific object of the image with link information of the content information in a format desired by the user. The combined fusion information may be stored in the user terminals 1a, 1b, 1c, transmitted as an SNS message, or uploaded to a home page.

3 is a block diagram of a control unit according to an embodiment of the present invention.

Referring to FIG. 3, the control unit 12 of the present invention includes a program support unit 121 supporting a user operation for linking content information in a format desired by the user, and an image selected by the user in an image output area on the program screen. An image inserting unit 122 for inserting a message, an object identification unit 123 for checking an object in response to designating a specific target of the output image, and a content information format so that a content information type to be matched to the object can be selected. A content information format providing unit 124 to be provided, and a fusion information generation unit 125 for fusing link information between an object and content information in a format selected by a user.

Here, the object identification unit 123 may detect an object through deep learning learning.

For example, as an object detection method, a method of detecting an object using region proposal extraction including a method of generating a heat map based on a convolution neural network (CNN) is used, or , Using a method of detecting an object by performing object region extraction and adjustment of a bounding box based on a multi-tasking learning method using object properties, or extracting region suggestions for images Then, a method of detecting an object by extracting features of each of the region proposals through feature extraction based on a convolutional neural network from the extracted region proposals may be used. For such an object detection method, refer to “A method and system for video surveillance based on artificial intelligence using deep learning” in Patent Application No. 10-2017-0036874.

Meanwhile, since object detection is based on deep learning learning based on vast amounts of data, it is preferable to be performed in the SNS server 2.

The control unit 12 of the present invention configured as described above displays the authoring application on the screen in response to a user operation for executing the authoring application.

Subsequently, when a specific image is selected by the user using the menu of the authoring application, the image is inserted into the image output area on the program screen.

Next, when the user designates a specific object in the output image, the process of analyzing and confirming the specific object is performed.

When an object is identified through analysis of a specific object, the content information type providing unit 124 provides a content information type menu so that a content information type to be matched can be selected in response to the object. Accordingly, when a specific content information type is selected by the user, a content information generating application capable of generating the corresponding content information is executed. Here, it is preferable that the content information is set and stored in advance corresponding to the object. Accordingly, it is preferable to provide a preset content information format when checking the object.

Meanwhile, when a user creates corresponding content information and links it to an object, fusion information that combines the object and link information is generated.

Thereafter, the fusion information is stored in the storage unit 13 or transmitted to the counterpart terminal. In this case, when a specific target transmits an image designated as an SNS message, fusion information in which the image and link information are combined is transmitted, and additional information about the image may be transmitted together.

Herein, a method of producing fusion information using an image of the present invention using the system configured as described above will be described.

4 is a flowchart of a method of producing fusion information using an image according to an embodiment of the present invention.

Referring to FIG. 4, the controller 12 inserts a corresponding image into an image output area in response to image selection information input through a menu of an authoring application (S1).

When a specific object is selected from the image, the existence of the object in the specific object is analyzed and checked, and if the object exists, a content information format corresponding to the object is provided (S2).

When content information is generated using the provided content information format, fusion information is generated by combining the object and link information (S3).

Thereafter, the fusion information in which the image and link information are combined may be stored or transmitted (S4).

Here, the content information format may be a search site previously designated by the user. When the content information format is a search site, after searching for an object, specific information may be designated as content information from information output as a search result. That is, any one of various formats including, for example, a price comparison site, a shopping mall site, YouTube, and a memo may be designated as content information. In addition, as a content information format, a notepad that a user can directly create and edit may be used.

Meanwhile, content information may be automatically matched corresponding to the corresponding object. That is, the content information format matched according to the type of object may be preset by the user or the like.

5 is a flowchart illustrating a process of a control unit according to an embodiment of the present invention.

Referring to FIG. 5, when there is a request for execution of an authoring application by a user, the program support unit 121 of the control unit 12 executes the authoring application (S11).

On the other hand, when a specific image is selected in response to a user manipulation using the menu of the executed authoring application, the image insertion unit 122 of the control unit 12 inserts the specific image into the image output area on the screen of the authoring application (S12). ).

Subsequently, when a specific object is selected by the user from the specific image (S13), the object identification unit 123 of the control unit 12 analyzes the object existing in the specific object (S14). When an object exists, the content information format providing unit 124 of the controller 12 provides the content information format matched to the object on the screen (S15). On the other hand, when an object does not exist or an object cannot be analyzed or recognized, a specific content information format can be directly designated.

When a specific content information type is selected by the user, the program support unit 121 of the control unit 12 executes a content information generation application capable of generating content information according to the specific content information type (S16). In this case, the content information generating application may be executed within the authoring application. Accordingly, a task of linking the content information generated through the content information generating application to the object may be executed through the menu of the authoring application.

Accordingly, in response to the user's manipulation of the content generation application, the control unit 12 generates content information (S17).

Subsequently, in response to the link operation using the menu of the authoring application executed, the fusion information generation unit 125 of the control unit 12 generates fusion information by combining the object and the link information (S18).

Accordingly, the control unit 12 may store fusion information in which the image and link information are combined, or may perform a transmission processing operation in response to a transmission processing request by a user (S19). At this time, when storing and transmitting, additional information including information on the object, the type of link tool applied to the object (shape of the area), the size of the area, and the location of the area (or object) may be further included. I can.

Example

6 is a diagram showing types of content information formats according to an embodiment of the present invention.

Referring to FIG. 6, a plurality of objects may be designated in one image, and each object may be linked with various types of content information. That is, information about sunglasses uses a price comparison site as content information. That is, information about sunglasses is set so that a link is made to a price comparison site. Meanwhile, the information on the necklace uses the shopping mall site as content information. That is, the information on the necklace is set so that a link to the shopping mall site is made.

As described above, the present invention does not provide one standardized content information for a plurality of objects existing in an image, but supports the user to select desired or most easily transmitted content information.

7 is a diagram showing types of content information formats according to another embodiment of the present invention.

Referring to FIG. 7, a case in which content information is linked in a format desired by a user for any content such as hobbies, leisure, music, and cooking is illustrated. In this embodiment, the case of using media articles, YouTube, Facebook, and maps as content information, and it can be seen that all of these content information are linked to a single image. In addition, navigation, scrap, etc. may be re-linked as content information.

In addition, it can be used for promotion, introduction, catalog, and homepage. According to the present embodiment, it can be seen that a long and long promotional page, in which all contents must be read in order to grasp a specific content, is simply displayed as one image. Accordingly, the user can select and read only the information he wants on the corresponding display screen, and a link to the content information set in advance will be made in response to the selection of the desired information.

Then, the process of generating fusion information using images will be briefly described here.

When a specific area or specific object is designated for at least one specific image, a name of an object for the specific area or specific object is generated. Then, one of the content information (blog, cafe, shopping mall, lowest price, price comparison, customer reviews, etc.) to be matched to the object is selected. If you click the Apply Link button composed of the menu of the content information generating application, the content information is linked to the object.

Meanwhile, in the present embodiment, a method of directly generating an object name in a specific area or specific object is illustrated, but the object may be analyzed and confirmed through image analysis on a specific area or specific object. That is, an object is identified through image analysis on a specific area or a specific target, and the name of the object is automatically generated, or preset content information is output for the object, or the output content information is linked to the object. You can do it.

On the other hand, briefly explaining the linking process for an object from the generated fusion information, when a click is made on an image at an arbitrary position, a link tag is displayed and information about the object (title, etc.) is output. do. Then, when a desired object or title is clicked, a link is made to the corresponding content information. Meanwhile, information on the object may be re-entered by the user.

In addition, popular content and customer needs can be identified, and for this, it can be configured to identify popular content and customer needs by number of views by image and link tag. In other words, it is possible to predict the user's usage pattern, the customer's interest and needs based on the user experience.

Meanwhile, as an example of the page category configuration, for example, all main pages of a magazine may be composed of images, and when a specific image is selected from among the images, a page in which the specific image is enlarged is output. Subsequently, when a user clicks at an arbitrary location, a link tag is displayed and information (title, etc.) about the object is displayed. That is, it can be configured to continuously move to the detail page in response to an image click.

And, as an additional function, it provides additional functions such as basic profile, follower corner, following, like view, person recommendation, scrap, comment view, link copy, and immediate sharing function, and functions that require SNS consider user experience and accessibility. To upgrade the mod.

In addition, it is possible to save gallery and view/share/edit/edit/upload detailed contents by link tag. That is, a share/edit (edit)/upload icon is provided on the screen so that a corresponding function can be executed, and a new image (post) that has been edited (modified) can be registered.

8 and 9 are diagrams according to whether views and statistics exposure are set according to an embodiment of the present invention.

8 and 9, it is possible to set the number of views and statistics exposed (public) or non-exposed (private) on the image for the link tag. In the case of disclosure, detailed statistics on link tags and user trend analysis information are provided. If you display the statistics display icon next to the information of the link tag (object), and click the statistics display icon, you can switch to the detailed statistics screen. Also, you can switch to the detailed statistics screen by clicking the number of views displayed on the image.

10 and 11 are diagrams showing an action tag according to an embodiment of the present invention.

Referring to FIGS. 10 and 11, the action tag presented in this embodiment is a modified example of the link tag, as an area vector display for the periphery of an object, and can be supported in various animation formats such as GIF and FLASH. You can increase the effect of focusing attention through the movement of the action tag.

The method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

11: interface unit
12: control unit
121: Program Support Department
122: image insertion section
123: object verification unit
124: Content information format provision unit
125: fusion information generation unit
13: storage unit

Claims (11)

An interface unit supporting user manipulation;
A controller configured to match a specific object of the selected image with content information of a specified format, and generate fusion information in which link information of the content information and the specific object are combined; And
Convergence information production apparatus using an image comprising; a storage unit that stores an image and stores the image and the fusion information. The method of claim 1,
The control unit,
A program support unit supporting a user operation for linking the content information;
An image insertion unit for inserting the selected image into the image output area on the program screen;
An object identification unit that checks an object in response to designating a specific object of the output image;
A content information format providing unit that provides a content information format to select a content information format to be matched with the object; And
Convergence information production apparatus using an image comprising a; a fusion information generation unit for fusing the link information of the object and the content information of the selected format. The method of claim 1,
In the object verification unit,
A method of detecting an object using region proposal extraction, which includes a method of generating a heat map based on a convolution neural network (CNN), and multitasking learning using object properties. -tasking learning) method to detect an object by extracting an object region and adjusting a bounding box, performing region proposal extraction for an image, and based on a convolutional neural network from the extracted region proposals An apparatus for producing fusion information using an image using any one of methods of detecting an object by extracting the features of each of the region proposals through feature extraction of. The method of claim 1,
The control unit further includes a communication unit connected to a network. Inserting the image into an image output area of an authoring application in response to an image selection;
If a specific object is selected from the image, analyzing and confirming whether an object exists in the specific object, and if the object exists, providing at least one content information format corresponding to the object; And
When at least one content information from among the content information formats is generated, generating fusion information by combining link information of the content information with the object. The method of claim 5,
The convergence information production method using an image that is stored or transmitted as an SNS (Social Network Service) message. The method of claim 6,
In the storage and transmission, the method for producing fusion information using an image further includes information on the object, a type of link tool applied to the object, and additional information including region information including the location of the object. The method of claim 5,
The method of analyzing and confirming the existence of the object,
A method of detecting an object using region proposal extraction, which includes a method of generating a heat map based on a convolution neural network (CNN), and multitasking learning using object properties. -tasking learning) method to detect an object by extracting an object region and adjusting a bounding box, performing region proposal extraction for an image, and based on a convolutional neural network from the extracted region proposals A method of producing fusion information using an image using any one of the methods of detecting an object by extracting the features of each of the region proposals through feature extraction of. The method of claim 5,
When generating the content information, a method for producing fusion information using an image that executes a content information generating application capable of generating specific content information according to the content information type. The method of claim 5,
A method of producing fusion information using an image displaying a link tag on the object. The method of claim 10,
The link tag is an action tag, a method of producing fusion information using an image.

Images