JP5704909B2 - Attention area detection method, attention area detection apparatus, and program - Google Patents

Description

  The present invention relates to attention area detection in an image, and more particularly to a method of describing visual saliency in attention area detection and attention area detection using the visual saliency.

  For example, Patent Document 1 proposes the following method as a method for detecting a group of regions meaningful to a human being from an input image. That is, a plurality of types of basic feature images are extracted from the input image, and a multi-resolution image that is a multi-resolution expression is extracted. In addition, by extracting a plurality of resolution difference images, which are differences between images of different resolutions for each type of multi-resolution image, and integrating the resolution difference images of different resolutions for each type of resolution difference image, Extract images. Furthermore, a region of interest can be detected as a region having a saliency above a threshold value in the obtained visual saliency image.

JP 2009-3615 A

Hido, S .; Tsuboi, Y .; , Kashima, H .; , Sugiyama, M .; , & Kanamori, T .; Statistical outer detection using directivity ratio estimation. Knowledge and Information Systems, to appear.

  However, when the visual saliency is calculated by directly using the basic feature image from such an input image, it is easily affected by noise due to environmental and observational factors, and the detection accuracy of the attention area is reduced. There was a problem.

In order to solve the above-described problem, an attention area detection method according to the present invention includes a first extraction area and a second extraction area different from the first extraction area for each of a plurality of points in an input image. And an extraction region setting step for extracting feature amounts from each of the first extraction region and the second extraction region, and the first region set for the same point. A calculation step of calculating visual saliency at the plurality of points based on a probability density relationship between an extraction region and a feature amount of the second extraction region;
A detection step of detecting a region of interest based on visual saliency at the plurality of points .

  According to the present invention, it is possible to improve the detection accuracy of a region of interest.

It is a functional lineblock diagram of the attention field detecting device concerning a 1st embodiment. It is a figure explaining the function of a detection part. It is a figure explaining a data extraction area. It is a figure explaining the detection point from which the maximum value of visual saliency was detected. It is a figure explaining a local attention area group. It is a figure explaining the attention area set. It is a figure explaining the function of a learning part. It is a figure explaining a parameter candidate. It is a figure explaining the process of a coefficient calculation part. It is a figure explaining determination of a parameter. It is a figure explaining a parameter candidate. It is a block diagram which shows the hardware constitutions of the information processing apparatus which implement | achieves an attention area detection apparatus.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing a functional configuration of a region of interest detection device 1 according to the first embodiment of the present invention. The object identification device according to the present embodiment is realized using a semiconductor integrated circuit (LSI). As illustrated in FIG. 1, the attention area detection device 1 includes a learning unit 11, a detection unit 12, and a control unit 13. These components correspond to the functions performed by the attention area detection device 1.

  The function performed by the attention area detection device 1 is roughly divided into two. One of them is a learning function, which is executed by the learning unit 11. The other is a detection function, which is executed by the detection unit 12. Here, the learning unit 11 calculates the parameter α used by the detection unit 12. In addition, the detection unit 12 detects a region of interest in the input image using the parameter α calculated by the learning unit 11. Further, the control unit 13 controls each component of the object identification device 1.

  On the other hand, the detection result of the attention area is transmitted to a higher-level device of the attention area detection device 1 and applied to various applications (such as an autofocus function of a digital still camera and an abnormality detection function of a security camera).

  FIG. 2 is a block diagram illustrating a functional configuration of the detection unit 12. As illustrated in FIG. 2, the detection unit 12 includes a feature amount calculation unit 121 that is a first calculation unit, a visual saliency calculation unit 122 that is a second calculation unit, a maximum value search unit 123, and an integration unit 124. Is done.

  FIG. 3 is a diagram for explaining the data extraction area. As shown in FIG. 3, the feature amount calculation unit 121 executes extraction region setting for setting a training data extraction region and a verification data extraction region in an input image input from the outside of the attention region detection device 1. A predetermined number of low-order feature quantities are extracted from a region at random.

  Here, the training data extraction area and the verification data extraction area are given as two circular areas having different sizes. Of these, the circular area with the larger radius is set as the verification data extraction area as the first extraction area, and the circular area with the smaller radius is set as the training data extraction area as the second extraction area.

  Further, the radius of the training data extraction region and the verification data extraction region, the number of low-order feature amounts to be extracted, and their types (luminance value, edge strength, texture, etc.) are determined in advance by the learning unit 11 and are controlled by the control unit 13. Communicated.

  Note that the low-order feature amount obtained here is output to the visual saliency calculating unit 122.

  The visual saliency calculator 122 calculates a visual saliency S at an arbitrary point in the input image based on the low-order feature obtained by the feature calculator 121. Specifically, based on the ratio of probability density estimated using the low-order feature value obtained from the training data extraction region and the low-order feature value obtained from the verification data extraction region, The saliency S is calculated.

Here, the density ratio (p _test / p _train ) between the probability density p _train of the low-order feature quantity in the training data extraction region and the probability density p _test of the low-order feature quantity in the verification data extraction region is, for example, non-patent It can be calculated by using the density ratio estimation method of Document 1.

  From this, the visual saliency S is the ratio of the probability density of the low-order feature quantity when the low-order feature quantity extracted by the feature quantity calculation unit 121 is a single type (for example, only the luminance value Y). It is given by the reciprocal 1 / σ of the standard deviation σ.

The visual saliency S is as follows when there are a plurality of types of low-order feature amounts extracted by the feature amount calculation unit 121 (for example, when there are three types of luminance value Y, edge strength E, and texture T). It becomes like this. That is, the standard deviation (σ _Y , σ _E , σ _T ) of the ratio of the parameter α = (α _Y , α _E , α _T ) input from the learning unit 11 and the probability density due to each low-order feature amount is calculated. Using a linear sum of reciprocals (1 / σ _Y , 1 / σ _E , 1 / σ _T ), it is given as in equation (1).

Further, the visual saliency S can be easily expanded when the number of low-order feature amounts extracted by the feature amount calculation unit 121 (for example, N types) can be easily extended. The degree S is given by equation (2). Alternatively, more generally, the visual saliency S may be a non-linear function related to σ _n (n = 0 to _N ) as in Expression (3).

  The obtained visual saliency S is output to the local maximum search unit 123.

  The local maximum search unit 123 uses the feature amount calculation unit 121 and the visual saliency calculation unit 122 to calculate the visual saliency S = S (x, y) at various points (x, y) in the input image. In order to calculate and obtain a statistical distribution of the visual saliency S, a point giving a maximum value (or a value equal to or greater than a predetermined threshold) is detected.

The point p _k (k = 0 to K) at which the maximum value (or a value equal to or greater than the predetermined threshold value) is detected is output to the integration unit 124. However, K represents the number of points (x, y) at which the maximum value (or a value equal to or greater than a predetermined threshold value) of the visual saliency S is detected.

The integration unit 124 corresponds to the detection point p _k (k = 0 to K) at which the maximum value of the visual saliency S (or a value equal to or greater than a predetermined threshold) obtained by the maximum value search unit 123 is detected. The local attention area setting for setting the area as the local attention area is executed. Then, these local attention areas are integrated based on the distance d between the two detection points.

For example, as shown in FIG. 4, it is assumed that the maximum value of visual saliency S (or a value equal to or greater than a predetermined threshold value) is obtained at three points ( _pi , _pj , _pk ) in the input image. In FIG. 4, p _i , p _j , and _pk are the centers of the respective circular regions, and the size of each radius corresponds to the size of the radius of the training data extraction region. First, paying attention to the points p _i and p _j , if the distance d _ij between these points is smaller than the predetermined threshold value d _th , the two points (p _i , p _j ) are integrated into the same group and are locally A group of attention areas is generated. Next, paying attention to the points p _j and p _k , if the distance d _jk between these points is smaller than the predetermined threshold value d _th , the two points (p _j , p _k ) are integrated as the same group. Similarly, paying attention to the points p _i and p _k , if the distance d _ik between these points is smaller than the predetermined threshold value d _th , the two points are integrated as the same group. However, the predetermined threshold value d _th is determined by the learning unit 12 and transmitted by the control unit 13.

This is executed for all sets of points p _k (k = 0 to K) at which the maximum value of visual saliency S (or a value equal to or greater than a predetermined threshold) is detected. Thereby, the local attention area corresponding to the point p _k (k = 0 to K) is integrated into a plurality of groups (local attention area group) g _m (m = 0 to M) (FIG. 5). Further, a total value S _m (m = 0 to M) of visual saliency S is calculated for each group g _m (m = 0 to M), and a rectangular region including the group g _{m ′} that gives the maximum value is calculated. The region of interest is set (FIG. 6), and the region-of-interest setting is executed with this as the final region of interest. However, M represents the number of groups (M = 3 in the example of FIG. 5).

  Note that the position and size of the region of interest on the input image obtained here are output to the control unit 12.

  The above means corresponds to an example of the detection unit 12 in the attention area detection device 1.

  FIG. 7 is a block diagram showing a functional configuration of the learning unit 11. As shown in FIG. 7, the learning unit 11 includes an image database 111 and a coefficient calculation unit 112. The learning unit 11 determines the parameter α used in the detection unit 12 based on a GT (ground truth) image stored in the image database 111. Here, the GT image is an image in which a region of interest is defined by a rectangular frame indicated by a dotted line in FIG. 6 as a meaningful object region in the input image, and its position and size are set in advance. The coefficient calculation unit 112 learns the parameter α so that the same attention area detection result can be obtained using the detection unit 12.

Specifically, the parameter α candidates α _{0, m} (m = 0 to M) are given by the coordinates of random points in the N-dimensional space. However, N represents the number of types of low-order feature values, and in this embodiment, three types (N = 3) of luminance value Y, edge strength E, and texture T are considered. M represents the number of candidates for parameter α.

Then, the parameter α candidates α _{0, m} (m = 0 to M) are coordinates of random points in a sphere with a radius 1 in a three-dimensional space (α _Y , α _E , α _T ) as shown in FIG. Given in. Here, the candidate number M of the parameter α is relatively small (for example, 10 to 50) when it is desired to shorten the learning time, and is relatively large (for example, 100 to 500) when the detection accuracy of the attention area is important. Set to.

Also, the coefficient calculation unit 112 uses the detection unit 12 to detect the detection accuracy R _{0, m} (m = 0 to M) when each of the parameter α candidates α _{0, m} (m = 0 to M) is used. Measure. Here, as shown in FIG. 9, the coefficient calculation unit 112 compares the GT image stored in the image database 111 with the attention area detection result obtained using the detection unit 12. The detection accuracy R _{0, m} is given by the average value of the area ratio (s / s ′) between the overlapping area s and the area s ′ of the region of interest in the GT image.

As a result, as shown in FIG. 10, it is assumed that the candidate α _{0, m ′} for the parameter α that gives the best detection accuracy R _{m ′} can be identified. Next, as shown in FIG. 11, the coefficient calculation unit 112 converts a new parameter α candidate α _{1, m} (m = 0 to M) to a point α in the three-dimensional space (α _Y , α _E , α _T ). It is given by the coordinates of a random point in the sphere having a radius of 1 * γ (0 <γ <1) with _{0, m ′} as the center. Similarly, the coefficient calculation unit 112 uses the detection unit 12 to detect the detection accuracy R _{1, m} (m = 0) when each of the parameter α candidates α _{1, m} (m = 0 to M) is used. ~ M).

The coefficient calculation unit 112 repeats the above operation in the same manner, and when the radius of the sphere becomes smaller than the predetermined threshold γ _th , the coefficient calculation unit 112 aborts the process and outputs the parameter α obtained at that time to the control unit 13. However, the predetermined threshold γ _th is determined by trial and error as a non-negative real number. The above corresponds to an example of the learning unit 11 in the attention area detection device 1.

  The attention area detection result obtained in this way is transmitted to a higher hierarchy of the attention area detection device 1. For example, in a situation where the target object is picked up by the robot arm after the attention area is detected, the attention is transmitted to a device, a program, or the like for controlling the robot arm and the attention area detection device 1 and applied to various applications. .

  The above corresponds to an example of the attention area detection device 1.

[Other Embodiments]
In the first embodiment, the attention area detection device executes both the learning function and the identification function. However, only one of the learning function and the detection function may be executed.

  In the first embodiment, in the feature amount calculation unit 121, the training data extraction region and the verification data extraction region are two circular regions, but other arbitrary shapes such as an elliptical shape and a rectangular shape are used as well as a circular shape. It may be used. Further, the final attention area is not limited to the rectangular area, but may be another shape.

  In the first embodiment, the visual saliency calculating unit 122 calculates the visual saliency S using Expression (2), but instead of using Expression (2), the area s of the training data extraction region is calculated. And equation (4) may be used. Moreover, you may use Formula (5) using the arbitrary function f (s) of the area s of a training data extraction area | region.

  Further, in the first embodiment, the visual saliency calculation unit 122 calculates the visual saliency S using the probability density of the low-order feature quantity in the training data extraction area and the probability of the low-order feature quantity in the verification data extraction area. As a relationship with the density, it was calculated based on the density ratio between the two. However, the visual saliency S may be calculated based on absolute differences between histogram bins corresponding to the respective histogram bins.

  Note that the present invention can be applied to a system (for example, a copier, a facsimile machine, etc.) consisting of a single device even when applied to a system composed of a plurality of devices (for example, a host computer, interface device, reader, printer, etc.). You may apply.

  The present invention can also be realized by performing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and the computer (or CPU or MPU) of the system or apparatus reads out and executes the program. It is processing to do.

  FIG. 12 is a block diagram illustrating a hardware configuration of an information processing apparatus that realizes the above-described attention area detection device by executing a program.

  The CPU 201 executes various programs and controls each part of the apparatus. The ROM 202 is a non-volatile memory, and stores programs and the like necessary for initial operation of the information processing apparatus. A RAM 203 provides a work area for the CPU 201 and temporarily stores a program read from the secondary storage device 204. The secondary storage device 204 records the program 210 used by the CPU 201 and stores an image used in the image database. Note that the program 210 includes an OS 211, an application 212, a module 213, and data 214.

  Each device 201 to 204 exchanges information through the bus 205. The information processing apparatus is connected to a display 206, a keyboard 207, a mouse 208, and an I / O device 209 via a bus 205.

  A display 206 is used to display information such as a processing result and a progress of the processing to the user. A keyboard 207 and a mouse 208 are used for inputting an instruction from the user, and in particular, the mouse 208 is used for inputting a position on the display. The I / O device 209 is used for capturing an image to be processed. For example, the I / O device 209 captures an input image from a photographing apparatus that photographs a target object. Also, the I / O device 209 can output the attention area obtained as the information processing result to another information processing apparatus such as a photographing apparatus or an image processing apparatus.

Claims (15)

An extraction region setting step of setting a first extraction region and a second extraction region different from the first extraction region for each of a plurality of points in the input image;
A feature amount extraction step of extracting a feature amount from each of the first extraction region and the second extraction region;
A calculation step of calculating visual saliency at the plurality of points based on a probability density relationship between the first extraction region and the second extraction region set for the same point; attention area detection method characterized by having a detection step of detecting a region of interest based on the visual saliency of the plurality of points.   The attention area detection method according to claim 1, wherein in the feature quantity extraction step, a plurality of types of feature quantities are extracted from each of the first extraction area and the second extraction area.   The attention area detection method according to claim 2, wherein the plurality of types of feature amounts include any one of a luminance value, an edge strength, and a texture.   In the calculating step, the visual saliency is calculated based on a difference absolute value between corresponding histogram bins of the histogram of the feature amount of the first extraction region and the histogram of the feature amount of the second extraction region. The attention area detection method according to claim 1, wherein the attention area is detected.   In the calculating step, the visual saliency is calculated based on a ratio of the probability density with respect to the feature amount obtained in the first extraction region and the probability density with respect to the feature amount obtained in the second extraction region. The attention area detection method according to claim 1, wherein the attention area is detected.   6. The attention area detection method according to claim 1, wherein in the detection step, an attention area is detected based on a statistical distribution of the visual saliency. The detection step includes
A local attention area setting step of setting a plurality of local attention areas based on the visual saliency,
Integrating the plurality of local attention areas based on the size of their distance to generate a group of local attention areas;
A specific step of calculating a total value of visual saliency for each local attention area group, and specifying a local attention area group that gives the maximum value;
The attention area detection method according to claim 6, further comprising: an attention area setting step of setting an area including the specified local attention area group as the attention area. Extraction region setting means for setting a first extraction region and a second extraction region different from the first extraction region for each of a plurality of points in the input image;
Feature quantity extraction means for extracting a feature quantity from each of the first extraction area and the second extraction area;
Calculating means for calculating visual saliency at the plurality of points based on a probability density relationship between the first extraction region and the feature amount of the second extraction region set for the same point; An attention area detection apparatus comprising: a detection means for detecting an attention area based on visual saliency at the plurality of points.   9. The attention area detection apparatus according to claim 8, wherein the feature quantity extraction unit extracts a plurality of types of feature quantities from each of the first extraction area and the second extraction area.   The attention area detection apparatus according to claim 9, wherein the plurality of types of feature amounts include any one of a luminance value, an edge strength, and a texture.   The calculation means calculates the visual saliency based on a difference absolute value between histogram histograms corresponding to the histogram of the feature amount of the first extraction region and the histogram of the feature amount of the second extraction region. The attention area detection device according to any one of claims 8 to 10, wherein   The calculation means calculates the visual saliency based on a ratio of a probability density with respect to a feature amount obtained in the first extraction region and a probability density with respect to the feature amount obtained in the second extraction region. The attention area detection device according to any one of claims 8 to 10, wherein   The attention area detection apparatus according to claim 8, wherein the detection unit detects an attention area based on a statistical distribution of the visual saliency. The detection means includes
Local attention area setting means for setting a plurality of local attention areas based on the visual saliency,
An integration means for generating a group of local attention areas by integrating the plurality of local attention areas based on the size of their distance;
Calculating a total value of visual saliency for each local attention area group, and specifying means for specifying a local attention area group that gives the maximum value;
The attention area detection device according to claim 13, further comprising attention area setting means for setting an area including the identified local attention area group as the attention area.   The program for making a computer perform each process of the attention area detection method of any one of Claim 1 to 7.

Images