JPH08163565A - Motion vector detection method and device therefor - Google Patents

Abstract

PURPOSE: To highly accurately detect a motion vector from picture data. CONSTITUTION: The pictures of a present frame are divided into plural small areas in a picture division part 4, the pictures of one frame before from a memory 2 are divided into the plural small areas by a memory control part 3 respectively and the corresponding small areas are made to correspond to each other in a motion vector detection part 5. A judgement part 6 outputs a correspondence result as the motion vector when it is reliable, and when the correspondence result is not reliable, an edge is detected in an edge detection part 7 and further, the contour line of an object is detected in a contour line detection part 8 from the edge. A weighting part 9 weights both side parts of the contour line and weighted data are made to correspond to the data of the memory 2 in the motion vector detection part 5 again. Thus, by largely weighting a part to be considered as important within the small area, the movement of many parts is neglected and the detection accuracy of the motion vector is improved.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion vector detecting method and apparatus used for encoding moving images and stereo images, image blur correction, moving object recognition, three-dimensional shape measurement of an object, and the like.

[0002]

2. Description of the Related Art As a conventional motion vector detecting method,
U.S. Pat. No. 3,890,462 and Japanese Patent Publication No. 60-46
There are a spatiotemporal gradient method (gradient method), a Fourier transform method for performing the spatiotemporal gradient method in frequency space, a correlation method based on a correlation operation, and a block matching method (template matching method) described in Japanese Patent No. 878. For the spatiotemporal gradient method, see BKPHorn et al., Artificial Int.
elligence 17, p. 185-203 (1981). Regarding the block matching method, Morio Onoue et al. 17, No. 7, p. 63
4-640, July 1976.

In these methods, two images obtained at frame intervals or field intervals are each divided into a plurality of small areas (blocks), and the corresponding areas of the two images are searched in small area units. Detect a motion vector. Therefore, when a plurality of movements are mixed in the small area, correct association cannot be performed in principle.

As a countermeasure against this, for example, M. Bierling: "Hi
erarchical Estimation of Displacement Vector Field
s ", PCS '87, or subsampling the input image, or" Motion Detection Method in Moving Images Using Hierarchical Pixel Information "by Tominaga et al. (IEICE Transactions D-II Vol. J72-D-II, N
o. 3, p. 395 to 403, March 1989), by averaging and compressing images, etc. to convert a full size original image to a compressed image into a hierarchical image data group, A hierarchical method of increasing the resolution of motion vector detection is often adopted.

[0005] More directly, for example, "Motion Detection by Hierarchical Pixel Information-Method of Improving Spatial Resolution of Motion-" by Tominaga et al. (Journal of the Institute of Electronics, Information and Communication Engineers D-II).
Vol. J72-D-II, No. 10, p. 1750
~ 1752, October 1989), there is also a method of gradually increasing the resolution of motion vector detection by gradually changing the block size itself from a large block size to a small block size. Further, as described in JP-A-6-89343, the block is divided into small child blocks,
Perform matching calculation using the child block in advance,
There is also proposed a method of setting a child block having no similar pattern having a correlation value of a certain value or more as a dead zone and searching for a corresponding point in the parent block again. The block is synonymous with the small area of the image.

[0006]

However, according to the above-mentioned conventional hierarchical method, it is impossible to detect corresponding points with high accuracy due to the influence of aliasing distortion that occurs when subsampling an image, and the image is reduced by averaging. There are problems that it takes time and labor to repeat the processing, and that a large-capacity image memory for storing a plurality of image groups having different sizes is required.

Further, in the above-mentioned conventional example in which the block size is changed, a dedicated image memory address control circuit for changing the block size is required, and in the method of obtaining the dead zone, a plurality of blocks are processed each time one block is processed. Since the matching operation by the child block is executed in advance, there is a problem that the calculation amount becomes huge.

Further, as a common problem, since the effect of occlusion occurring in the boundary area between the object and the background is not taken into consideration, there is a problem in that the accuracy of the matching is deteriorated in the process of increasing the resolution in the occlusion section. There is.

The present invention has been made in order to solve the above problems, and provides a first invention (claims 1, 2, 3, 4).
5) is aimed at the movement of images between time series images,
It is intended to improve the motion vector detection accuracy of a small area when there are a plurality of movements in the small area of interest.

A second aspect of the present invention (claims 2, 3, and 6) is directed to the movement of images between multi-viewpoint images, and when there is an object with a different subject distance in a small region of interest, the small region is detected. The purpose is to improve the accuracy of motion vector detection in a region.

The third aspect of the present invention (claims 2, 3, and 7) is directed to both the movement of images between time-series images and the movement of images between multi-viewpoint images, and The purpose is to improve the accuracy of motion vector detection.

According to a fourth aspect of the present invention (claims 2, 3, and 8), when there are a plurality of movements in a small region of interest, it is particularly desirable to improve the accuracy of motion vector detection at the center point of the small region. Has an aim.

A fifth aspect of the present invention (claims 2, 3, and 9) is that, when there are a plurality of movements in a small region of interest, the motion vector detection accuracy of an object in front of the small region is particularly high. The purpose is to increase.

The second, third, and tenth aspects are intended to improve the motion vector detection accuracy of an object having a larger area, especially when there are a plurality of movements in a small area of interest. There is.

[0015]

In order to achieve each of the above objects, the first invention (claims 1, 2, 3, 4, 5) is based on a contour line from a group of edges in a small region of interest. It is characterized by having a contour line detecting step or means for detecting.

The second invention (claims 2, 3, and 6) has a contour line detecting means for detecting a contour line from an edge group in a small region of interest, as in the first invention. Is characterized by.

The third invention (claims 2, 3, and 7) is, similarly to the first and second inventions, a contour line detecting means for detecting a contour line from an edge group in a small region of interest. It is characterized by having.

A fourth aspect of the present invention (claims 2, 3, and 8) is characterized by having weighting means for performing weighting in a small region of interest.

The fifth aspect of the present invention (claims 2, 3, and 9) is characterized in that it has a weighting means for weighting the small region of interest, as in the case of the fourth aspect.

The sixth invention (claims 2, 3 and 10) is characterized in that, like the above-mentioned fourth invention, it has a weighting means for weighting the small region of interest.

[0021]

In the first aspect of the invention, the contour line detecting means uses the time change amount of the image densities on both sides of the edge of the small region of interest to detect an object and a background or another object. It operates so as to detect the contour line that is the boundary of.

In the second aspect of the invention, the contour line detecting means uses the change amount of the image density on both sides of the edge when the images are input from a plurality of viewpoints, and the object and the background at different object distances, or It operates to detect a contour line that is a boundary with another object.

In the third aspect of the present invention, the contour line detecting means detects the contour line of the object by using the continuity of the motion vector detected from the small area of interest and the surrounding small areas. To work.

In the fourth invention, the weighting means sets the image area in the small area including the center point of the small area to be heavy, and the image area in the small area not including the center point to be light. Operate.

In the fifth invention, the weighting is
In the small area, the image area in the small area corresponding to an object at a closer distance is set to be heavier, and the image area in the small area corresponding to an object at a longer distance is set to be lighter.

In the sixth aspect of the present invention, the weighting means weights the image area in the small area corresponding to an object having a larger area in the small area and weights the image area in the small area corresponding to an object having a smaller area. It works like setting the image area of.

[0027]

The first to third embodiments of the present invention will be described below with reference to the drawings. [First Embodiment] FIG. 1 shows an embodiment of a motion vector detecting device according to the present invention. The configuration of FIG. 1 has a second and a third configuration.
Is also used in the embodiment of. In FIG. 1, reference numeral 1 is an image input unit for inputting image data input from an external device such as a camera or playback device, 2 is a memory such as a frame memory for storing input image data, and 3 is an image stored in the memory 2. Is divided into small areas, and a memory control unit 4 that controls reading of image data in units of small areas is an image dividing unit that divides the input image data of the current frame into the same small areas as the small areas of the memory 2. .

Reference numeral 5 is a motion vector detecting section for detecting a motion vector by associating two image data output from the memory 2 and the image dividing section 4 or by the output of a weighting section 9 which will be described later. The determination unit that determines the reliability of the detected motion vector, 7 is the edge detection unit that detects the edge of the image in the data of the small area that is determined to have low reliability, and 8 is the contour line of the object from the detected edge. The contour line detection unit 9 detects the contours, and the weighting unit 9 weights regions on both sides of the detected contour line and sends the weighted regions to the motion vector detection unit 5. Reference numeral 10 is a control unit such as a CPU that controls each unit.

The memory 2, the memory control unit 3, and the image division unit 4 constitute an image division unit, and the motion vector detection unit 5 and the judgment unit 6 constitute a motion vector detection unit.
Further, the edge detection unit 7 and the contour line detection unit 8 constitute a contour line detection means.

FIG. 2 shows a processing flow of the motion vector detecting method according to the embodiment of the present invention. Hereinafter, the processing contents will be described in the order of steps with reference to FIGS. (Step S1) The image input unit 1 discretizes image data obtained from a photoelectric conversion element represented by a CCD and stores it in the memory 2. (Step S2) The memory control unit 3 divides the image data stored in the memory 2 into a plurality of small areas, moves from the plurality of small areas to a target small area, and Read the image data in. At the same time, the image division unit 4 uses one frame (or one field) of the image data.
The image data input later is divided into a plurality of small areas corresponding to the above small areas.

(Step S3) The motion vector detecting section 5 performs a calculation operation for associating two images of the image stored in the memory 2 and the latest image data after the above one frame in units of small areas. Temporarily detect a vector. The calculation method may be the spatiotemporal gradient method or the Fourier transform method described in (Prior art), or the correlation method or the block matching method, and the method is not particularly limited. When passing through this step S3 for the first time, a conventional associative calculation without weighting to an image to be described later is performed.

(Step S4) The judging section 6 judges the reliability of the motion vector obtained as a result of the correspondence between the images. As a criterion, there are various features such as whether a sufficient amount of features is included in a small area, the spatial frequency is not too high, the correlation value is sufficiently high, or there are multiple correlation value peaks. One of them may be appropriately selected or a plurality of them may be combined depending on the purpose, and there is no particular limitation. Here, the result of the above correlation, which is determined to have sufficiently high reliability, is output as it is from step S8 as the correct motion vector of the small region of interest. On the other hand, if the reliability is insufficient, the process proceeds to step S5 and below.

(Step S5) Steps S5 and S6 are combined to form a contour line detecting step by the contour line detecting means. In step S5, the edge detection unit 7 detects an edge from the image data in the small area of interest at present. As an edge detection method, various operators based on first-order differential and second-order differential operations of image data have been proposed so far, and they may be appropriately selected or combined according to the purpose, and are not particularly limited. (Step S6) The contour line detection unit 8 detects a contour line from the edge detected in step 5. The contour line here means each boundary line of a plurality of objects that move differently.

Next, the principle of contour line detection will be described with reference to FIG.
This will be specifically described below. FIGS. 3 (a) and 3 (b) show a state in which an automobile crosses two screens to the left. The small areas 10 and 30 on the two screens are in a correspondence relationship and include the edges 10a and 30a of the door of the automobile. The small areas 20 and 40 also correspond to each other and include the contour lines 20a and 40a of the rear portion of the automobile. Small area 1
The edges 10a and 30a included in 0 and 30 are a part of the texture inside the object (the automobile in this case), and the image density values on both sides of the edge do not change with the passage of time. On the other hand, the edges included in the small areas 20 and 40 are the contours of the object, and new backgrounds appear one after another over time, so the image density values on one side of the edges change greatly. Therefore, in step S6, the time variation amount of the image density on both sides of the edge detected in the previous step S5 is calculated, and if the variation amount exceeds a certain value, it is determined that the edge is the contour line of the object. , If it does not exceed a certain value, it is judged as the texture of the object. As the time variation amount of the image density on both sides of the edge, for example, any one of the following equations (1) to (5) may be calculated.

| ∂ (g ₁ −g _r ) / ∂t |> (set value 1) ……… (1) | ∂ (g ₁ / g _r ) / ∂t |> (set value 2) ……… (2) │ (∂g ₁ / ∂t)-(∂g _r / ∂t) ｜> (set value 3) ………… (3) ｜ ∂g ₁ / ∂t |> (set value 4) …… … (4) | ∂g _r / ∂t |> (Set value 5) ……… (5)

Here, g ₁ and g _r respectively represent image density values on both sides of the edge. Equation (1) is the time variation of the image density difference on both sides of the edge, Equation (2) is the variation of the image density ratio on both sides of the edge, and Equation (3) is the difference of the time variation on both sides of the edge. Equations (4) and (5) calculate the time variation amounts on both sides of the edge. Set value 1 for each
When the number exceeds ~ 5, it is determined that the edge is a contour line. If the calculation result does not exceed the set value, it is determined that a highly reliable motion vector cannot be detected from this small area, the process returns to step S2, and the target area is shifted to the next small area.

(Step S7) When the contour line is detected in step S6, the weighting unit 9 weights the image data in the small area of interest. The weighting method is basically that the image area to be emphasized is heavy and the other image areas are light. The image area to be emphasized here is usually the center point in the small area of interest at present. FIG. 4 shows a small area (block) of the image of interest at present, and the X mark area 50 includes the center point 0. Therefore, for example, a mask as shown in FIG. 5 is applied to the small area. In this example, the area 50 is masked with a weighting coefficient of "1" and the area 60 is masked with a weighting coefficient of "0". In addition, for example, as in the case of detecting an obstacle in an autonomous vehicle, there may be a case where a subject region in a short distance is emphasized. For example, in FIG.
Assuming that 0 is a region of an object at a short distance, the mask is shown in FIG.
It becomes as shown in. It should be noted that the vertical relationship of the subject can be inferred from the similarity between the small areas in the surroundings or the division relationship of the textures in the small areas during the iterative calculation. In addition, it is visually natural to place importance on the area that occupies the largest area in the small area of interest, which is advantageous in encoding. In that case, as shown in FIG. 5, the large area may be masked with "1" and the other areas may be masked with "0".

After the weighting is performed as described above, the process returns to step S3, and the motion vector detecting unit 5 executes the association for each weighted small area to detect the motion vector. Note that although binary weighting factors are used in FIGS. 5 and 6, it is of course possible to use multivalued weighting factors. Similar weighting can be performed when there are a plurality of contour lines in a small region of interest and the number of regions divided by the contour line is three or more. (Step S8) The motion vector of the small region of interest is output. The determination unit 6 checks again the reliability of the detected motion vector, outputs the motion vector if there is reliability, and performs the processing of the contour line detection step again if there is no reliability.

As described above, according to this embodiment,
When detecting a motion vector from a time-series image, a contour line is extracted from the image, and weighting is applied to a small region so that the region of interest among multiple regions bounded by the contour line is more emphasized. By doing so, it is possible to reduce the influence of extra motion in a region other than the region of importance included in the small region of interest. In particular, in the normal case where the attention point coincides with the center point of the small area, the motion vector of the attention point is accurately calculated by giving priority to the image area (area 50 in FIG. 4) in the small area including the center point. Can be detected. Further, when the point of interest is in a short-distance object, for example, when the present device is mounted on an autonomous vehicle, a robot, etc., by emphasizing the image area in the small area corresponding to the object in the short distance, The motion vector of the short-range object can be accurately detected. Furthermore, when applying to the encoding of an image in a large area where the point of interest is visually conspicuous, the motion vector of the large area is accurately detected by emphasizing the maximum area in the small area. be able to.

[Second Embodiment] In the first embodiment, the motion vector is detected from the time-series image obtained for each frame or field, but in the second embodiment, it is obtained from multiple viewpoints. A case where a motion vector is detected from a plurality of images will be described. The procedure of the process is the same as the flowchart of FIG. 2, but the contents of each step and the configuration of the device are slightly different in the following points. The image input unit 1 in FIG. 1 is configured to input a plurality of image data,
Along with this, the memory 2 is configured to store a plurality of image data. Further, the image dividing unit 4 is configured to divide each of the plurality of input image data into small areas. Therefore, in steps S1 and S2, processing such as image input and image division is performed for each of the plurality of image data.

In steps S5 and S6, the amount of change in the image density value on both sides of the edge detected from the plurality of image data picked up from each viewpoint is calculated. The content of step 6 will be specifically described with reference to FIG. 7 when two image data are used. FIG. 7A is an image of a subject (mountain and a stationary car) taken from the left viewpoint, and FIG. 7B is an image taken from the right viewpoint. In such a stereo image, the position to be imaged changes depending on the size of the subject distance. This is called parallax, and the closer the object is to the camera, the larger it moves on the imaging surface. In the case of FIG. 7, a region of a stationary automobile at a short distance is moving much more than a mountain at a long distance. By associating the images of the stereo pair as shown in FIG. 7, the distance to the subject can be obtained by using the principle of triangulation. Note that the result of the correspondence between the images is also called a motion vector here.

The small areas 100 and 300 in FIG. 7 are in a correspondence relationship, and include a mountain ridge line in the distant view as a contour line. The small areas 200 and 400 also have a correspondence relationship,
It includes the contours of the front of the car. Small area 10
Since the edges included in 0 and 300 are very far from the distance between the left and right cameras, the parallax between the left and right images is extremely small. On the other hand, the edges included in the small areas 200 and 400 are the contours of objects (vehicles) in a short distance, and the background portion changes relatively greatly as the viewpoint changes, so the image density value on one side of the edges is It changes a lot. Therefore, by calculating the variation amount of the image densities on both sides of the edge between the viewpoints by using the equations (1) to (5) as in the first embodiment, a plurality of movements in the same small area ( == here, parallax), the contour line indicating the boundary can be extracted.

As described above, according to this embodiment,
When detecting a motion vector from an image group obtained from a plurality of viewpoints, a contour line is extracted from the image, and the contour line is used as a boundary.
By weighting the small area so as to give more importance to the area of interest, it is possible to reduce the influence of extra motion included in the small area of interest.

[Third Embodiment] In the first and second embodiments described above, the texture is determined based on the change amount of the image density on both sides of the edge in the small area of the image of interest. Although the contour line is distinguished from the contour line, in the present embodiment, the texture and the contour line are distinguished from each other based on the continuity of the motion vector detected from the neighboring small regions surrounding the small region of interest. That is, when a motion vector different from the motion vector in the neighborhood of the small region of interest is being output, it can be determined that there is a contour line that causes occlusion in the small region of interest. .

Therefore, in the third embodiment, every time the motion vector detecting unit 5 detects a motion vector in each small area in the reliability judgment processing in step S4 of FIG. Calculate the deviation from the motion vector. Next, the judgment unit 6 compares the deviation value with the set value, and if the deviation value is smaller than the set value, the motion vector is output as it is, and if the deviation value exceeds the set value, the process proceeds to step S5 and thereafter. The operations after step S5 are the same as those in the first and second embodiments. It should be noted that, subsequently, after weighting is performed in step S8, it is possible to improve the detection accuracy of the motion vector of the small region of interest at present by performing the correspondence repeatedly in step S3. The present embodiment is suitable for a situation where there are few moving objects or objects with different subject distances in an image, the size of a small area is relatively large, and the correlation of motion vectors between adjacent small areas with high resolution is high. It is applicable to any of the cases of the first and second embodiments.

[0046]

As described above, according to the first aspect of the present invention (claims 1, 2, 3, 4, 5), it is possible to accurately detect a motion vector of a point of interest from a time series image. There is an effect that can be.

According to the second invention (claims 2, 3, and 6), there is an effect that it is possible to accurately detect a motion vector of a spot of interest from among images input from a plurality of viewpoints.

According to the third invention (claims 2, 3, and 7), there is an effect that it is possible to accurately detect the motion vector of the point of interest from both the time-series image and the multi-viewpoint image.

According to the fourth invention (claims 2, 3, and 8), there is an effect that the motion vector at the center of the small region of the image of interest can be accurately detected.

According to the fifth invention (claims 2, 3, and 9), there is an effect that the motion vector of the short-distance object within the small area of the image of interest can be accurately detected.

According to the sixth invention (claims 2, 3, and 10), there is an effect that the motion vector of the large area portion in the small area of the image of interest can be accurately detected.

Further, according to the present invention, there is an effect that the memory capacity and the calculation amount can be reduced.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of the present invention.

FIG. 2 is a flowchart showing the operation of the embodiment.

FIG. 3 is an explanatory diagram of an edge based on a contour line and a texture.

FIG. 4 is an explanatory diagram of a small area of an image.

FIG. 5 is an explanatory diagram of a weighting mask.

FIG. 6 is an explanatory diagram of a weighting mask.

FIG. 7 is an explanatory diagram of an edge with a large parallax and an edge with a small parallax.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 image input unit 2 memory 3 memory control unit 4 image division unit 5 motion vector detection unit 6 determination unit 7 edge detection unit 8 contour line detection unit 9 weighting unit

Claims (10)

[Claims] 1. A motion vector detection method for dividing image data into a plurality of small areas and detecting a motion vector for each small area, wherein a contour line detecting step of detecting a contour line of an object in the small area, It is characterized by further comprising a weighting step of weighting the image data in the small area with the detected contour line as a boundary, and a motion vector detecting step of detecting a motion vector by associating the image data with each other. Motion vector detection method. 2. A motion vector detecting means for detecting a motion vector by performing an associating operation between image dividing means for dividing a plurality of image data into a plurality of small areas and corresponding image data of a plurality of small areas. And a contour line detecting means for detecting a contour line from the image data of the small area in accordance with the result of the above-mentioned correspondence calculation, and weighting the image data on both sides of the detected contour line as a boundary. A motion vector detecting device comprising weighting means for adding weighted image data to the motion vector detecting means and corresponding to image data obtained from the image dividing means. 3. The motion vector detecting means determines the reliability of the result of the associating operation according to a predetermined criterion, outputs a reliable operation result as a motion vector, and outputs an unreliable operation result. The motion vector detection device according to claim 2, which is configured to be added to the contour line detection means. 4. The motion vector detecting device according to claim 2, wherein the contour line detecting means is configured to detect an edge from the image data and calculate a change amount of the image density on both sides of the detected edge. . 5. The motion vector detecting apparatus according to claim 4, wherein the amount of change in image density on both sides of the edge uses a time change amount of image density on both sides of the edge. 6. The motion vector detecting device according to claim 4, wherein the amount of change in image density on both sides of the edge uses the amount of change in image density on both sides of the edge when input is made from a plurality of viewpoints. 7. The motion vector detecting apparatus according to claim 2, wherein the contour line detecting means calculates continuity of motion vectors detected from the small area of interest and the surrounding small areas. 8. The weighting means is for weighting an image area in the small area including the center point of the small area and lightly weighting an image area in the small area not including the center point. The motion vector detection device described. 9. The weighting means weights an image area corresponding to an object at a short distance in the small area and weights an image area corresponding to an object at a long distance. Motion vector detection device. 10. The motion vector detecting apparatus according to claim 2, wherein the weighting means weights an image area corresponding to an object having a large area among the small areas and lightly weights the other image areas.