CN104809719A - Virtual view synthesis method based on homographic matrix partition - Google Patents

Abstract

The invention discloses a virtual view synthesis method based on homographic matrix partition. The virtual view synthesis method based on homographic matrix partition comprises the following steps of 1) calibrating left and right neighboring view cameras to obtain the internal reference matrixes of the left and right neighboring view cameras and a basis matrix between the left and right neighboring view cameras, deriving an essential matrix from the basis matrix, performing singular value decomposition on the essential matrix, and computing the motion parameters including a rotation matrix and a translation matrix between the left and right neighboring view cameras; 2) performing interpolation division on the rotation matrix and the translation matrix to obtain sub homographic matrixes from left and right neighboring views to a middle virtual view; 3) applying the forward mapping technology to map two view images to a middle virtual view image respectively through the sub homographic matrixes, taking the mapping graph of one of the images as a reference coordinate system and performing interpolation fusion on the mapped two images to synthesize a middle virtual view image. The virtual view synthesis method based on the homographic matrix partition has the advantages of being high in synthesis speed, simple and effective in process and high in practical engineering value.

Description

Based on the method for the virtual view synthesis of homography matrix segmentation

Technical field

The present invention relates to the method for a kind of virtual view synthesis, especially a kind of method of the virtual view synthesis based on homography matrix segmentation, belongs to technical field of image processing.

Background technology

Virtual view synthesis refers to and utilizes computing machine, synthesizes original non-existent visual point image.Virtual view interpolation is the given camera array put in advance according to certain rule, takes multiple images simultaneously, adopts corresponding viewpoint interpolation technique to calculate middle image sequence often between adjacent two visual angles.

In recent years, along with subject height intersection such as computer vision, image procossing, computer graphicses.Virtual view synthesis is widely used in the project such as three-dimensional reconstruction and panoramic mosaic.Viewpoint interpolation technique is widely used in virtual reality, special efficacy photography, three-dimensional television and animation field.Successful story comprises the Quick Time VR virtual reality system being applied in " bullet time " in the Olympic Games and Apple.

Virtual view interpolation technique develops more than 20 year, the method based on image interpolation that classical method is taught by the interpolation method based on offset vector and the Seitz of Apple researcher chen.Within 2004, Microsoft researcher Zitnick proposes a kind of video View Synthesis system schema of new Image Based Rendering technology, program off-line obtains sync pulse jamming video flowing, then obtain the structural parameters of camera array by the stereo algorithm of the segmentation based on color and set up high-quality image corresponding relation, Video Composition procedural depth locus of discontinuity automatically extracts shade and is used for reducing human error, finally realizes the synthesis of video alternately.The advantage of the program be obtain Video Composition quality but its speed be difficult to ensure real-time.2008, Farin etc. it is also proposed the virtual view interpolation technique of base length by any arrangement array retrained.The people such as Guillemaut it is also proposed the optimization viewpoint interpolation method based on Iamge Segmentation, and the compound movement scene that the moving camera for multiple different resolution is taken carries out Iamge Segmentation reconstruction.

At present, the existing algorithm relative complex based on virtual view interpolation synthesis virtual image, real-time performance are poor, stability can not be guaranteed, and some algorithm needs special hardware support, and Project Realization is difficult, and engineering practical value is not high.

Summary of the invention

The object of the invention is the defect in order to solve above-mentioned prior art, provide a kind of method of the virtual view synthesis based on homography matrix segmentation, the method has the advantage that aggregate velocity is fast, process is simple effectively, engineering practical value is high.

Object of the present invention can reach by taking following technical scheme:

Based on the method for the virtual view synthesis of homography matrix segmentation, said method comprising the steps of:

1) adopt external standard size gridiron pattern standardization to demarcate left and right adjacent view camera, obtain the internal reference matrix K of left and right adjacent view camera _l, K _rand the basis matrix F between the adjacent view camera of left and right, derive essential matrix E by basis matrix F, SVD svd is carried out to essential matrix, calculate the kinematic parameter between the adjacent view camera of left and right: rotation matrix R and translation matrix T;

2) interpolation segmentation is carried out to rotation matrix R and translation matrix T, obtain left and right adjacent view to the sub-homography matrix H mapping to a certain position between adjacent view _liand H _ri;

3) by sub-homography matrix H _liand H _rirespectively visual angle, left and right figure is carried out image mapped, the image after a selected wherein visual angle maps, as the reference coordinate system of composograph, uses bilinear interpolation to carry out interpolation fusion, each position scene image in resultant motion process.

As a kind of preferred version, step 1) described employing external standard size gridiron pattern standardization demarcates left and right adjacent view camera, obtains the internal reference matrix K of left and right adjacent view camera _l, K _rand the basis matrix F between the adjacent view camera of left and right, specific as follows:

1.1) gridiron pattern aimed at by camera, shooting at least three group gridiron pattern photos;

1.2) utilize opencv calibration function or matlab calibration tool case, by many groups gridiron pattern photo of shooting, calibrate the internal reference matrix K of left and right adjacent view camera _l, K _rand the basis matrix F of left and right adjacent view camera.

As a kind of preferred version, step 1) describedly derive essential matrix E by basis matrix F, SVD svd is carried out to essential matrix, calculates the kinematic parameter between the adjacent view camera of left and right: rotation matrix R and translation matrix T, specific as follows:

1.3) essential matrix E is derived by basis matrix F:

$E=K_r^T{\mathrm{FK}}_l---\left(1\right)$

Wherein, F is the basis matrix between the adjacent view camera of left and right, K _lfor left camera internal reference, K _rright camera internal reference;

1.4) translation matrix T=[t is established ₁, t ₂, t ₃], then the antisymmetric matrix of translation matrix T is:

${\left[t\right]}_x=\left[\begin{array}{ccc}0& -t_3& t_2\\ t_3& 0& -t_1\\ -t_2& t_1& 0\end{array}\right]---\left(2\right)$

1.5) essential matrix E is represented by rotation matrix R and translation matrix T, as follows:

E＝[T] _xR (3)

1.6) by carrying out SVD svd to essential matrix, rotation matrix R and translation matrix T is obtained.

As a kind of preferred version, step 2) described interpolation segmentation is carried out to rotation matrix R and translation matrix T, obtain left and right adjacent view to the sub-homography matrix H mapping to a certain position between adjacent view _liand H _ri, specific as follows:

2.1) if given calibration system, projection matrix P=K [I|0], P '=K ' [I|0], then corresponding homography matrix H is as follows:

H＝K′(R-tn ^T/d)K ^-1(4)

Wherein, K and K ^-1represent first, second visual angle camera internal reference respectively, n ^tfor the unit normal vector of reference planes, d is the degree of depth of External Reference plane, and outerplanar normal vector is v=n ^t/ d;

2.2) by 3 couples of corresponding point (X of H matrix and outerplanar not conllinear _i, X ' _i), derive outerplanar normal vector v

Obtained by homography matrix H:

X′ _i＝HX _i(5)

Obtained by formula (4):

X′ _i＝[K′(R-tn ^T/d)K ^-1]X _i(6)

Vector X ' _i[K ' (R-tn ^t/ d) K ^-1] X _iconllinear, obtains thus:

X′ _i×[K′(R-tv)K ^-1]X _i＝0 (7)

By 3 pairs of corresponding point of outerplanar not conllinear, simultaneous formula (7), obtains outerplanar normal vector v;

2.3) homography matrix that two visual angles are mapped to a certain position in camera motion process is built, be called sub-homography matrix, being constructed by of sub-homography matrix obtains carrying out interpolation to rotation matrix R and translation matrix T, definition visual angle 1 is the starting point of camera motion, visual angle 2 is the terminal of camera motion, so camera is after piece image is taken at visual angle 1, move to visual angle 2 shooting second width image, the total amount of exercise of terminal is moved to (R (θ), t) represents from starting point;

2.4) suppose to want the virtual image that in resultant motion process, some viewpoint i go out, so from the movement angle of camera, calculate the amount of exercise (R (θ that visual angle 1 goes out to viewpoint i _i), t _i) and the amount of exercise (R (θ-θ that goes out to viewpoint i of visual angle 2 _i), t-t _i), obtain visual angle 1 to viewpoint i virgin homography matrix H according to this kinematic parameter _liwith visual angle 2 to viewpoint i virgin homography matrix H _ri, according to formula (4), obtain sub-homography matrix as follows:

$H_{\mathrm{li}}=K^{\′}(R\left({\mathrm{\θ}}_i\right)-t_i{v}_1^T)K^{-1}---\left(8\right)$

$H_{\mathrm{ri}}=K(R(\mathrm{\θ}-{\mathrm{\θ}}_i)-(t-t_i)v_2^T)K^{\′-1}---\left(9\right)$

Wherein, H _lifor visual angle 1 is to viewpoint i virgin homography matrix, H _rifor visual angle 2 is to viewpoint i virgin homography matrix, K and K ^-1represent first, second visual angle camera internal reference respectively, for visual angle 1 reference planes normal vector, for visual angle 2 reference planes normal vector.

As a kind of preferred version, step 3) each position scene image in described resultant motion process, specific as follows:

3.1) sub-homography matrix H is used _liand H _ri, calculate visual angle 1 and be mapped to the subimage 2 that the subimage 1 at viewpoint i place and visual angle 2 are mapped to viewpoint i place;

3.2) this subimage 1 and subimage 2 are carried out additive fusion, obtain the virtual visual point image at viewpoint i place.

The present invention has following beneficial effect relative to prior art:

1, the inventive method can carry out the real-time performance of elevator system by the basis matrix of the internal reference matrix of off-line calibration adjacent view camera and adjacent view camera, and speed of rebuilding is fast, and robust performance is high.

2, the inventive method is relatively effectively simple, does not need special hardware supported, has very high engineering practical value.

Accompanying drawing explanation

Fig. 1 is the method flow diagram of the virtual view synthesis based on homography matrix segmentation of the embodiment of the present invention 1.

Fig. 2 is the homograph schematic diagram of the adjacent two width images of the embodiment of the present invention 1;

Fig. 3 is that the outerplanar normal vector of the embodiment of the present invention 1 solves schematic diagram;

Fig. 4 is view transformation and the motion segmentation schematic diagram of the embodiment of the present invention 1;

Fig. 5 is the bilinear interpolation principle schematic of the embodiment of the present invention 1.

Embodiment

Below in conjunction with embodiment and accompanying drawing, the present invention is described in further detail, but embodiments of the present invention are not limited thereto.

Embodiment 1:

As shown in Figure 1, the method for the virtual view synthesis based on homography matrix segmentation of the present embodiment, comprises the following steps:

1) adopt external standard size gridiron pattern standardization to demarcate left and right adjacent view camera, obtain the internal reference matrix K of left and right adjacent view camera _l, K _rand the basis matrix F between the adjacent view camera of left and right, essential matrix E is derived by basis matrix F, SVD (Sigular Value Decomposition) svd is carried out to essential matrix, calculate the kinematic parameter between the adjacent view camera of left and right: rotation matrix R and translation matrix T, specific as follows:

1.1) gridiron pattern aimed at by camera, shooting at least three group gridiron pattern photos;

1.2) utilize opencv calibration function or matlab calibration tool case, by many groups gridiron pattern photo of shooting, calibrate the internal reference matrix K of left and right adjacent view camera _l, K _rand the basis matrix F of left and right adjacent view camera.

1.3) essential matrix E is derived by basis matrix F:

$E=K_r^T{\mathrm{FK}}_l---\left(1\right)$

Wherein, F is the basis matrix between the adjacent view camera of left and right, K _lfor left camera internal reference, K _rright camera internal reference;

1.4) translation matrix T=[t is established ₁, t ₂, t ₃], then the antisymmetric matrix of translation matrix T is:

${\left[t\right]}_x=\left[\begin{array}{ccc}0& -t_3& t_2\\ t_3& 0& -t_1\\ -t_2& t_1& 0\end{array}\right]---\left(2\right)$

1.5) essential matrix E is represented (essential matrix equals antisymmetric matrix and is multiplied by rotation matrix) by rotation matrix R and translation matrix T, as follows:

E＝[T] _xR (3)

1.6) by carrying out SVD svd to essential matrix, rotation matrix R and translation matrix T is obtained.

2) interpolation segmentation is carried out to rotation matrix R and translation matrix T, obtain left and right adjacent view to the sub-homography matrix H mapping to a certain position between adjacent view (i.e. intermediate virtual viewpoint) _liand H _ri, specific as follows:

2.1) the homograph schematic diagram of adjacent two width images as shown in Figure 2, if given calibration system, projection matrix P=K [I|0], P '=K ' [I|0], then corresponding homography matrix H is as follows:

m′＝Hm

H＝K′(R-tn ^T/d)K ^-1(4)

Wherein, K and K ^-1represent first, second visual angle camera internal reference respectively, n ^tfor the unit normal vector of reference planes, d is the degree of depth of External Reference plane, and outerplanar normal vector is v=n ^t/ d, m are the points in figure in left reference planes, and m ' is the point in figure in right reference planes;

2.2) outerplanar normal vector as shown in Figure 3 solves schematic diagram, by 3 X of H matrix and outerplanar not conllinear ₁, X ₂, X ₃, derive outerplanar normal vector v

Obtained by homography matrix H:

X ₁₂＝HX ₁₁(5)

Obtained by formula (4):

X ₁₂＝[K′(R-tn ^T/d)K ^-1]X ₁₁(6)

Vector X ₁₂and X ₁₂=[K ' (R-tn ^t/ d) K ^-1] X ₁₁conllinear, obtains thus:

X ₁₂×[K′(R-tv)K ^-1]X ₁₁＝0 (7)

In like manner, X ₂and X ₃can obtain a prescription journey equally, simultaneous three equations can solve outerplanar normal vector v, therefore by 3 pairs of corresponding point of outerplanar not conllinear, namely can simultaneous equations, and obtain outerplanar normal vector v;

2.3) view transformation as shown in Figure 4 and motion segmentation schematic diagram, build the homography matrix that two visual angles are mapped to a certain position in camera motion process, be called sub-homography matrix, being constructed by of sub-homography matrix obtains carrying out interpolation to rotation matrix R and translation matrix T, definition visual angle 1 is the starting point of camera motion, visual angle 2 is the terminal of camera motion, so camera is after piece image is taken at visual angle 1, move to visual angle 2 shooting second width image, the total amount of exercise of terminal is moved to (R (θ), t) represents from starting point;

2.4) suppose to want the virtual image that in resultant motion process, some viewpoint i go out, so from the movement angle of camera, calculate the amount of exercise (R (θ that visual angle 1 goes out to viewpoint i _i), t _i) and the amount of exercise (R (θ-θ that goes out to viewpoint i of visual angle 2 _i), i-t _i), obtain visual angle 1 to viewpoint i virgin homography matrix H according to this kinematic parameter _liwith visual angle 2 to viewpoint i virgin homography matrix H _ri, according to formula (4), obtain sub-homography matrix as follows:

$H_{\mathrm{li}}=K^{\′}(R\left({\mathrm{\θ}}_i\right)-t_i{v}_1^T)K^{-1}---\left(8\right)$

$H_{\mathrm{ri}}=K(R(\mathrm{\θ}-{\mathrm{\θ}}_i)-(t-t_i)v_2^T)K^{\′-1}---\left(9\right)$

Wherein, H _lifor visual angle 1 is to viewpoint i virgin homography matrix, H _rifor visual angle 2 is to viewpoint i virgin homography matrix, K and K ^-1represent first, second visual angle camera internal reference respectively, for visual angle 1 reference planes normal vector, for visual angle 2 reference planes normal vector.

3) by sub-homography matrix H _liand H _rirespectively visual angle, left and right figure is carried out image mapped (application forward mapping technology), image after a selected wherein visual angle maps is as the reference coordinate system of composograph, bilinear interpolation is used to carry out interpolation fusion (namely merging two width images after mapping), each position scene image (i.e. intermediate virtual visual point image) in resultant motion process, specific as follows:

3.1) according to adjacent view to the sub-homography matrix H of virtual view _liand H _ri, adopt H _liand H _rirespectively visual angle, left and right figure is mapped to intermediate virtual viewpoint;

3.2) bilinear interpolation principle schematic as shown in Figure 5, supposes the value that will obtain P=(X, Y), supposes known Q ₁₁, Q ₁₂, Q ₂₁, Q ₂₂value, first obtain R in X-direction interpolation ₁and R ₂, then interpolation obtains P point coordinate in the Y direction;

3.3) be floating number due to what be multiplied by from visual angle 1 or visual angle 2 that matrix obtains, therefore adopt reversed interpolation technology; To the location of pixels of target image, first find out the floating-point position of original image, then utilize the shaping value interpolation of ambient sources pixel to make new advances value, therefore can obtain the virtual image that visual angle 1 and visual angle 2 are interpolated into middle visual angle;

3.4) image after a selected wherein multi-view image mapping is as the reference coordinate system of composograph, be interpolated into middle virtual image to visual angle 1 and visual angle 2 to merge, half respectively got to the pixel value of public domain and carries out additive fusion and obtain intermediate virtual multi-view image.

In sum, the inventive method can carry out the real-time performance of elevator system by the basis matrix of the internal reference matrix of off-line calibration adjacent view camera and adjacent view camera, has the advantage that aggregate velocity is fast, process is simple effectively, engineering practical value is high.

The above; be only patent preferred embodiment of the present invention; but the protection domain of patent of the present invention is not limited thereto; as gridiron pattern can replace with other independent signable object; anyly be familiar with those skilled in the art in the scope disclosed in patent of the present invention; be equal to according to the technical scheme of patent of the present invention and inventive concept thereof and replace or change, all belonged to the protection domain of patent of the present invention.

Claims (5)

1., based on the method for the virtual view synthesis of homography matrix segmentation, it is characterized in that: said method comprising the steps of:

1) adopt external standard size gridiron pattern standardization to demarcate left and right adjacent view camera, obtain the internal reference matrix K of left and right adjacent view camera _l, K _rand the basis matrix F between the adjacent view camera of left and right, derive essential matrix E by basis matrix F, SVD svd is carried out to essential matrix, calculate the kinematic parameter between the adjacent view camera of left and right: rotation matrix R and translation matrix T;

2) interpolation segmentation is carried out to rotation matrix R and translation matrix T, obtain left and right adjacent view to the sub-homography matrix H mapping to a certain position between adjacent view _liand H _ri;

3) by sub-homography matrix H _liand H _rirespectively visual angle, left and right figure is carried out image mapped, the image after a selected wherein visual angle maps, as the reference coordinate system of composograph, uses bilinear interpolation to carry out interpolation fusion, each position scene image in resultant motion process.

2. the method for the virtual view synthesis based on homography matrix segmentation according to claim 1, it is characterized in that: step 1) described employing external standard size gridiron pattern standardization demarcates left and right adjacent view camera, obtains the internal reference matrix K of left and right adjacent view camera _l, K _rand the basis matrix F between the adjacent view camera of left and right, specific as follows:

1.1) gridiron pattern aimed at by camera, shooting at least three group gridiron pattern photos;

1.2) utilize opencv calibration function or matlab calibration tool case, by many groups gridiron pattern photo of shooting, calibrate the internal reference matrix K of left and right adjacent view camera _l, K _rand the basis matrix F of left and right adjacent view camera.

3. the method for the virtual view synthesis based on homography matrix segmentation according to claim 1, it is characterized in that: step 1) described by basis matrix F derivation essential matrix E, SVD svd is carried out to essential matrix, calculate the kinematic parameter between the adjacent view camera of left and right: rotation matrix R and translation matrix T, specific as follows:

1.3) essential matrix E is derived by basis matrix F:

$E=K_r^T{\mathrm{FK}}_l---\left(1\right)$

Wherein, F is the basis matrix between the adjacent view camera of left and right, K _lfor left camera internal reference, K _rright camera internal reference;

1.4) translation matrix T=[t is established ₁, t ₂, t ₃], then the antisymmetric matrix of translation matrix T is:

${\left[t\right]}_x=\left[\begin{array}{ccc}0& -t_3& t_2\\ t_3& 0& -t_1\\ -t_2& t_1& 0\end{array}\right]---\left(2\right)$

1.5) essential matrix E is represented by rotation matrix R and translation matrix T, as follows:

E＝[T] _xR (3)

1.6) by carrying out SVD svd to essential matrix, rotation matrix R and translation matrix T is obtained.

4. the method for the virtual view synthesis based on homography matrix segmentation according to claim 1, it is characterized in that: step 2) described interpolation segmentation is carried out to rotation matrix R and translation matrix T, obtain left and right adjacent view to the sub-homography matrix H mapping to a certain position between adjacent view _liand H _ri, specific as follows:

2.1) if given calibration system, projection matrix P=K [I|0], P '=K ' [I|0], then corresponding homography matrix H is as follows:

H＝K′(R-tn ^T/d)K ^-1(4)

Wherein, K and K ^-1represent first, second visual angle camera internal reference respectively, n ^tfor the unit normal vector of reference planes, d is the degree of depth of External Reference plane, and outerplanar normal vector is v=n ^t/ d;

2.2) by 3 couples of corresponding point (X of H matrix and outerplanar not conllinear _i, X ' _i), derive outerplanar normal vector v

Obtained by homography matrix H:

X′ _i＝HX _i(5)

Obtained by formula (4):

X′ _i＝[K′(R-tn ^T/d)K ^-1]X _i(6)

Vector X ' _i[K ' (R-tn ^t/ d) K ^-1] X _iconllinear, obtains thus:

X′ _i×[K′(R-tv)K ^-1]X _i＝0 (7)

By 3 pairs of corresponding point of outerplanar not conllinear, simultaneous formula (7), obtains outerplanar normal vector v;

2.3) homography matrix that two visual angles are mapped to a certain position in camera motion process is built, be called sub-homography matrix, being constructed by of sub-homography matrix obtains carrying out interpolation to rotation matrix R and translation matrix T, definition visual angle 1 is the starting point of camera motion, visual angle 2 is the terminal of camera motion, so camera is after piece image is taken at visual angle 1, move to visual angle 2 shooting second width image, the total amount of exercise of terminal is moved to (R (θ), t) represents from starting point;

2.4) suppose to want the virtual image that in resultant motion process, some viewpoint i go out, so from the movement angle of camera, calculate the amount of exercise (R (θ that visual angle 1 goes out to viewpoint i _i), t _i) and the amount of exercise (R (θ-θ that goes out to viewpoint i of visual angle 2 _i), t-t _i), obtain visual angle 1 to viewpoint i virgin homography matrix H according to this kinematic parameter _liwith visual angle 2 to viewpoint i virgin homography matrix H _ri, according to formula (4), obtain sub-homography matrix as follows:

$H_{\mathrm{li}}=K^{\′}(R\left({\mathrm{\θ}}_i\right)-t_i{v}_1^T)K^{-1}---\left(8\right)$

$H_{\mathrm{ri}}=K(R(\mathrm{\θ}-{\mathrm{\θ}}_i)-(t-t_i)v_2^T)K^{\′-1}---\left(9\right)$

Wherein, H _lifor visual angle 1 is to viewpoint i virgin homography matrix, H _rifor visual angle 2 is to viewpoint i virgin homography matrix, K and K ^-1represent first, second visual angle camera internal reference respectively, for visual angle 1 reference planes normal vector, for visual angle 2 reference planes normal vector.

5. the method for the virtual view synthesis based on homography matrix segmentation according to claim 1, is characterized in that: step 3) each position scene image in described resultant motion process, specific as follows:

3.1) sub-homography matrix H is used _liand H _ri, calculate visual angle 1 and be mapped to the subimage 2 that the subimage 1 at viewpoint i place and visual angle 2 are mapped to viewpoint i place;

3.2) this subimage 1 and subimage 2 are carried out additive fusion, obtain the virtual visual point image at viewpoint i place.