DE102014211869A1 - Method and device for detecting at least one object in a mobile unit environment - Google Patents

Abstract

The invention relates to a method (200) for detecting at least one object (160) in an environment (165) of a mobile unit (100), in particular of a vehicle and / or a robot. The method (200) comprises a step (210) of reading in a first image (400) containing the object (160) of the environment (165) of the mobile unit (100) that was acquired at a first time, wherein the first image (400 ) is read in by a camera system (120) arranged in or on the mobile unit (100). The method (200) further comprises a step (220) of reading in at least one second image (410) containing the object (160) of the environment (165) of the mobile unit (100) acquired at a second time, the second image (410) is read in by the camera system (120) and wherein the mobile unit (100) rests between the first and second times or does not move more than a tolerance distance from a reference position (155) in the environment (165) of the mobile unit (100) has been. Finally, the method (200) comprises a step (230) of determining the object (160) in the environment (165) of the mobile unit (160) using the first (400) and second (410) images.

Description

State of the art

The present invention relates to a method for detecting at least one object in an environment of a mobile unit, to a corresponding device and to a corresponding computer program.

The start-up of a vehicle (whether autonomous, semi-autonomous or controlled by a driver) can be secured with sensors to avoid colliding with obstacles or driving over obstacles or driving on unsuitable sections (curb, precipice, etc.).

Ultrasonic sensors are most commonly used today to detect nearby obstacles and avoid collisions, e.g. B. when maneuvering the vehicle. Other distance measuring sensors, such. As stereo camera, short-range radar, laser scanner or time-of-flight camera are suitable for such tasks.

Disclosure of the invention

Against this background, with the approach presented here, a method for recognizing at least one object in a mobile unit environment, furthermore a device which uses this method and finally a corresponding computer program according to the main claims are presented. Advantageous embodiments emerge from the respective subclaims and the following description.

• – Einlesen eines das Objekt enthaltenden ersten Bildes der Umgebung der Mobileinheit, das zu einem ersten Zeitpunkt erfasst wurde, wobei das erste Bild von einem in oder an der Mobileinheit angeordneten Kamerasystem eingelesen wird;
• – Einlesen zumindest eines das Objekt enthaltenden zweiten Bildes der Umgebung der Mobileinheit, das zu einem zweiten Zeitpunkt erfasst wurde, wobei das zweite Bild von dem Kamerasystem eingelesen wird und wobei die Mobileinheit zwischen dem ersten und zweiten Zeitpunkt ruht oder um nicht mehr als eine Toleranzstrecke gegenüber einer Referenzposition in der Umgebung der Mobileinheit bewegt wurde; und
• – Ermitteln des zumindest einen Objektes in der Umgebung der Mobileinheit unter Verwendung des ersten und zweiten Bildes.

The approach presented here creates a method for recognizing at least one object in an environment of a mobile unit, in particular a vehicle and / or a robot, the method having the following steps:

• Reading in a first image of the environment of the mobile unit containing the object acquired at a first time, the first image being read in by a camera system arranged in or on the mobile unit;
• Reading at least one second image containing the object of the environment of the mobile unit acquired at a second time, the second image being read in by the camera system and the mobile unit resting or not more than a tolerance distance between the first and second times a reference position has been moved in the vicinity of the mobile unit; and
• Determining the at least one object in the environment of the mobile unit using the first and second images.

In the present case, a mobile unit can be understood to be a unit or an element which is movable in relation to other objects in the environment. For example, such a mobile unit may be a vehicle or a robot component that is movable or can be moved, for example, relative to an object (for example, a foreign vehicle parked in front of the vehicle or a curb or obstacle in a moving path of a robot arm). An environment of the mobile unit may, for example, be understood to mean a closer spatial area around the mobile unit in which there may be objects with which, for example, during a movement of the mobile unit the mobile unit could collide and would be damaged. By a camera system may be understood, for example, an optical unit having an image sensing sensor for providing the first and second images. In addition, it is conceivable that the camera system also has other optical components such as a mirror or a lens system to change a perspective of the image acquisition sensor. In this way, with the camera system, a first and second image of the environment of the mobile unit can be detected, from which subsequent position of the object or the object itself in the environment of the mobile unit is determined. The mobile unit itself will not, or only slightly, between the first and second times, i. within a tolerance range from a reference position in the environment of the mobile unit. By way of example, such a tolerance range or a tolerance range may be less than 10 cm, preferably less than 1 cm, more preferably less than 5 mm, relative to the reference position in the vicinity of the mobile unit. Such a reference position may be, for example, a point on a footprint of the mobile unit (for example, the lane of the vehicle as a mobile unit or a hall floor of a robot).

The approach presented here is based on the recognition that even a small or very slightly moving mobile unit already makes small changes in the perspective of the camera system, but these are sufficient to determine a three-dimensional image of the object in the surroundings of the mobile unit. From this, the position of the object in the environment of the mobile unit can then also be determined, for example. For this determination of the position of the object in the environment of the mobile unit, conventional algorithms can be used, these algorithms are already very efficient and already numerically optimized. Also can be used for the approach presented here on in modern vehicles usually already installed as standard camera systems whose images continue to be used for the evaluation according to the approach presented here can be. In this way, it is advantageously possible to achieve further use of already available image data with only low costs by modifying a correspondingly designed evaluation unit.

An embodiment of the approach presented here is expedient in which, in the step of reading in the second image, an image is read in as a second image which was taken from another position of the camera system or at least one component of the camera system with respect to at least one element of the mobile unit, as the first picture. Under such a different position, under which the second image was taken with respect to the first image, it can be understood, for example, that a change of perspective of the viewing direction of the camera system (with the mobile unit resting) on the environment of the mobile unit. For example, such a change of perspective may have occurred by changing the position of the camera system or at least one component of the camera system with respect to a component of the mobile unit. Such an embodiment of the approach presented here offers the advantage of a particularly accurate possibility for determining the position of the object in the environment of the mobile unit.

Another advantage is an embodiment of the approach presented here, in which the first and second image are read in by a monocular camera system in the steps of reading in the first image and reading in the second image. Such an embodiment of the approach presented here allows the use of images of one of the low-cost camera systems, whereby appropriately designed image processing algorithms, which are already known and technically mature, can be used to easily determine the position of the object in the environment of the mobile unit.

Also conceivable is an embodiment of the approach presented here, in which an image is read in as a second image in the step of reading in the second image, which image is read in at a relative speed between the mobile unit and the reference point of zero hourly kilometers.

Also, according to a further embodiment of the approach presented here, in the step of reading in the second image, the second image is an image which was acquired by an element folded away from the mobile unit at the second time. Such a folded-down element may, for example, be a door or a vehicle element protruding element (for example a mirror) or a robot as a mobile unit robot arm in the case of the vehicle as a mobile unit. Such an embodiment of the approach presented here has the advantage that the movement of the hinged element can significantly change the viewing direction of the camera system between the recording of the first image and the second image, so that the position of the first and second image is very precisely Object can be determined in the environment of the mobile unit.

Also advantageous is an embodiment of the approach presented here, comprising a step of moving at least a part of the camera system with respect to at least one element of the mobile unit before the step of reading in the second image. It is conceivable, for example, to actively change the position of the camera system in order to achieve the greatest possible change in the viewing angle of the camera system between the first image and the second image. In this way it can also be achieved that the most accurate possible determination of the position of the object in the environment of the mobile unit is possible from the first and second image.

In particular, according to an embodiment of the approach presented here, in the step of moving a mirror of the camera system with respect to a camera of the camera system can be moved. Such an embodiment of the approach presented here has the advantage that only a small element of the camera system needs to be moved between the recording of the first image and the recording of the second image in order to achieve a change in the viewing direction of the camera system. This leads to an improvement in the robustness, the operating costs and the manufacturing costs of such a camera system without significant loss of precision in determining the position of the object in the environment of the mobile unit.

Also, according to a further embodiment of the approach presented here, in the step of reading in the first and / or second image, the relevant image can be read by a camera system which provides at least one image for utilization in another driver assistance function. Such a driver assistance function may be, for example, a lane keeping function or a light control function for controlling the vehicle light on the basis of an optical image of the camera system. Such an embodiment of the approach presented here has the advantage of multiple use of images of the camera system, which can be inexpensively another comfort feature with little additional effort can be realized.

Also conceivable is an embodiment of the approach presented here, in which the step of reading the second image in response to a Detection and / or read-in signal takes place, in particular wherein the detection and / or read-in signal represents an external physical quantity about an imminent use of the mobile unit. Such a detection signal or read-in signal may, for example, be a signal representing information indicating that the second image has been acquired by the camera system. Also, the detection signal or read-in signal may be a signal that signals the camera system to capture and provide the second image. The detection signal or read-in signal can be triggered, for example, by a sensor which can detect a state change or a physical action on the mobile unit. For example, such a sensor may detect the opening of a door or flap of the mobile unit (especially if the mobile unit is a vehicle) and generate the corresponding detection signal or read-in signal. In this case, it can be assumed that by opening the door or flap of the mobile unit, a slight change in the perspective of the camera system has taken place, which is sufficient to record the first image and the second image from different perspectives, so that the first and second Image is a determination of the object or the spatial position of the object in the environment of the mobile unit is possible.

According to another embodiment of the approach presented here, the method may comprise a step of controlling a movement of the mobile unit using the determined object and / or the determined position of the object in the environment of the mobile unit. Such an embodiment of the approach presented here offers the advantage of an early warning or an early control intervention in the movement of the mobile unit in order to prevent damage to the mobile unit and / or the object as much as possible.

The approach presented here also creates a device that is designed to perform the steps of a variant of a method presented here in appropriate facilities to drive or implement. Also by this embodiment of the invention in the form of a device, the object underlying the invention can be solved quickly and efficiently.

In the present case, a device can be understood as meaning an electrical device which processes sensor signals and outputs control and / or data signals in dependence thereon. The device may have an interface, which may be formed in hardware and / or software. In the case of a hardware-based embodiment, the interfaces can be part of a so-called system ASIC, for example, which contains a wide variety of functions of the device. However, it is also possible that the interfaces are their own integrated circuits or at least partially consist of discrete components. In a software training, the interfaces may be software modules that are present, for example, on a microcontroller in addition to other software modules.

Also of advantage is a computer program product or computer program with program code which can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and for carrying out, implementing and / or controlling the steps of the method according to one of the embodiments described above is used, especially when the program product or program is executed on a computer or a device.

The approach presented here will be explained in more detail below with reference to the accompanying drawings. Show it:

1 a block diagram of a vehicle as a mobile unit with a device according to an embodiment of the present invention;

2 a flowchart of a method according to an embodiment of the present invention;

3 a perspective view of a built-in front camera system and the determination of an adjusted height adjustment of the vehicle before the detection of the second image;

4 a plurality of images from which the object in the vicinity of the vehicle can be determined as a mobile unit; and

5 an image in which the changes of corresponding pixel locations are shown in two different images, which are useful for determining the object.

In the following description of favorable embodiments of the present invention, the same or similar reference numerals are used for the elements shown in the various figures and similar acting, with a repeated description of these elements is omitted.

1 shows a vehicle as a mobile unit 100 that a device 110 according to an embodiment of the present invention. The device 110 is with a camera system 120 connected, the viewing direction, for example, in a forward direction 130 of the vehicle is directed. The device 110 for determining the object or a position of the object in the environment of the vehicle 100 , an interface 135 for reading in from the camera system 120 provided first and second pictures, as well as a unit 140 for determining the object and / or the position of the object in the environment of the vehicle using the via the interface 135 read in pictures.

For example, an occupant enters via a (left) door 145 the vehicle 100 , is the spring bearing the vehicle 100 when sitting down the occupant on a vehicle seat 150 the vehicle 100 Lower yourself on the left side, without, for example, its position to a reference point 155 for example, a section of a roadway as a footprint of the vehicle 100 forms, is significantly changed. The occupant therefore becomes the vehicle seat 150 in a stationary vehicle 100 occupy. By lowering the vehicle 100 on the left side when boarding the occupant through the vehicle door 140 thus becomes the camera system 120 minimally leftward or downwards, as indicated by the dashed line of the camera system 120 in the 1 is shown. It can from the camera system 120 at a first time, in which the camera system 120 in the solid right representation 1 is, a first image to the interface 135 be delivered in which an object 160 in the environment 165 of the vehicle 100 is shown. At a second time after a vehicle occupant the vehicle seat 150 through the vehicle door 125 occupied by the (slightly moving) camera system 120 the second image of the interface 135 be provided, in which also the object 160 is included. The object 160 is contained in the two images in different places, which are due to the different directions of the camera system 120 are served at the first time or at the second time. In the unit for determining, it is then possible from the at least first and second image to locate the location at which the object is located 160 has been mapped accordingly, from this information with appropriately designed algorithms, the objects and / or the position of the object 160 in the neighborhood 165 of the vehicle 100 to determine or determine.

At a time of entry of the occupant into the vehicle 100 or to be able to recognize the driver's seat, for example, a sensor 167 be provided, which is a signal 168 about the occupancy of the driver or vehicle seat 150 gives. This signal 168 can then as a trigger signal for detecting the second image by the camera system 120 or used as a read-in signal for processing the second image in the device for detecting. In this way it can be ensured that the second image has advantageously been detected at a point in time during which a small change in the perspective of the camera of the camera system 120 is done. This allows a clearly secure recognition of the object 160 in the environment 165 of the vehicle 100 ,

Analogously, it is also conceivable that the camera system 120 not like in the 1 illustrated embodiment in a vehicle 100 is arranged, but on a robotic arm, as an object 160 can detect an obstacle in a path of movement of the robot arm. The approach described in more detail below can be used analogously, regardless of whether a position of an object 160 in an environment 165 of a vehicle 100 or one (in 1 not shown) environment of a robot to be detected. In this sense, the generic term mobile unit is to be understood, which is intended to designate a unit which itself can be moved, such as the vehicle 100 or the robot.

2 shows a flowchart of an embodiment of the approach presented here as a method 200 for recognizing at least one object and / or one position of the object in an environment of a mobile unit, in particular of a vehicle and / or a robot. The procedure 200 includes a step 210 reading in a first image containing the object of the environment of the mobile unit that was acquired at a first time, wherein the first image is read in by a camera system arranged in or on the mobile unit. Furthermore, the method comprises 200 one step 220 reading in at least one second image containing the object of the environment of the mobile unit acquired at a second time, the second image being read by the camera system and the mobile unit resting between the first and second times or by no more than a tolerance distance a reference position has been moved in the vicinity of the mobile unit. Finally, the process includes 200 one step 230 determining the object and / or the position of the object in the environment of the mobile unit using the first and second images.

The object of the approach presented here can be seen in the obstacle detection or even measurement of the environment during the standstill of a vehicle or robot. For this purpose, a single monocular camera is sufficient for the proposed method 120 which, as such, is not intended to measure the environment 165 during the stoppage of the vehicle 100 or in general the mobile unit is suitable. The survey of the environment 165 happens in the field of view of the camera 120 , An application of the method 200 on several / other cameras 120 on / in vehicle 100 or robot is of course possible and would increase the detection area.

The capture of the environment 165 during the stoppage of the vehicle 100 is useful because the decision as to whether and in what direction obstacle-free movement is possible before the actual movement of the vehicle 100 / Robot takes place and thus does not need to be corrected. This expresses a smarter behavior for possibly semi-autonomous or autonomous vehicles 100 (especially when starting) off. Furthermore, an object of the approach presented here, existing sensors such as the camera system 120 and their infrastructure. This concerns already installed cameras 120 (eg for driver assistance or all-round purposes), which today are predominantly monocular, as well as their lines, control units 110 , Algorithms, etc .. This generates added value or enables cost-effective overall solutions.

With a monocular camera 120 who are not concerned about their environment 165 moving, it is generally not possible to make a three-dimensional survey of the environment. Also a movement of the camera 120 around itself - more precisely: around its projection center - does not allow a three-dimensional measurement of the environment 165 ,

Only when the movement of the monocular camera 120 also has a translation portion, is a three-dimensional measurement of the environment 165 possible. Unfortunately, the three-dimensional information can not be obtained at standstill. Thus, no information about possible obstacles or unsuitable sections of the route is available. The present approach solves this problem in particular by using (small or smallest) movements of the camera 120 that have a translatory component. These movements can be either of a "passive" or "targeted" kind.

By "targeted" movement is to be understood that the movement is specifically carried out to the monocular camera 120 to carry out a three-dimensional survey. By "passive" movement, on the other hand, it should be understood that the movement already takes place for other reasons and is used for the procedure.

• – Jede Stelle an der Karosserie oder hinter Scheiben, von der aus die Umgebung 165 beobachtet wird, während die Karosserie sich im Stillstand (wenn auch nur leicht) bewegt bzw. bewegt wird.
• – Im Außenspiegelgehäuse bzw. am Außenspiegel, wenn eine automatische Anklappfunktion oder sonstige Verstellfunktion vorhanden ist.
• – In/an einer Tür oder im Außenspiegelgehäuse (auch ohne automatische Anklappfunktion), mit Nutzung der Bewegung der Tür und/oder Karosserie beim Ein-/Aussteigen.
• – Im Scheinwerfergehäuse unter Nutzung von beweglichen Teilen, z. B. für Kurvenlicht oder Leuchtweitensteuerung.
• – In/an der Heckklappe bzw. Kofferraumklappe, mit Nutzung der Bewegung beim automatischen oder manuellen Öffnen oder Schließen.

Examples of possible positions of the camera 120 , in which easily translational portions of the movements can be exploited (without claim to completeness):

• - Any place on the body or behind discs, from which the environment 165 is observed while the body is moving (even slightly) at standstill.
• - In the exterior mirror housing or on the exterior mirror if an automatic folding function or other adjustment function is available.
• - In / on a door or in the exterior mirror housing (also without automatic folding function), with use of the movement of the door and / or body when entering / exiting.
• - In the headlight housing using moving parts, eg. B. for cornering lights or headlight range control.
• - In / on the tailgate or trunk lid, using the movement during automatic or manual opening or closing.

• – Da nur monokulare Kamera(s) notwendig ist (sind) – und kostengünstiger als z. B. Stereokamera oder andere Sensoren.
• – Mehrfachnutzen von bestehenden Sensoren, die vorwiegend auf monokularer Basis sind (z. B. Kameras 120 für Fahrerassistenz, Rundumsicht).
• – Vermessung der Umgebung 165 während des Stillstands des Fahrzeuges.
• – Mehrfachnutzung der vorhandenen Algorithmen (z. B. optischer Fluss) und Systeme zur Bildauswertung, die während des Fahrzeugstillstands ungenutzte Kapazitäten bereitstellen können.
• – Die gewonnenen Umgebungsinformationen können nicht nur für die Fortbewegung genutzt werden, sondern auch, um weitere Funktionalität im Fahrzeug anzubieten, z. B. das Blockieren von Türen oder Heckklappen während des Öffnens, falls ein Gegenstand im Weg ist.

The approach presented here has several advantages. These include in particular:

• - Since only monocular camera (s) is necessary (are) - and cheaper than z. B. stereo camera or other sensors.
• - Multiple benefits of existing sensors, which are predominantly on a monocular basis (eg cameras 120 for driver assistance, all-round visibility).
• - Surveying the environment 165 during the stoppage of the vehicle.
• - Multiple use of existing algorithms (eg optical flow) and image analysis systems that can provide idle capacity during vehicle standstill.
• The obtained environmental information can be used not only for locomotion, but also to offer additional functionality in the vehicle, eg. As the blocking of doors or tailgates during opening, if an object is in the way.

Methods for the three-dimensional measurement of a scene with a monocular camera moved relative to the scene are known to the person skilled in the art. In the literature, this topic can be found in particular under the term "Structure from Motion (SfM)". A well-known textbook in this field is: Richard Hartley and Andrew Zisserman.-Multiple View Geometry in Computer Vision. Cambridge University Press. Second edition, 2004 ,

The approach presented here uses movements of the camera 120 concerning the environment 165 off, while, for example, the vehicle 100 still at a standstill.

These movements can also be small: movements in the order of a millimeter are already sufficient. Larger movements, z. B. in the order of centimeters, allow a 3D reconstruction of the surrounding environment 165 with high accuracy. Are the movements even larger (z. B. when opening the door 145 ), so a desired accuracy can be achieved with less effort, z. B. with lower image resolution and lower computational effort.

• – Bestimmung von korrespondierenden Punkten zwischen mindestens zwei Bildern der Szene, die zu unterschiedlichen Zeitpunkten aufgenommen wurden. Idealerweise ist die Szene dabei im Wesentlichen unbewegt. Trotzdem bewegen sich die korrespondierenden Punkte im Bild, da sich die Kamera 120 relativ zur Szene zumindest geringfügig bewegt. Verfahren zur Bestimmung von korrespondierenden Punkten in Bildfolgen sind dem Fachmann bekannt, z. B. unter den Begriffen „Motion Analysis“ oder „Optic(al) Flow“. Wenn hier von der Suche nach korrespondierenden Punkten gesprochen wird, so ist es in der Praxis zutreffender von korrespondierenden (kleinen) Bildregionen zu sprechen, nach denen gesucht wird.
• – Optional vorgesehen sein kann eine Bestimmung der Eigenbewegung der Kamera 120 aus der Bildfolge (dem Fachmann bekannt im Rahmen von SfM) oder aus der durch die Konstruktion bekannten Mechanik der Bewegung. Hierzu können auch Signale von Sensoren oder Gebern oder Schaltern oder Lichtschranken oder Trägheitssensoren oder Magneten oder dergleichen ausgenutzt werden, z. B. Winkelinformationen von Stellmotoren oder Getrieben, die die Lage der Kamera 120 in Beziehung zu zumindest einer weiteren Komponente der Mobileinheit 100, hier des Fahrzeugs erfasst.
• – Rekonstruktion von Zwischeninformationen in geeigneter Form, sodass eine weitere Stufe basierend auf den Zwischeninformationen eine Entscheidung treffen kann. Diese Zwischeninformation kann beispielsweise eine dreidimensionale Rekonstruktion des Umfelds sein, die aus den korrespondierenden Punkten durch Triangulation mit der SfM-Methode gewonnen wird. Aus einer solchen dreidimensionalen Rekonstruktion, z. B. einer 3D-Punktwolke, können dann weitere Zwischeninformationen extrahiert werden, z. B. über das Vorhandensein von Punkten in einer bestimmten Höhe über der Fahrbahn, sodass sie als Hindernis 160 zu interpretieren sind. Aber auch das Fehlen von erwarteten Punkten auf der Fahrbahn kann keine Zwischeninformationen darstellen, aus der z. B. auf das mögliche Vorhandensein eines Abgrunds als Objekt 160 geschlossen werden kann. Es ist nicht zwangsläufig notwendig, eine dreidimensionale Rekonstruktion des Umfelds 165 oder Objekten 160 darin vorzunehmen, denn vielfach können bereits aus Informationen, die mit geringerem Aufwand zu erhalten sind, die notwendigen Entscheidungen getroffen werden. Beispielsweise kann die eindimensionale Information über ein Maß der Verschiebung zwischen korrespondierenden Bildpunkten schon ausreichen, um eine Entscheidung zu treffen. In einem Beispiel wird dies später noch verdeutlicht.
• – Besagte Entscheidungsstufe mit Ansteuerung einer Aktuatorik, z. B. um ein automatisches Anfahren freizugeben oder zu verhindern, z. B. wenn ein Hindernis 160 im Wege steht oder wenn das Fahrzeug 100 in einen Abgrund stürzen könnte.

For obtaining the three-dimensional information due to a small movement of the camera 120 With regard to the environment, for example, the following steps are carried out:

• - Determination of corresponding points between at least two images of the scene, which were taken at different times. Ideally, the scene is essentially unmoved. Nevertheless, the corresponding points move in the picture, as the camera 120 at least slightly moved relative to the scene. Methods for determining corresponding points in image sequences are known to the person skilled in the art, for. For example, under the terms "motion analysis" or "optic (al) flow". When speaking of the search for corresponding points, in practice it is more appropriate to speak of corresponding (small) image regions that are searched for.
• Optionally, provision may be made for determining the proper motion of the camera 120 from the image sequence (known to those skilled in the context of SfM) or from the mechanics of motion known by the construction. For this purpose, signals from sensors or encoders or switches or light barriers or inertial sensors or magnets or the like can be exploited, for. B. angle information of servomotors or transmissions, the position of the camera 120 in relation to at least one other component of the mobile unit 100 , captured here of the vehicle.
• Reconstruction of intermediate information in a suitable form, so that a further stage based on the intermediate information can make a decision. This intermediate information can be, for example, a three-dimensional reconstruction of the environment, which is obtained from the corresponding points by triangulation with the SfM method. From such a three-dimensional reconstruction, z. B. a 3D point cloud, then more intermediate information can be extracted, for. For example, the presence of points at a certain height above the roadway, making them an obstacle 160 to interpret. But also the lack of expected points on the roadway can not represent intermediate information, from the z. B. on the possible presence of an abyss as an object 160 can be closed. It is not necessarily necessary to have a three-dimensional reconstruction of the environment 165 or objects 160 In many cases, the necessary decisions can already be made from information that can be obtained with less effort. For example, the one-dimensional information about a degree of shift between corresponding pixels may be enough to make a decision. In an example, this will be clarified later.
• - said decision stage with control of an actuator, z. B. to release an automatic start or prevent, for. B. if an obstacle 160 gets in the way or if the vehicle 100 could plunge into a chasm.

• – Bewegung des Außenspiegel-Gehäuses, wobei die Kamera mit diesem Gehäuse verbunden ist. Es kann beispielsweise eine Mitnutzung der Bewegung der Anklapp-Funktion des Spiegels im Parkmodus oder aus dem Parkmodus des Fahrzeugs 100 zur Ermittlung von einem Objekt 160 oder der Position desselben erfolgen. Dabei reicht bereits eine kleine Bewegung aus. Es braucht also der Spiegel nicht ganz angeklappt werden, sondern es wird nur eine kleine Strecke von der möglichen Bewegungsstrecke ausgenutzt.
• – Bewegung des Außenspiegels, wobei die Kamera 120 mit diesem Spiegel verbunden ist. Es erfolgt eine Mitnutzung der Bewegung der Spiegelverstellung durch einen Fahrer des Fahrzeugs 100.
• – Bewegung der Fahrer- oder Beifahrertür 145, wobei die Kamera mit der Tür 145 oder mit dem Spiegel oder Spiegelgehäuse an der Tür 145 verbunden ist. Mitnutzung der Bewegung der Tür 145 beim Öffnen oder Schließen. Hierbei kann die Kamera (die auch schon vorhanden und sind z. B. für Rundumsicht- oder Topview-Funktionen vorgesehen sind) auch dazu genutzt werden, um eine mögliche Kollision der Tür 145 z. B. mit der Garagenwand oder einem anderen Fahrzeug rechtzeitig zu erkennen.
• – Bewegung der Heckklappe oder Kofferraumklappe, wobei die Kamera mit der Klappe verbunden ist. Häufig sind solche Kameras als Rückfahrkameras vorgesehen. Im Falle von elektrisch öffnenden/schließenden Klappen ist es denkbar, dass eine kleine Bewegung (z. B. im Millimeter-Bereich) gezielt durchgeführt wird, damit mit der Kamera 120 eine Vermessung durchgeführt werden kann.

Below is a list of ways that a camera can perform at least a small amount of movement without the vehicle 100 need to move. These possibilities will be briefly explained below and are not explicitly shown in the figures.

• - Movement of the exterior mirror housing, the camera is connected to this housing. It may, for example, a joint use of the movement of the folding function of the mirror in the parking mode or from the parking mode of the vehicle 100 to determine an object 160 or its position. A small movement is enough. So it does not need to be folded completely, but only a small distance of the possible movement distance is used.
• - Movement of the exterior mirror, with the camera 120 connected to this mirror. There is a shared use of the movement of the mirror adjustment by a driver of the vehicle 100 ,
• - Movement of the driver or passenger door 145 , with the camera with the door 145 or with the mirror or mirror housing on the door 145 connected is. Sharing the movement of the door 145 when opening or closing. Here, the camera (which is already available and are provided, for example, for all-round view or topview functions) can also be used to a possible collision of the door 145 z. B. with the garage wall or another vehicle in time to recognize.
• - Moving the tailgate or trunk lid, with the camera connected to the flap. Frequently, such cameras are provided as rear view cameras. In the case of electrically opening / closing flaps, it is conceivable that a small movement (eg in the millimeter range) is purposefully carried out, thus with the camera 120 a survey can be performed.

• – Bewegung des Fahrzeugaufbaus aufgrund des Einsteigens (oder Aussteigens) von Personen. Diese Bewegung entsteht durch die Masse der Personen und das Einfedern des Fahrzeugs. Die Bewegung liegt in der Größenordnung von Millimetern. Da der ganze Fahrzeugaufbau sich dabei bewegt, können alle Kameras davon profitieren.
• – Nutzung der Bewegung durch eine aktive Federung oder eine Fahrzeughöhenverstellung. Viele höherwertige Fahrzeuge sind mit einem System ausgestattet, das ein Anheben / Absenken zwischen Rad und Karosserie erlaubt. Es ist denkbar, dass solche Bewegungen mit kleinem (und für die Insassen kaum wahrnehmbarem) Hub gezielt durchgeführt werden, um Vermessungen durchzuführen. Davon können wieder alle Kameras rund um das Fahrzeug profitieren.
• – Nutzung der Bewegung der Kurvenlicht-Schwenkeinrichtung oder Leuchtweitenregulierung für Scheinwerfer: Aus einer Vielzahl von Gründen (gute Übersicht im Frontbereich, Schutz durch Scheinwerfer/Scheibe, wenig Verschmutzung, wenig Vereisung, Vorhandensein von Spiegeln, Vorhandensein einer Scheinwerfer-Reinigungsanlage) kann es sinnvoll sein, eine Kamera im Scheinwerfer-Gehäuse unterzubringen. Sehr viele Fahrzeuge können ihre Scheinwerfer um einige Grad nicken (Leuchtweitenregulierung) oder schwenken (Kurvenlicht). Viele neuere Fahrzeuge weisen eine schwenkbare kleine Maske auf, mit der entgegenkommende Fahrzeuge so ausmaskiert werden, dass diese vom Scheinwerfer nicht geblendet werden. Für die genannten Bewegungen von Komponenten im Scheinwerfer sind in der Regel Elektromotoren als Aktuatoren vorgesehen. Es ist denkbar, dass die Aktuatoren vor dem Anfahren eine gezielte (kleine) Bewegung durchführen, damit mithilfe der Kameras das Anfahren abgesichert werden kann. Dazu ist die jeweilige Kamera so anzubringen, dass sie von einem Aktuator mitbewegt wird und dass die Kamerabewegung dabei auch eine translatorische Komponente aufweist.
• – Aufbringen eines Antriebs-Drehmoments auf mindestens ein Rad (vorzugsweise auf die ganze Antriebsachse), während die Räder noch gebremst sind. Hierdurch vollführt das Fahrzeug eine Aufbäumbewegung (vorne oder hinten, je nach Antriebsachse). Diese Bewegung kann von jeder der vorhandenen Kameras für das hier vorgestellte Verfahren genutzt werden.
• – Eine Kamera kann aus einem Versteck oder aus einem Schutzgehäuse oder hinter einem Schutzdeckel herausbewegt werden und die Phase der Bewegung nutzen. Eine Schutzabdeckung kann in erster Linie dafür gedacht sein, die Kamera vor Verschmutzung oder Beschädigung oder Vandalismus oder Sichtbarkeit zu schützen. Die Bewegungsmöglichkeit der Kamera ist bereits dazu vorgesehen, um die Kamera in eine für die Primärfunktion geeignete Funktion zu bringen. Während dieser Bewegung kann die Kamera aber bereits Bilder aus unterschiedlichen Perspektiven für das hier vorgestellte Verfahren (als Sekundärfunktion) gewinnen.
• – Alternativ zu einer Bewegung der Kamera besteht in allen Fällen auch die Möglichkeit, stattdessen einen Spiegel zu bewegen und mit der Kamera über den bewegten Spiegel die Szene zu beobachten. Die Kamera 120 kann dann starr mit dem Fahrzeugaufbau verbunden sein. Insbesondere dort, wo ohnehin schon spiegelnde Flächen vorhanden sind (z. B. Reflektor im Scheinwerfer oder Außenspiegel), bietet sich diese Möglichkeit an.

A particularly useful application in connection with the tailgate is to avoid the collision with the garage ceiling, the garage door or other obstacles that impede a free swinging. The typical installation position of the camera in the area of the handle bar offers ideal conditions for detecting or measuring an obstacle during swinging due to the flap movement and for stopping the flap in good time, so that contact and damage is avoided.

• Movement of the vehicle body due to boarding (or disembarking) of persons. This movement is caused by the mass of people and the deflection of the vehicle. The movement is on the order of millimeters. As the entire vehicle body moves, all cameras can benefit.
• - Use of the movement by an active suspension or a vehicle height adjustment. Many higher value vehicles are equipped with a system that allows lifting / lowering between the wheel and the body. It is conceivable that such movements with small (and barely perceptible to the occupants) stroke are selectively carried out to perform surveys. All cameras around the vehicle can benefit from this again.
• - Use of the movement of the cornering light pivoting device or headlamp leveling for headlights: For a variety of reasons (good overview in the front area, protection by headlamp / disc, little pollution, little icing, presence of mirrors, presence of a headlight cleaning system), it may make sense to house a camera in the headlight housing. Many vehicles can nod their headlights by a few degrees (headlamp leveling) or swivel (cornering light). Many newer vehicles have a swiveling small mask, with the oncoming vehicles are masked out so that they are not dazzled by the headlight. For the above movements of components in the headlights electric motors are usually provided as actuators. It is conceivable that the actuators make a targeted (small) movement before starting so that the approach can be secured with the aid of the cameras. For this purpose, the respective camera is to be mounted so that it is moved by an actuator and that the camera movement thereby also has a translational component.
• - Applying a drive torque to at least one wheel (preferably on the entire drive axle) while the wheels are still braked. As a result, the vehicle performs a Aufbäumbewegung (front or rear, depending on the drive axle). This movement can be used by any of the existing cameras for the procedure presented here.
• - A camera can be moved out of a hiding place or from a protective housing or behind a protective cover and use the phase of the movement. A protective cover may be primarily intended to protect the camera from dirt or damage or vandalism or visibility. The possibility of movement of the camera is already provided to bring the camera in a function suitable for the primary function. During this movement, however, the camera can already gain images from different perspectives for the process presented here (as a secondary function).
• - As an alternative to a movement of the camera in all cases, it is also possible to move a mirror instead and to observe the scene with the camera via the moving mirror. The camera 120 can then be rigidly connected to the vehicle body. In particular, where reflective surfaces are already present (eg reflector in the headlight or exterior mirror), this possibility is appropriate.

While the previously mentioned examples minimize the overhead for the camera-based obstacle detection because they share existing possibilities of movement anyway, there is of course always the possibility of providing new possibilities of movement.

For example, a camera 120 be moved up and down or sideways, to capture images from different perspectives.

As an illustrative proof of the practicality of the approach presented here is the experiment described below.

For example, a wide-angle monocular camera is rigidly mounted on a vehicle in the area of the radiator grille, so that the camera is the area that is barely visible to the driver in front of the vehicle 100 detected, 3 shows a perspective view of such a camera 120 in a test setup in a scene with obstacles 160 in front of a vehicle 100 , in the area of the grille, a monocular experimental camera 120 is appropriate. In front of the vehicle 100 There are thus obstacles 160 so that a start can be avoided forward.

The vehicle used 100 has an air suspension, in which the ground clearance can be varied. The raising and lowering can be controlled electronically. The camera 120 takes running pictures of the scene in front of the vehicle 100 on. Between the first picture 400 and the second picture 410 out 4 the stroke is 15 mm. This can be a difference image 420 to create a better visual clarification of the difference perspective representation of the objects or obstacles 160 in the pictures 400 and 410 in the 4 is shown.

Between these pictures 400 and 410 a correspondence analysis (search for corresponding pixels) is performed with an optical flow algorithm. This provides about 100,000 corresponding pixel pairs in 5 as motion vectors 500 are shown. The exact number is actually insignificant, as long as there are enough, so that there is so much correspondence on each potential obstacle that the obstacle can be recognized as such. From the different lengths and directions of these vectors 500 can be z. B. gain information about the three-dimensional scene with known SfM algorithms (Structure from Motion). 5 thus shows a result of correspondence formation between the two pictures 400 and 410 out 4 ,

If the amount of the stroke between shots is known, then a metric scaling factor for the three-dimensional reconstruction can be obtained therefrom. Even with the naked eye is out 5 already recognizable that the obstacles 160 , which stand in the way of a start-up here, can be easily detected or detected by the correspondence analysis. Even a much smaller stroke of, for example, just one millimeter would already be sufficient here for the obstacles 160 so that a function can react appropriately.

The person skilled in the art can specify here further suitable algorithms in order to carry out a corresponding evaluation and to make a suitable decision. The formation of three-dimensional intermediate information with SfM methods is therefore useful, but not absolutely necessary.

A decisive advantage of the approach proposed here is the fact that it can be used in existing systems and can use existing cameras and infrastructure (lines, control units, algorithms, etc.).

As already explained above, the approach proposed here can also be generalized and z. B. transferred to robots.

The embodiments described and shown in the figures are chosen only by way of example. Different embodiments may be combined together or in relation to individual features. Also, an embodiment can be supplemented by features of another embodiment.

Furthermore, the method steps presented here can be repeated as well as executed in a sequence other than that described.

If an exemplary embodiment comprises a "and / or" link between a first feature and a second feature, then this is to be read so that the embodiment according to one embodiment, both the first feature and the second feature and according to another embodiment either only first feature or only the second feature.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited non-patent literature

• Richard Hartley and Andrew Zisserman.-Multiple View Geometry in Computer Vision. Cambridge University Press. Second edition, 2004 [0039]

Claims (13)

Procedure ( 200 ) for recognizing at least one object ( 160 ) in an environment ( 165 ) of a mobile unit ( 100 ), in particular a vehicle and / or a robot, wherein the method ( 200 ) comprises the following steps: - reading in ( 210 ) one of the object ( 160 ) containing the first image ( 400 ) the environment ( 165 ) of the mobile unit ( 100 ) captured at a first time, the first image ( 400 ) from or in the mobile unit ( 100 ) arranged camera system ( 120 ) is read; - read in ( 220 ) at least one object ( 160 ) second image ( 410 ) the environment ( 165 ) of the mobile unit ( 100 ) detected at a second time, the second image ( 410 ) from the camera system ( 120 ) and the mobile unit ( 100 ) is at rest between the first and second times or by not more than one tolerance distance from a reference position ( 155 ) in the neighborhood ( 165 ) of the mobile unit ( 100 ) was moved; and - determining ( 230 ) of the object ( 160 ) in the neighborhood ( 165 ) of the mobile unit ( 160 ) using the first ( 400 ) and second ( 410 ) Picture. Procedure ( 200 ) according to claim 1, characterized in that in the step of reading ( 220 ) of the second image ( 410 ) a picture as second picture ( 410 ) which is read from another position of the camera system ( 120 ) or at least one component of the camera system ( 120 ) in relation to at least one element ( 145 ) of the mobile unit ( 100 ) was taken as the first picture. Procedure ( 200 ) according to one of the preceding claims, characterized in that in the step of determining ( 230 ) a position of the object ( 160 ) in relation to the mobile unit ( 100 ) is determined. Procedure ( 200 ) according to one of the preceding claims, characterized in that in the steps of reading in ( 210 ) of the first image ( 400 ) and reading in ( 220 ) of the second image ( 410 ) the first ( 400 ) and second ( 410 ) Image from a monocular camera system ( 120 ) are read. Procedure ( 200 ) according to one of the preceding claims, characterized in that in the step of reading ( 220 ) of the second image ( 410 ) as a second image ( 410 ) an image is read in from a mobile unit at the second time ( 100 ) folded element ( 145 ) was recorded. Procedure ( 200 ) according to one of the preceding claims, characterized by a step of moving at least a part of the camera system ( 120 ) with regard to at least one element ( 145 ) of the mobile unit ( 100 ) before the reading step ( 240 ) of the second image ( 410 ). Procedure ( 200 ) according to claim 6, characterized in that in the step of moving a mirror of the camera system ( 120 ) with respect to a camera of the camera system ( 120 ) is moved. Procedure ( 200 ) according to one of the preceding claims, characterized in that in the step of reading ( 210 . 220 ) of the first ( 400 ) and / or second ( 410 ) Image the relevant image from a camera system ( 120 ), which provides at least one image for utilization in another driver assistance function. Procedure ( 200 ) according to one of the preceding claims, characterized in that the reading-in step ( 210 ) of the second image ( 410 ) in response to a detection and / or read-in signal ( 168 ), in particular wherein the detection and / or read-in signal ( 168 ) represents a physical quantity that contains information about an imminent use of the mobile unit ( 100 ). Procedure ( 200 ) according to one of the preceding claims, characterized by a step of controlling a movement of the mobile unit ( 100 ) using information about the detected object ( 160 ) and / or the determined position of the object ( 160 ) in the environment of the mobile unit ( 100 ). Contraption ( 119 ), which is designed to handle all the steps of a process ( 200 ) according to one of the preceding claims. Computer program adapted to perform all steps of a procedure ( 200 ) according to one of the preceding claims. A machine-readable storage medium having a computer program stored thereon according to claim 12.

Images