DE102014114272B4 - Method for controlling a robot arrangement guided by image acquisition - Google Patents

Abstract

A method (200) for controlling a robot assembly (101) comprising: operating a robotic arm (108) to perform a start action at a start position; moving the robotic arm (108) from the start position to a first position; a controller (102) executing a visual processing subroutine (300), the visual processing subroutine (300) comprising: determining whether a first portion (X1) will be ready to undergo a first action by the robotic arm (108) As soon as the robotic arm (108) reaches the first position, a positional deviation of a second part (X2) from a second position is determined, it is determined whether the second part (X2) will be ready for a second action by the robotic arm (108 ) when the robot arm (108) reaches the second position; andthe first action is performed on the first part (X1) using the robot arm (108) when the robot arm (108) is located within a positional deviation of the first part (X1) from the first position that is detected by the visual processing subroutine (300). wherein the steps of operating, moving, executing and performing are repeated, wherein the first position is newly provided as the start position, the new action is newly provided as the start action, the second part (X2) is the first part (X1 ), the second position is newly provided as the first position, the second action is newly provided as a first action, and the positional deviation of the second part (X2) from the second position is the positional deviation of the first part (X1) from the first position wherein the visual processing subroutine (300) is an embedded process that operates continuously in the background; the visual processing subroutine (300) comprises identifying a next portion of a designated location to be subjected to a next action; determining whether a camera (128) is ready to capture an image of the identified portion from the designated location; and determining that the identified part is in a position where the camera (128) is capable of capturing the image of the identified part; and wherein determining whether the identified part is in a position where the camera (128) is capable of capturing the image of the identified part is based on input from at least one part sensor (124).

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority of the provisional U.S. Application No. 61 / 886,764 filed on 4 October 2013 and which is hereby incorporated by reference in its entirety.

LICENSE RIGHTS OF THE GOVERNMENT

This invention has been carried out with Government support under contract number DE-EE-0002217 awarded by the Department of Energy. The government has certain rights to the invention.

TECHNICAL AREA

The present disclosure relates to a system and method for controlling a robot assembly.

BACKGROUND

At least some manufacturing processes use a robotic assembly to assemble certain devices. For example, batteries may be assembled using a robotic assembly. The robotic assembly includes at least one movable robotic arm capable of performing an action on a part, such as drilling or welding. A conventional robot assembly uses precision tools, fixtures, or grates that allow the robot to handle parts at preprogrammed positions and orientations. This rigid, expensive placement of parts also limits the flexibility of the robot assembly to a particular or only one device.

A robot assembly with camera image capture for steering the part position is a new way of assembling parts without expensive hardware for precise part positioning, thus making production more flexible and less expensive. However, the acquisition and processing of image capture images requires time and proper setup for the desired accuracy and reliability of the robotic steering. It is therefore useful to control the process of robotic arm manipulation and image acquisition in conjunction to optimize the manufacturing process and thereby minimize the time required to fully assemble a device using the robotic system.

The publication DE 11 2009 001 414 T5 discloses a control method for a conveyor having a robot arm and an optical sensor, in which workpieces on the conveyor are detected by the optical sensor, an actual conveyor state is calculated for each detected workpiece, and the workpieces are processed using the robot arm.

In the publication DE 10 2009 034 529 A1 For example, there is disclosed a method for a visual guidance and recognition system that does not require calibration. In this case, an image of a workpiece to be learned is detected by a camera mounted on a faceplate of a robot arm and used together with a distance from workpiece to robot arm for learning the workpiece, so that during operation a workpiece can be detected and the robot arm can be precisely positioned.

The publication US 2013/0 166 061 A1 discloses an object grasper having a camera for taking images of a plurality of disordered objects from which positions and orientations of the plurality of objects are estimated. Based on the estimated positions and orientations, the objects are taken individually and stored at a suitable location.

SUMMARY

The present disclosure relates to a method of controlling a robot assembly using cameras. In the presently disclosed method, the cameras may be used to direct the movement of a robotic arm of the robotic assembly. Using the presently disclosed method, the cameras capture an image of a part while the robotic arm is in motion, rather than waiting for the robotic arm to reach its target position, thereby minimizing the cycle time of a manufacturing process.

In one embodiment, the method comprises a main process comprising the steps of: operating a robotic arm to perform an action at a start position; Moving the robotic arm from the start position to a first position; Determining, from an image acquisition process method, whether a first part from the first position will be ready to undergo a first action by the robotic arm when the robotic arm reaches the first position; Commencing execution of the visual processing method to determine the positional deviation of the second portion from the second position and whether the second portion is ready to undergo a second action by the robotic arm when the robotic arm reaches the second position; and performing a first one Action on the first part using the robot arm, wherein the positional deviation of the first part from the first position is predetermined by the image acquisition process method. The steps are repeated for the next robotic assembly cycle, with the first position and action being re-scheduled as the new starting position, the second part, the second position and the second action being the new first part, the new first position or the new first action is provided and the positional deviation of the second part is provided as a positional deviation of the new first part.

The method also includes the image capture process method, an embedded process that operates continuously in the background, and includes the steps of: identifying the next part of a designated location to be actioned; Checking that the camera is ready to capture the image of the identified part from the intended position; Check that the identified part is in position for image acquisition using a parts sensor; Taking an image of the identified part from the intended position using a camera while the robot arm moves toward the intended position; Determining a positional deviation of the identified part from the intended position based at least in part on the position of the identified part as the robot arm moves toward the intended position; Set the status of the identified part from the intended position ready to undergo action.

The present disclosure also relates to systems for controlling a robotic assembly. In one embodiment, the system includes a robotic assembly having a robotic arm and an arm actuator operatively coupled to the robotic arm. The arm actuator is configured to move the robotic arm from a first position to a second position. The system further includes a conveyor assembly that includes a plurality of conveyor belts. Each conveyor belt is configured to move a plurality of parts including a first part and a second part. The system also includes a visual arrangement containing multiple cameras. Each camera is aligned on at least one of the conveyor belts. Therefore, each camera is configured to take a picture of at least one of the parts. The system also includes a system controller in electronic communication with the arm actuator and the multiple cameras and conveyor belts. The system controller is programmed to execute the following instructions: operating a robot arm to perform an action at a start position; Moving the robotic arm from the start position to a first position; Determining, from an image acquisition process method, whether a first part at the first position will be ready to undergo a first action by the robotic arm when the robotic arm reaches the first position; Commencing execution of the visual processing method to determine the positional deviation of the second portion from the second position and whether the second portion is ready to undergo a second action by the robotic arm when the robotic arm reaches the second position; and performing a first action on the first part using the robotic arm with the positional deviation of the first part from the first position predetermined by the image acquisition process method. The steps are repeated for the next robot mounting cycle, wherein the first position and action are newly provided as the new start position, the second part, the second position and the second action as the new first part, the new first position or the new first action will be redefined and the positional deviation of the second part will be redefined as the positional deviation of the new first part.

The system controller is also programmed to perform the image capture process process, an embedded process that operates continuously in the background. The image capture process method includes the steps of: identifying the next part from a designated location to be actioned; Checking if a camera is ready to capture the image of the identified part at the designated position; Checking that the identified part is in a position for taking pictures; Taking an image of the identified part from the intended position using a camera while the robot arm moves to the intended position; Determining a positional deviation of the identified part from the intended position based at least in part on the position of the identified part as the robot arm moves toward the intended position; Set the status of the identified part from the intended position ready to undergo action.

The foregoing features and advantages and other features and advantages of the present invention will be readily apparent from the following detailed description of the best modes for carrying out the invention when taken in conjunction with the accompanying drawings.

list of figures

• 1 Fig. 10 is a schematic block diagram of a system for assembling a device, which system may control a robot assembly;
• 2 FIG. 10 is a flow chart illustrating a method of controlling the robot assembly; FIG.
• 3 FIG. 10 is a flowchart illustrating a visual processing method which is part of the in 2 can be illustrated method; and
• 4 FIG. 4 is a flowchart illustrating a predictive optimizing procedure which is part of the in 2 can be illustrated method.

PRECISE DESCRIPTION

With reference to the figures, wherein like reference numerals correspond to like or similar components in the several views, there is illustrated 1 a schematic way a system 100 for assembling a device (eg a battery) that consists of several parts X consists. The system 100 includes a system controller 102 with at least one processor 104 and at least one memory 106 (or any suitable non-transient computer-readable mass storage medium). The processor 104 may be one or several combinations of application specific integrated circuits (ASICs), electronic circuits, and central processing units (eg, microprocessors). The memory 106 may be one or several combinations of a read only memory, a programmable read only memory, a random access memory, a hard disk drive, or any other suitable non-transitory computer readable mass storage medium. The processor 104 can execute executable instructions from a controller stored in the memory 106 are stored. The controller-executable instructions may be software or firmware programs or routines, combinatorial logic circuits, sequential logic circuits, input / output circuits and devices, suitable signal conditioning and buffering circuits, and other components that provide the functionality described in detail below becomes. It is considered that the system controller 102 the memory 106 may not contain, because the memory 106 outside the system controller 102 can be located.

The system 100 further comprises a robot assembly 101 with at least one robotic arm 108 and at least one arm actuator 110 , such as a motor, for moving the robot arm 108 , The robot arm 108 can be at least one elongated prop 112 include that with the arm actuator 110 is effectively coupled. For example, the elongated support 112 with the arm actuator 110 be mechanically connected. Therefore, the arm actuator 110 the robot arm 108 Press to the elongated support 112 in three degrees of freedom to move (eg to rotate or move). In addition to the elongated prop 112 includes the robot arm 108 at least one gripping organ 114 and a joint 116 which is the elongated prop 112 and the gripping organ 114 connects with each other. As used herein, the term "gripping member" refers to a tool that is capable of performing work on a part X be able to. As examples without limitation, the gripping member 114 a gripper, a drill or a welding tool. Consequently, the gripping member 114 for example, a part X grab, drill or weld. The joint 116 can have a further three degrees of freedom and therefore allows the gripping member 114 , relative to the elongated support 112 to move. Although the drawings have an arm-actor 110 show, the robot assembly 101 two or more actuators 110 contain. For example, an arm actuator 110 the elongated support 112 move and another arm actor 110 can the gripping organ 114 relative to the elongated prop 112 move.

The robot arm 108 is with the arm actuator 110 effectively coupled and therefore can the arm actuator 110 the robot arm 108 actuate. Upon actuation of the robot arm 108 can the elongated prop 112 move or the gripping organ 114 can do an action on the part X To run. For example, the arm actuator 110 the robot arm 108 press to the part X with the gripping organ 114 to take or let go. Alternatively or additionally, the arm actuator 110 the robot arm 108 Press to the gripping organ 114 relative to the elongated prop 112 to turn. The arm actor 110 can the robot arm 108 in response to an input or command from the system controller 102 actuate. As a result, the system controller is located 102 in communication, such as electronic communication, with the arm actuator 110 ,

The system controller 102 is also in communication (eg in electronic communication) with a conveyor arrangement 118 , The conveyor arrangement 118 can be part of the system 100 be and includes several conveyor belts 120 , which are each designed to the parts X to transport and move. The conveyor arrangement 118 also contains a conveyor actuator 122 , about a motor, with the conveyor belts 120 is effectively coupled. The conveyor actuator 122 One can be switched to the conveyor belts 120 to move. In addition to the conveyor actuator 122 includes the conveyor actuator 122 parts sensors 124 that with every conveyor belt 120 are effectively connected. Every part sensor 124 can be the presence or absence of a part X in a special place L along the conveyor belt 120 detect. In the illustrated embodiment, each part sensor 124 a photoelectric sensor, such as a sensor with a light-emitting diode (LED sensor), which is capable of the presence or absence of a part X in the special place L along the conveyor belt 120 to detect. Every part sensor 124 is in communication (eg electronic communication) with the system controller 102 , Consequently, the system controller 102 the presence or absence of a part X in the special place L based on the input from the part sensors 124 determine.

The system 100 further includes a visual or optical system 126 in communication (eg in an electronic communication) with the system controller 102 , The visual arrangement 126 can change the position and orientation of each part X on the conveyor belts 120 detect. In the illustrated embodiment, the visual arrangement includes 126 several cameras 128 for taking pictures of the parts X are able to. Every camera 128 is a conveyor belt 120 effectively assigned and therefore can take pictures of the parts X on the corresponding conveyor belts 120 take up. Every camera is special 128 on at least one conveyor belt 120 aligned so that every camera 128 is designed to take a picture of at least one of the parts X on the appropriate conveyor belt 120 take. The cameras 128 are in communication (eg in an electronic communication) with the system controller 102 , Therefore, the system controller 102 the operation of all cameras 128 he can coordinate and position the parts X on the conveyor belts 120 based at least in part on the pictures taken by the cameras 128 be recorded. The system controller 102 in turn can send commands to the robot assembly 101 based at least in part on the position and orientation of the parts X on the conveyor belts 120 send. The visual arrangement 126 can be a light source 130 included to the conveyor assembly 118 to illuminate and thereby minimize the effects of an ambient light or a lack thereof when the cameras 128 a picture of the parts X take up.

To improve the quality of the pictures taken by the cameras 128 can be recorded, certain characteristics of the visual arrangement 126 be tuned to detect errors, such as when a part X not moving as planned, and around the system 100 in response to such detected errors. For example, the luminous flux of the light source 130 adjusted to the above-discussed effects of ambient light on the cameras 128 to minimize. The system controller 102 can be programmed to have certain physical characteristics of the part X can identify. It is useful physical features of the part X to identify the system controller 102 in determining the position and orientation of the part X can support. In addition, the exposure time of the cameras 128 also be adjusted to improve the quality of the pictures taken by the cameras 128 be recorded. As used herein, the term "exposure time" refers to the amount of time an aperture of the camera is opened to allow light to be applied to the photographic sensor of the camera 128 falls. In addition, the contrast threshold of the visual arrangement 126 be adjusted to improve the quality of the pictures taken by the cameras 128 be recorded. In the present disclosure, the term "contrast" refers to a measure of the difference in brightness between light and dark areas in an image, and the term "contrast threshold" refers to the minimum perceived contrast of a camera.

Regarding 2 The present disclosure also relates to a method 200 for controlling the robot assembly 101 , In this process 200 can the cameras 128 a picture of a part X while the robot arm is picking up 108 moved to minimize the time needed to assemble the parts X necessary to completely assemble a device such as a battery. The procedure 200 starts with step 202 , In step 202 the system controller commands 102 the robot arm 108 over the arm actuator 110 to perform a startup action on a startup part. For example, the system controller 102 the robot arm 108 command, a start part X to suspend or remove. Alternatively, the system controller 102 in step 202 the robot arm 108 command the start part X to drill or weld. Specifically, the system controller 102 command that the gripping organ 114 Work on the start part X performs. While the gripping organ 114 Work on the start part X executes, is the robot arm 108 in a starting position. step 202 includes that robot arm 108 is pressed to a start action on a start part X in response to a command from the system controller 102 perform. After running the startup action on the startup part X sets the system controller 102 in step 202 an action marker F the start position is off, indicating that the next action at the start position can not be executed immediately, and at the same time, the starting position for a new image capture is from the visual processing method 300 ( 3 ) available when the same position of the robot arm 108 as the second Position is visited again. Next is the procedure 200 to step 204 further.

In step 204 the system controller commands 102 the robot arm 108 to move from a starting position to a first position. For this purpose, the system controller 102 a command signal (or a move command) to the arm actuator 110 send. In response to this command signal, the arm actuator becomes 110 pressed to the robot arm 108 to move from the starting position to the first position. Therefore, step includes 204 that the robot arm 108 is moved from the start position to a first position. Next is the procedure 200 to step 206 further.

In step 206 represents the system controller 102 Determine if the first part X1 will be ready, a first action by the robot arm 108 to be subjected as soon as the robot arm 108 reached the first position. The first action may be the same as or different from the previous one. For example, the first action may be to cancel the first part X1 , letting go of the first part X1 , drilling the first part X1 , the welding of the first part X1 or any other action taken by the end effector 114 at the first part X1 can be executed.

To determine if the first part X1 will be ready, a first action by the robot arm 108 to be subjected as soon as the robot arm 108 reaches the first position, the system controller is supported 102 on the visual or optical processing method or subroutine 300 ( 3 ), which will be described in detail below. The visual processing method 300 ( 3 ) can run continuously in the background while the system controller 102 the procedure 200 performs. By using the visual processing method 300 ( 3 ) may be the system controller 102 Use the action flag F of the first position to determine if the first part X1 will be ready, a first action by the robot arm 108 to be subjected to when the robot arm 108 reaches the first position, at least in part, based on an offset of the action position. In the present disclosure, the term "offset of the action position" means the positional deviation of a part X1 from the intended first position of the robot arm 108 that in the store 106 is stored and for the processor 104 as well as for the controller 102 is accessible. Thus, the "positional deviation of a part from a designated position" refers to the distance from the location of a part to a designated location in the memory 106 is stored. The offset of the action position can be determined by the visual processing method 300 ( 3 ) and the visual processing method 300 takes into account the position and orientation of a part X (eg the first part X1 or the second part X2 ) when the robot arm 108 moved to the action position. In step 206 represents the system controller 102 based on the offset of the action position of the first part X1 Determine if the first part X1 will be ready, a first action by the robot arm 108 to be subjected as soon as the robot arm 108 reached the first position. In other words, the system controller 102 based on the positional deviation of the first part X1 with respect to the intended first position of the robot arm 108 Determine if the first part X1 will be ready, a first action by the robot arm 108 to be subjected as soon as the robot arm 108 reached the first position. The position deviation of the first part X1 with respect to the intended first position of the robot arm 108 is based, at least in part, on the position and orientation of the first part X1 at the time the camera is turned on 108 a picture of the first part X1 has recorded. As discussed above, the camera takes 128 a picture of the first part X1 and process this while the robot arm 108 moved to the first position.

When the system controller 102 determines that the first part X1 will not be ready for a first action by the robotic arm 108 to be subjected to when the robot arm 108 reaches the first position, leads the system controller 102 step 206 (including the visual processing method 300 ) again until the system controller 102 determines that the first part X1 Will be ready, the first action by the robot arm 108 to be subjected to when the robot arm 108 reached the first position. If the first part X1 specifically, will not be positioned within the positional deviation (ie, the offset of the action position) using the visual processing method 300 was determined, then performs the procedure 200 step 206 again off. Before the system controller 102 step 206 again, he can use the visual processing method 300 and the robot arm 108 instruct to stop for a predetermined amount of time. If the first part X1 however, will be ready for a first action by the robotic arm 108 to be subjected as soon as the robot arm 108 reaches the first position, then goes the procedure 200 to step 208 further.

At step 208 the system controller starts 102 with the execution of the visual processing method 300 ( 3 ) in connection with a process of moving the robot arm 108 from the first position to a second position. After the execution of step 208 goes the procedure 200 to step 210 further.

At step 210 actuates the arm actuator 110 the robot arm 108 to the first action on the first part X1 perform when the robot arm 108 reached the first position. Therefore, step includes 210 in that a first action on the first part is made using the robotic arm 108 only executed when the first part X1 within the position deviation from the first position. As discussed above, the robotic arm guides 108 the first action on the first part X1 in response to a previous command from the system controller 102 out. The first action may include that the first part X1 is included that the first part X1 Let go of that first part X1 is drilled that the first part X1 is welded, or any other action, by the gripping organ 114 can be executed. After the robotic arm 108 the first action on the first part X1 has executed, sets the system controller 102 the action flag F of the first position to OFF, indicating that the next action can not be performed immediately at the first position and, at the same time, that the first position for a new image capture from the visual processing method 300 ( 3 ) is available when the same position of the robot arm 108 will be visited again. After performing step 210 increases the procedure 200 at step 212 the part counter by one and immediately returns to step 204 back. This time, however, the system controller commands 102 in step 204 the robot arm 108 to move from the first position or the newly provided start position to a second position which becomes the new first position. The remaining steps are repeated, taking into account that the robot arm 108 moved to the new first position, and it may be that the robot arm 108 must perform an action on a second part or a new first part when the robot arm 108 reached the second or the new first position.

By using the method 200 takes the system 100 a picture of a part X on (eg the first part X1 or the second part X2 ) while the robot arm 108 moved to a first position instead of waiting for the robot arm 108 reaches the second position, minimizing the cycle time of the manufacturing process. Using the captured image of the part X can the system controller 102 determine if the second part X2 Will be ready for a second action by the robot arm 108 to be subjected as soon as the robot arm 108 reached the second position. However, this finding takes place before the robotic arm 108 reached the second position. Therefore, the process can 200 also be referred to as a proactive method. During the procedure 200 and 300 move the conveyor belts 120 continuously the parts X (eg the first and second part X1 . X2 ) relative to the robot arm 108 , It is therefore considered that the procedure 200 also the step of moving the parts X relative to the robot arm 108 using one or more conveyor belts 120 may include.

Regarding 3 As discussed above, the system controller may be 102 a visual processing method 300 Run to see if that part X Will be ready for action by the robotic arm 108 to be subjected as soon as the robot arm 108 reached the intended position. The visual processing method 300 is continuously running in the background while the process 200 is performed.

The visual processing method 300 starts with step 302 , In step 302 the system controller determines 102 via the parts counter in step 212 , which part X (eg the first part X1 or the second part X2 ) action by the robot arm 108 should be subjected. As examples without limitation, the system controller determines 102 in step 302 , which part X from the robot arm 108 grabbed, released, drilled or welded. The part X , an action by the robotic arm 108 may be referred to as the identified part. Next is the system controller 102 with step 304 continued.

In step 304 represents the system controller 102 determine if the camera 128 taking a picture of the part X can record that in step 302 has been identified (ie, the identified part) available to capture an image (or video) of the identified part. If the camera 128 for example, can not take an image of the identified part because it is currently taking an image of another part or an action is taking place at the camera position, the action flag F is in the ON state, then sets the system controller 102 notice that the camera 128 is not available, and the step 304 is repeated until the camera 128 is available. If the camera 128 is available, the action marker F is in the OFF state, then the procedure continues 300 with step 306 continued.

In step 306 represents the system controller 102 determines whether the identified part is in a position to capture an image of that part (ie the intended part position). This determination may be based, at least in part, on an input from at least one of the parts sensors 124 based. If the identified part is not in the intended part position and the camera 128 therefore, can not take an image of the identified part, then the procedure repeats 300 step 306 , At this time the conveyor belt is moving 120 the identified part relative to the robotic arm 108 to move. When the identified part is in the intended part position and the camera 128 therefore a picture of can pick up the identified part, then the procedure goes 300 to step 308 further.

In step 308 the system controller receives 102 an image of the identified part, previously from the camera 128 has been recorded. As above with reference to step 206 The camera can be discussed 128 an image of the identified part X (eg the first part X1 or the second part X2 ) while the robot arm 108 relative to the conveyor belts 20 emotional. After receiving the image of the identified part X goes the procedure 300 to step 310 further.

In step 310 the system controller determines 102 an offset of the action position. As used herein, the term "offset of the action position" means the positional deviation of the identified part (eg, the first part X1 or the second part X2 ) from its intended position (eg, a second position) of the robotic arm 108 , Specifically, the system controller 102 the offset of the action position (ie the position deviation from the intended position of the robot arm 108 ) based at least in part on the input of the information obtained from the captured image of the identified part X (eg the first part X1 or the second part X2 ) were won. Then the procedure continues 300 with step 312 continued.

In step 312 sets the system controller 102 the action marker F based at least in part on an offset of the action position to ON, indicating that the part X Will be ready for action by the robotic arm 108 to be subjected to when the robot arm 108 reached the intended position. Both the action marker F as well as the specific offset of the action position are sent to the other routine (ie to the procedure 200 ), so that the system controller 102 can determine if the identified part X Will be ready for action by the robotic arm 108 to be subjected as soon as the robot arm 108 the intended position in step 208 reached.

Regarding 4 The present disclosure also relates to a method 400 to determine how many predictive actions can be performed while the robot arm 108 moved from a first position to a second position. As used herein, the term "anticipatory action" means capturing an image of a part X (eg the first part X1 or the second part X2 ), while the robot arm 108 moved and before the robotic arm 108 reached his target position. In other words, the process can 400 used to determine how many pictures of one or more parts of the cameras 128 can be recorded while the robot arm 108 moved from a first position to the second position. The procedure 400 Therefore, it may be referred to as a predictive optimization method, and may be used as part of a batch process, for example. As used herein, the term "batch process" refers to a process of stacking multiple parts together. The procedure 400 starts at step 402 ,

In step 402 the system controller determines 102 the time span that the robot arm 108 needed to take a first action (eg picking up, releasing, drilling, welding) on a first part X1 execute (ie the time of the first action). Consequently, the term "time of the first action" means the time required for the robot arm 108 the first action on a first part X1 performs. The time of the first action may be in the memory 106 of the system controller 102 get saved. step 402 comprising determining the period of time required by the robotic arm 108 needed to get the first action on the first part X1 complete. Then the procedure goes 400 to step 404 further.

In step 404 the system controller determines 102 the time span that the robot arm 108 required to travel from the first position to the second position along a programmed path while moving at a constant or variable speed (ie the time of the travel). The term "travel time" means the period of time that the robot arm 108 required to travel from the first position to the second position while moving along a programmed path at a constant or variable speed. step 404 includes determining a period of time that the robotic arm 108 needed to drive from the first position to the second position. The time of travel can be in the memory 106 of the system controller 102 get saved. After the system controller 102 has determined the time of travel, the procedure goes 400 to step 406 further.

In step 406 the system controller determines 102 the time span that the robot arm 108 needed to do a second action (eg picking up, releasing, drilling, welding) on a second part X2 execute (ie the time of the second action). Consequently, the term "time of the second action" means the period of time that the robot arm 108 needed to do the second action on the second part X2 perform. step 406 includes determining the period of time that the robotic arm 108 needed to complete the second action on the second part. The time of the second action may be in the memory 106 of the system controller 102 get saved. Then the procedure goes 400 to step 408 further.

In step 408 the system controller determines 102 the total number of repetitive motion sequences that the robot arm 108 must execute to complete a cycle in accordance with the instructions contained in the memory 106 stored (ie the total number of repetitive motion sequences). In the present disclosure, the term "total number of repeating motion sequences" refers to the total number of repeating motion sequences that the robot arm 108 to complete a cycle. The total number of repeating motion sequences may be in the memory 106 get saved. Next is the procedure 400 with step 410 continued.

In step 410 the system controller determines 102 the total number of predictive actions required to complete a cycle in accordance with the instructions contained in the memory 106 stored (ie the total number of predictive actions). As discussed above, the term "anticipatory action" means taking an image of a part X (eg the first part X1 or the second part X2 ), while the robot arm 108 emotional. Consequently, step includes 410 in that the total number of images needed to complete the cycle is determined. Specifically, the system controller 102 determine the total number of predictive actions by setting the total number of predictive actions equal to the total number of repetitive motion sequences generated in step 408 was determined. For example, if one cycle has five respective motion sequences of the robotic arm 108 needed, then determines the system controller 102 in that the total number of predictive actions is also five. Next is the procedure 400 to step 412 further.

In step 412 the system controller determines 102 the number of possible predictive actions while the robot arm 108 from the first position to the second position (ie the number of possible predictive actions along an arm travel). As used herein, the term "arm travel" means the path taken by the robotic arm 108 passes through when moving from a first position to a second position. To determine the number of possible predictive actions along the arm travel, the system controller subtracts 102 one of the value of the total number of predictive actions in step 410 was determined. If the total number of predictive actions (the ones in step 410 was determined), for example, is five, then subtracts the system controller 102 one in five and determines that the total number of possible predictive actions while the robot arm 108 from the first position to the second position is four. After determining the number of possible predictive actions, the procedure continues 400 with step 414 continued.

In step 414 represents the system controller 102 determines if the number of possible predictive actions along the arm travel is zero. If the number of possible predictive actions along the arm travel is zero, the procedure continues 400 with step 416 continues, where the system controller 102 set the number of possible predictive actions to one. After changing the number of possible predictive actions to one, the procedure goes 400 to step 418 further. On the other hand, if the number of possible predictive actions is not zero, then the procedure goes 400 directly to step 418 further.

In step 418 the system controller determines 102 the amount of time the cameras take 128 require to perform all possible predictive actions based at least in part on the number of possible predictive actions that are involved in step 412 or 416 has been determined (ie the total time of possible proactive actions). The "total time of possible anticipatory actions" denotes the time taken by the cameras 128 to perform all possible predictive actions (ie, capturing sub-images) in step 412 or 416 were determined. Next is the system controller 102 with step 420 continued.

wobei:

• ∑_nT(image_i) die Zeitspanne ist, welche die Kameras 128 benötigen, um alle möglichen vorausschauenden Aktionen auszuführen, zumindest teilweise beruhend auf der Anzahl der möglichen vorausschauenden Aktionen, die in Schritt 412 oder 416 bestimmt wurde;
• T(A → B) die Zeitspanne ist, die der Roboterarm 108 benötigt, um von der ersten Position zu der zweiten Position entlang eines programmierten Wegs zu fahren, während er sich mit einer konstanten oder variablen Geschwindigkeit bewegt (d.h. die Zeit des Verfahrwegs), die in Schritt 404 bestimmt wurde;

In step 420 represents the system controller 102 Determine if all the possible predictive actions that were previously in step 412 or 416 have been determined, can actually be performed while the robot arm 108 moves from the first position to the second position. For this purpose, the system controller compares 102 the total time of the possible anticipatory actions with the sum of the following items: (a) the time of the first action, which in step 402 was determined; (b) the time of travel in step 404 was determined; and (c) the time of the second action, which in step 406 was determined. For example, the system controller 102 to execute the step 420 calculate the following equation: $${\displaystyle {\Sigma }_{n}T\left(imaG{e}_i\right)>T\left(A\to B\right)+P\left(A\right)+P\left(B\right)}$$

in which:

• Σ _n T (image _i ) is the time span which the cameras use 128 need to perform all possible predictive actions, based at least in part on the number of possible predictive actions that are involved in step 412 or 416 was determined;
• T (A → B) is the time span that the robot arm 108 is required to travel from the first position to the second position along a programmed path while moving at a constant or variable speed (ie, the time of the travel) shown in step 404 was determined;

P (A) is the time span that the robot arm 108 needed to do a first action (eg picking up, releasing, drilling, welding) on a first part X1 to execute (ie the time of the first action), in step 402 was determined; and P (B) is the time span that the robot arm 108 needed to do a second action (eg picking up, releasing, drilling, welding) on the second part X2 to execute (ie the time of the second action), in step 406 was determined. The process of determining the total possible predictive actions is not necessarily performed in real time. The denominations can be estimated offline. In addition, the action time and the travel time of a single step are not necessarily the same.

If the total time of the possible predictive actions is not greater than the sum of (a) the time of the first action, which in step 402 was determined; (b) the time of travel in step 404 was determined; and (c) the time of the second action, which in step 406 was determined, then the procedure moves 400 with step 422 continued. In step 422 the system controller commands 102 that the cameras 128 perform the number of possible predictive actions (ie images of the parts X record) as in step 412 or 416 was determined while the robot arm 108 moves from the first position to the second position. step 422 also includes performing the possible predictive actions described in step 412 or 416 were determined, using the cameras 128 ,

When in step 420 the total time of the possible predictive actions is also greater than the sum of (a) the time of the first action, which is in step 402 was determined; (b) the time of travel in step 404 was determined; and (c) the time of the second action, which in step 406 was determined, then the process returns 400 to step 412 back, where the system controller 102 the number of possible predictive actions as discussed above re-adjusted. In determining how many pictures can be taken while the robot arm 108 moved from the first position to the second position, the method takes into account 400 the image processing time; the special action of the robot arm 108 ; the acceleration, deceleration and speed of the robot arm 108 ; and the operation of the conveyor belts 120 ,

Although the best modes for carrying out the invention have been described in detail, those familiar with the art to which this invention relates will recognize various alternative constructions and embodiments for practicing the invention within the scope of the appended claims.

Claims (5)

A method (200) for controlling a robot assembly (101), comprising: operating a robotic arm (108) to perform a start action at a start position; moving the robotic arm (108) from the start position to a first position; performing, via a controller (102), a visual processing subroutine (300), the visual processing subroutine (300) comprising: determining whether a first portion (X1) will be ready undergoing a first action by the robotic arm (108) become as soon as the robot arm (108) reaches the first position; a positional deviation of a second part (X2) from a second position is determined; determining whether the second part (X2) will be ready to undergo a second action by the robotic arm (108) as soon as the robotic arm (108) reaches the second position; and the first action is performed on the first part (X1) using the robot arm (108) when the robot arm (108) is located within a positional deviation of the first part (X1) from the first position indicated by the visual processing subroutine (300 ) was predetermined; wherein the steps of operating, moving, executing and performing are repeated, wherein the first position is newly provided as the start position, the new action is newly provided as the start action, the second part (X2) is newly provided as the first part (X1) the second position is newly provided as the first position, the second action is newly provided as a first action, and the positional deviation of the second part (X2) from the second position is newly provided as the positional deviation of the first part (X1) from the first position ; wherein the visual processing subroutine (300) is an embedded process that operates continuously in the background; wherein the visual processing subroutine (300) comprises identifying a next portion of a designated location to be subjected to a next action; determining whether a camera (128) is ready to capture an image of the identified portion from the designated location; and that it is determined if that is the identified part is located in a position where the camera (128) is able to capture the image of the identified part; and wherein determining whether the identified part is in a position where the camera (128) is capable of capturing the image of the identified part is based on an input from at least one part sensor (124). Method (200) according to Claim 1 wherein the visual processing subroutine (300) further comprises taking the image of the identified part from the intended position using the camera (128) while the robot arm (108) is moving to the intended position. Method (200) according to Claim 2 wherein a positional deviation of the identified part from the intended position is determined based at least in part on the position of the identified part as the robot arm (108) moves toward the intended position. Method (200) according to Claim 3 wherein a status of the identified part is set from the intended position ready to be subjected to the next action. Method (200) according to Claim 1 method of further comprising determining how many images can be captured by one or more portions of cameras (128) as the robotic arm (108) moves from the start position to the first position.

Images