JP6522488B2 - Machine learning apparatus, robot system and machine learning method for learning work taking-out operation - Google Patents

Description

  The present invention relates to a machine learning device, a robot system, and a machine learning method for learning the taking-out operation of a randomly placed work including a bulk-stacked state.

  DESCRIPTION OF RELATED ART Conventionally, the robot system which hold | grips and conveys the workpiece | work piled up in the cage-like box, for example with the hand part of a robot is known (for example, refer patent document 1, 2). In such a robot system, for example, position information of a plurality of workpieces is acquired using a three-dimensional measuring device installed above a cage-like box, and the workpieces are obtained one by one based on the position information. It is taken out by the hand part.

Patent No. 5642738 gazette Patent No. 5670397 gazette

  However, in the above-described conventional robot system, for example, how to extract a workpiece to be taken out from distance images of a plurality of workpieces measured by a three-dimensional measuring device and which position of the workpiece should be taken out in advance It needs to be set to Also, when taking out the work, it is necessary to program in advance how to operate the hand portion of the robot. Specifically, for example, it is necessary for a human to teach the robot an operation of taking out a work using a teaching pendant.

  Therefore, if the setting for extracting the workpiece to be taken out is not appropriate from the distance images of the plurality of workpieces, or the robot operation program is not properly created, the success rate when the robot takes out and transports the workpiece decreases. Moreover, in order to increase the success rate, it is necessary to improve the detection setting of the work and the operation program of the robot while searching for the optimum operation of the robot by trial and error.

  Therefore, an object of the present invention is to provide a machine learning apparatus capable of learning an optimal operation of a robot when taking out a randomly placed work, including a bulk-stacked state, without human intervention, in consideration of the above-mentioned situation. , Robot system and machine learning method.

According to the first configuration example of the first embodiment of the present invention, the machine learning device learns the operation of the robot for taking out the workpiece by the hand unit from the plurality of workpieces placed in randomness, including the bulk-stacked state Output data of a three-dimensional measuring device that measures a plurality of workpieces stacked in bulk and outputs three-dimensional position information of the surface of the plurality of workpieces, and the operation of taking out the workpiece by the hand unit A state quantity observation unit that observes a state quantity that sets the position, posture, and take-out direction of each, an operation result acquisition unit that obtains a result of the take-out operation of the robot that takes out the workpiece by the hand unit, and the state quantity observation Receiving the determination result of the state quantity observed by the unit and the success or failure of the extraction operation of the robot acquired by the operation result acquisition unit A learning unit for learning the take-out operation of the work, and the learning unit calculates a reward based on the determination result of success or failure of the take-out of the work from the operation result acquisition unit; A value function updating unit that has a value function that determines the value of the retrieval operation of the work, and updates the value function according to the reward, and the optimum of the hand unit obtained by the learning unit The machine learning which carries out the learning which updates the state variable which sets position, posture, and taking-out direction based on the value function, and operates the drive unit of the hand unit and the robot based on the updated state variable. An apparatus is provided. According to the second configuration example of the first embodiment of the present invention, the machine learning device learns the operation of the robot for taking out the workpiece by the hand unit from the plurality of workpieces placed in randomness, including the bulk-stacked state Output data of a three-dimensional measuring device that measures a plurality of workpieces stacked in bulk and outputs three-dimensional position information of the surface of the plurality of workpieces, and the operation of taking out the workpiece by the hand unit A state quantity observation unit that observes a state quantity that sets the position, posture, and take-out direction of each, an operation result acquisition unit that obtains a result of the take-out operation of the robot that takes out the workpiece by the hand unit, and the state quantity observation Receiving the determination result of the state quantity observed by the unit and the success or failure of the extraction operation of the robot acquired by the operation result acquisition unit A learning unit for learning the take-out operation of the work, the learning unit implementing a learning model for learning the take-out operation of the work, and a label output from the operation result acquisition unit; An error calculation unit that calculates an error based on the output of the learning model implemented in the learning unit; and a learning model updating unit that updates the learning model according to the error, obtained by the learning unit There is provided a machine learning device for operating the drive axes of the hand unit and the robot based on state variables which respectively set the optimum position, posture and take-out direction of the hand unit. The machine learning apparatus further instructs the robot to take out the workpiece based on the state variable which sets the optimum position, posture, and take-out direction of the hand unit obtained by the learning unit. Preferably, a decision making unit is provided to determine the data.

  According to a second embodiment of the present invention, there is provided a machine learning device for learning an operation of a robot for taking out a workpiece from a plurality of randomly placed workpieces including a bulked state by a hand unit, A state quantity observing unit for observing a state quantity of the robot including output data of a three-dimensional measuring instrument for measuring a three-dimensional map for each work, and a result of a taking-out operation of the robot for taking out the work by the hand unit An operation result acquisition unit, an output from the state amount observation unit, and an output from the operation result acquisition unit, and an operation amount including a measurement parameter of the three-dimensional measuring instrument is the state amount of the robot and the extraction operation. And a learning unit for learning in association with the result of the above. It is preferable that the machine learning apparatus further includes a decision making unit which determines the measurement parameter of the three-dimensional measuring device with reference to the operation amount learned by the learning unit.

  The state quantity observation unit can also observe a state quantity of the robot including output data of a coordinate calculation unit that calculates the three-dimensional position of each work based on the output of the three-dimensional measuring device. The coordinate calculation unit may further calculate an attitude of each work and output data of the calculated three-dimensional position and attitude of each work. The operation result acquisition unit can use output data of the three-dimensional measuring device. The machine learning apparatus further includes a pre-processing unit that processes output data of the three-dimensional measuring device before input to the state amount observation unit, and the state amount observation unit outputs output data of the pre-processing unit. Preferably, it is received as a state quantity of the robot. The pre-processing unit can uniformly align the direction and height of each workpiece in the output data of the three-dimensional measuring device. The operation result acquisition unit can acquire at least one of success or failure of taking out the work, a damaged state of the work, and an achievement degree when the taken out work is passed to a post-process.

  The learning unit has a reward calculation unit that calculates a reward based on the output of the operation result acquisition unit, and a value function that determines the value of the retrieval operation of the work, and updates the value function according to the remuneration And a value function updating unit. The learning unit has a learning model that learns the taking-out operation of the work, and an error calculating unit that calculates an error based on an output of the operation result acquiring unit and an output of the learning model; And a learning model updating unit that updates the learning model accordingly. The machine learning device preferably comprises a neural network.

  According to a third embodiment of the present invention, there is provided a machine learning device for learning an operation of a robot for taking out a workpiece from a plurality of randomly placed workpieces including a bulked state by a hand unit, A state quantity observing unit for observing a state quantity of the robot including output data of a three-dimensional measuring instrument for measuring a three-dimensional map for each work, and a result of a taking-out operation of the robot for taking out the work by the hand unit An operation amount acquiring unit, an output from the state quantity observing unit, and an output from the operation result acquiring unit, and an operation amount including command data for instructing the robot to take out the workpiece from the robot A robot system comprising a machine learning apparatus comprising: a learning unit that learns in association with a state quantity and a result of the extraction operation , And the robot, and the three-dimensional measuring instrument, a robot system comprising a control device, the controlling the robot and the three-dimensional measuring instrument, respectively is provided.

  According to a fourth embodiment of the present invention, there is provided a machine learning device that learns the operation of a robot that takes out workpieces by a hand unit from a plurality of randomly placed workpieces, including bulk-stacked states, A state quantity observing unit for observing a state quantity of the robot including output data of a three-dimensional measuring instrument for measuring a three-dimensional map for each work, and a result of a taking-out operation of the robot for taking out the work by the hand unit An operation result acquisition unit, an output from the state amount observation unit, and an output from the operation result acquisition unit, and an operation amount including a measurement parameter of the three-dimensional measuring instrument is the state amount of the robot and the extraction operation. A robot system provided with a machine learning apparatus comprising a learning unit for learning in association with the result of And measuring device, a robot system and a control device for controlling each said robot and said three-dimensional measuring device is provided.

  The robot system includes a plurality of the robots, the machine learning device is provided for each of the robots, and the plurality of machine learning devices provided in the plurality of robots mutually communicate data via a communication medium. Are preferably shared or exchanged. The machine learning device may reside on a cloud server.

  According to a fifth embodiment of the present invention, there is provided a machine learning method for learning an operation of a robot for taking out a workpiece from a plurality of randomly placed workpieces including a bulked state by a hand unit, The state quantity of the robot including the output data of the three-dimensional measuring instrument which measures the three-dimensional map of each work is observed, the result of the taking-out operation of the robot taking out the work by the hand unit is obtained, and the state quantity observation Receiving an output from the unit and an output from the operation result acquisition unit, and associating an operation amount including command data for instructing the robot to take out the workpiece from the state quantity of the robot and the result of the take-out operation A machine learning method is provided.

  According to the machine learning device, the robot system, and the machine learning method according to the present invention, it is possible to learn the optimum operation of the robot when taking out the work placed in a mess including the bulked state without human intervention. Play an effect.

FIG. 1 is a block diagram showing a conceptual configuration of a robot system according to an embodiment of the present invention. FIG. 2 is a view schematically showing a model of a neuron. FIG. 3 is a view schematically showing a three-layered neural network configured by combining the neurons shown in FIG. FIG. 4 is a flow chart showing an example of the operation of the machine learning device shown in FIG. FIG. 5 is a block diagram showing a conceptual configuration of a robot system according to another embodiment of the present invention. FIG. 6 is a diagram for explaining an example of processing of a preprocessing unit in the robot system shown in FIG. FIG. 7 is a block diagram showing a modification of the robot system shown in FIG.

  Hereinafter, embodiments of a machine learning device, a robot system and a machine learning method according to the present invention will be described in detail with reference to the attached drawings. Here, in each drawing, the same reference numerals are given to the same members. Further, in the drawings, the same reference numerals are used to mean components having the same functions. In addition, in order to facilitate understanding, the drawings are appropriately scaled.

  FIG. 1 is a block diagram showing a conceptual configuration of a robot system according to an embodiment of the present invention. The robot system 10 according to this embodiment includes a robot 14 attached with a hand unit 13 for gripping the workpieces 12 stacked in a cage 11 and a three-dimensional measuring device for measuring a three-dimensional map of the surface of the workpiece 12 And a controller 16 for controlling the robot 14 and the three-dimensional measuring instrument 15, a coordinate calculation unit 19, and a machine learning device 20.

  Here, the machine learning apparatus 20 includes a state quantity observation unit 21, an operation result acquisition unit 26, a learning unit 22, and a decision making unit 25. The machine learning apparatus 20 learns and outputs an operation amount such as command data for instructing the robot 14 to take out the workpiece 12 or a measurement parameter of the three-dimensional measuring instrument 15 as described in detail later.

  The robot 14 is, for example, a six-axis articulated robot, and drive axes of the robot 14 and the hand unit 13 are controlled by the controller 16. The robot 14 is also used to take out the works 12 one by one from the box 11 installed at a predetermined position and sequentially move them to a specified place, for example, a conveyor or a work table (not shown).

  By the way, when the bulked workpieces 12 are taken out of the box 11, the hand portion 13 or the workpieces 12 may collide with or contact the wall of the box 11. Alternatively, the hand unit 13 or the work 12 may be caught on another work 12. In such a case, it is necessary to have a function of detecting the force acting on the hand unit 13 so that an overload applied to the robot 14 can be immediately avoided. Therefore, a six-axis force sensor 17 is provided between the tip of the arm of the robot 14 and the hand 13. Further, the robot system 10 according to the present embodiment also has a function of estimating the force acting on the hand unit 13 based on the current value of a motor (not shown) that drives the drive shaft of each joint of the robot 14 .

  Furthermore, since the force sensor 17 can detect the force acting on the hand portion 13, it can also determine whether the hand portion 13 is actually gripping the work 12. That is, since the weight of the work 12 acts on the hand 13 when the hand 13 grips the work 12, the detection value of the force sensor 17 exceeds the predetermined threshold after the work 12 is taken out. Then, it can be determined that the hand unit 13 grips the work 12. In addition, about judgment of whether the hand part 13 is holding the workpiece 12, for example, the imaging data of the camera used for the three-dimensional measuring instrument 15, the photoelectric sensor etc. which is not shown attached to the hand part 13 etc. It can also be judged by the output. Alternatively, the determination may be made based on data of a pressure gauge of a suction type hand described later.

  Here, the hand unit 13 may have various forms as long as the work 12 can be held. For example, the hand unit 13 has a form in which the work 12 is gripped by opening and closing two or more claws, or an electromagnet or negative pressure generating device that generates a suction force on the work 12. May be That is, in FIG. 1, although the hand part 13 is drawn as what grasps a work by two nail parts, it is needless to say that it is not limited to this.

  The three-dimensional measuring instrument 15 is provided at a predetermined position above the plurality of workpieces 12 by the support 18 in order to measure the plurality of workpieces 12. As the three-dimensional measuring instrument 15, for example, using a three-dimensional visual sensor that acquires three-dimensional position information by performing image processing on image data of the workpiece 12 captured by two cameras (not shown) it can. Specifically, three-dimensional maps (a plurality of workpieces stacked in bulk are applied by applying trigonometry, light cutting, time-of-flight, depth from Defocus, or a method using both of them. The position of the surface of 12) is measured.

The coordinate calculation unit 19 calculates (measures) the position of the surface of the plurality of workpieces 12 stacked in bulk, with the three-dimensional map obtained by the three-dimensional measuring instrument 15 as an input. That is, using the output of the three-dimensional measuring instrument 15, three-dimensional position data (x, y, z) for each work 12 or three-dimensional position data (x, y, z) and posture data (w) , P, r) can be obtained. Here, the state quantity monitoring unit 21 may receive both the three-dimensional map and the coordinate calculating portion 1 9 or these location data from the three-dimensional measuring device 15 (orientation data) is observing the state quantity of the robot 14 However, for example, only the three-dimensional map from the three-dimensional measuring instrument 15 can be received to observe the state quantity of the robot 14. In addition, as described later with reference to FIG. 5, the preprocessing unit 50 is added, and the preprocessing unit 50 generates a third order from the three-dimensional measuring instrument 15 before input to the state quantity observing unit 21. It is also possible to process (preprocess) the source map and input it to the state quantity observation unit 21.

  The correlation position between the robot 14 and the three-dimensional measuring instrument 15 is assumed to be determined in advance by calibration. Further, in the three-dimensional measuring device 15 of the present invention, a laser distance measuring device can be used instead of the three-dimensional visual sensor. That is, by measuring the distance from the position where the three-dimensional measuring instrument 15 is installed to the surface of each work 12 by laser scanning, or by using various sensors such as a single-eye camera, a tactile sensor, etc. The three-dimensional position data and posture (x, y, z, w, p, r) of the work 12 may be acquired.

  That is, in the present invention, for example, as long as data (x, y, z, w, p, r) of each work 12 can be acquired, any three-dimensional measuring instrument 15 to which any three-dimensional measuring method is applied is applied. be able to. Moreover, the aspect in which the three-dimensional measuring device 15 is installed is not specifically limited, either, For example, you may be fixed to a floor, a wall, etc., and may be attached to the arm part etc. of the robot 14.

  The three-dimensional measuring device 15 acquires a three-dimensional map of the plurality of works 12 stacked in bulk in the box 11 according to a command from the control device 16, and the coordinate calculation unit 19 calculates a plurality of three-dimensional maps based on the three-dimensional map. Data of the three-dimensional position (posture) of the work 12 is acquired (calculated), and the data is output to the control device 16 and the state quantity observation unit 21 and the operation result acquisition unit 26 of the machine learning device 20 described later. It has become. In particular, in the coordinate calculation unit 19, for example, the boundary between a certain work 12 and another work 12 and the boundary between a work 12 and a box 11 are estimated based on image data of a plurality of photographed works 12 , Data of the three-dimensional position of each work 12 is acquired.

  The data of the three-dimensional position of each work 12 is obtained, for example, by estimating the existing position and the holdable position of each work 12 from the positions of a plurality of points on the surfaces of the plurality of works 12 stacked in bulk. Refers to the data that has been Of course, the data of the three-dimensional position of each work 12 may include the data of the attitude of the work 12.

  Furthermore, the acquisition of the three-dimensional position and posture data for each work 12 in the coordinate calculation unit 19 also includes using a machine learning method. For example, it is also possible to apply object recognition or angle estimation from an input image or a laser distance measuring device using a method such as supervised learning described later.

  Then, when data of the three-dimensional position of each work 12 is input from the three-dimensional measuring instrument 15 to the control device 16 via the coordinate calculation unit 19, the control device 16 takes out the work 12 from the box 11. Control 13 operations. At this time, based on the command value (operation amount) corresponding to the optimal position, posture, and take-out direction of the hand unit 13 obtained by the machine learning device 20 described later, Not shown) is driven.

  Further, the machine learning device 20 is a variable of the photographing condition of the camera used for the three-dimensional measuring device 15 (measurement parameter of the three-dimensional measuring device 15: for example, exposure time adjusted at the time of photographing using It is also possible to learn the illuminance of the illumination system that illuminates the object, and control the three-dimensional measuring instrument 15 via the control device 16 based on the learned measurement parameter operation amount. Here, the variables of the position / posture estimation condition used by the three-dimensional measuring device 15 to estimate the existing position / posture of each workpiece 12 and the position / posture that can be held from the measured positions of the plurality of workpieces 12 are , And may be included in the output data of the three-dimensional measuring instrument 15 described above.

  Furthermore, the output data from the three-dimensional measuring instrument 15 is processed in advance by the pre-processing unit 50 or the like described in detail later with reference to FIG. 5, and the processed data (image data) is measured by the state quantity observation unit 21 As described above, it is also possible to give to. The operation result acquisition unit 26 can acquire, for example, the result obtained by taking out the work 12 by the hand unit 13 of the robot 14 from the output data from the three-dimensional measuring instrument 15 (output data of the coordinate calculation unit 19). In addition to that, for example, the achievement when the taken out work 12 is passed to the post process, and the operation result such as whether or not there is a change in state such as breakage of the taken out work 12, other means (eg, It goes without saying that the information can also be acquired via a camera, a sensor or the like provided in the post process. In the above, the state quantity observation unit 21 and the operation result acquisition unit 26 are functional blocks, and it is needless to say that one block can be regarded as achieving the functions of both.

  Next, the machine learning apparatus 20 shown in FIG. 1 will be described in detail. The machine learning apparatus 20 extracts useful rules, knowledge expressions, judgment criteria and the like from the collection of data input to the apparatus by analysis, outputs the judgment results, and learns knowledge (machine learning ) Has a function to The methods of machine learning are various, but roughly classified, for example, into "supervised learning", "unsupervised learning" and "reinforcement learning". Furthermore, in order to realize these methods, there is a method called "Deep Learning" in which extraction of the feature amount itself is learned. Note that these machine learning (machine learning device 20) may use a general-purpose computer or processor, but when GPGPU (General-Purpose computing on Graphics Processing Units), large-scale PC clusters, etc. are applied, it is faster. It is possible to process.

  First, supervised learning is a model in which a large number of input and result (label) data sets are given to the machine learning apparatus 20 to learn features in those data sets and to estimate the results from the input. That is, the relationship is acquired inductively. When this supervised learning is applied to the present embodiment, it can be used, for example, as a part that estimates a work position from sensor input, or a part that estimates an acquisition success probability for a work candidate. For example, it can be realized using an algorithm such as a neural network described later.

In addition, unsupervised learning is to give only a large amount of input data to the learning device, thereby learning how the input data has a distribution, and for the input data without providing the corresponding teacher output data. This is a method of learning with a device that performs compression, classification, shaping, etc. For example, features in those data sets can be clustered together, and so on. Using this result, by performing the allocation of the output so as to optimize it provided some criteria, it is possible to realize the prediction of output.

  In addition, as an intermediate problem setting between unsupervised learning and supervised learning, there is also one called semi-supervised learning, and for example, there is only a partial set of input and output data, and others It corresponds to the case of input only data. In the present embodiment, learning can be efficiently performed by using data (image data, simulation data, etc.) that can be acquired without actually moving the robot for unsupervised learning. .

Next, reinforcement learning will be described. First, consider the following as the problem setting of reinforcement learning.
The robot observes the state of the environment and decides the action.
The environment changes in accordance with some rules, and in addition, one's own actions may change the environment.
・ The reward signal comes back each time you take action.
・ What I would like to maximize is the sum of (discounted) rewards over the future.
• Learning starts from a state in which you do not know at all, or only incompletely, the consequences of the action. That is, the robot can obtain the result as data only after actually acting. In other words, it is necessary to search for the optimal action while trying and erroring.
-It is also possible to start learning from a good start point, with the pre-learned (the above-mentioned supervised learning and reverse reinforcement learning method) state as the initial state so as to imitate human motion.

  Reinforcement learning is not limited to judgment and classification, but also learning behavior to learn appropriate behavior based on the interaction given to the environment, that is, to maximize future rewards. Learn how to learn This means that, in the present embodiment, it is possible to acquire an action that affects the future, such as, for example, breaking the pile of the work 12 and making it easy to take the work 12 in the future. In the following, as an example, the explanation will be continued in the case of Q learning, but it is not limited to Q learning.

  Q learning is a method of learning a value Q (s, a) for selecting an action a under a certain environmental condition s. That is, in a certain state s, the highest action a of the value Q (s, a) may be selected as the optimum action. However, at first, the correct value of the value Q (s, a) is not known at all for the combination of the state s and the action a. Therefore, the agent (action entity) selects various actions a under a certain state s, and a reward is given to the action a at that time. Thereby, the agent learns the choice of the better action, that is, the correct value Q (s, a).

Furthermore, since we want to maximize the total rewards obtained in the future as a result of action, we aim to make Q (s, a) = E [[(γ ^t ) r _t ] finally. Here, E [] represents an expected value, t is time, γ is a parameter called a discount rate to be described later, r _t is a reward at time t, and Σ is a total at time t. The expected value in this equation is taken about when the state changes according to the optimal action, which is unknown and will be learned while searching. An update equation of such a value Q (s, a) can be represented, for example, by the following equation (1).

In the above formula (1), s _t represents the state of the environment at time t, a _t represents the action at time t. By the action a _t, the state changes to s _{t + 1.} r _{t + 1} represents the reward obtained by the change of the state. Also, the term with max is the Q value when selecting the action a with the highest Q value known at that time under the state s _{t + 1} , multiplied by γ. Here, γ is a parameter of 0 <γ ≦ 1 and is called a discount rate. Further, α is a learning coefficient and is in the range of 0 <α ≦ 1.

The above-mentioned formula (1) as a result of the trial a _t, based on the reward r _{t + 1} came back, represents a method for updating the evaluation value Q of the action a _t in state _{s t (s t, a t} ) ing. That is, rather than the evaluation value Q (s _t , a _t ) of the action a in the state s, the evaluation value Q (s _{t + 1} , max) of the best action max a in the next state by the reward r _{t + 1} and the action a If the sum of a _{t + 1} ) is larger, it indicates that Q (s _t , a _t ) is larger, and if it is smaller, Q (s _t , a _t ) is smaller. In other words, the value of a certain action in a certain state is brought closer to the value of the reward that is immediately returned as a result and the value of the best action in the next state due to that action.

  Here, the method of expressing Q (s, a) on a computer is a method of storing the values of all state action pairs (s, a) as a table, and Q (s, a) There is a way to prepare a function that approximates. In the latter method, the above equation (1) can be realized by adjusting the parameters of the approximation function by a method such as the probability gradient descent method. A neural network described later can be used as the approximation function.

  Also, a neural network can be used as a learning model of supervised learning, unsupervised learning, or an approximation algorithm of a value function in reinforcement learning. FIG. 2 is a view schematically showing a model of a neuron, and FIG. 3 is a view schematically showing a three-layer neural network configured by combining the neurons shown in FIG. That is, the neural network is configured by, for example, an arithmetic unit and a memory, etc., which simulate a model of a neuron as shown in FIG.

As shown in FIG. 2, the neuron outputs an output (result) y for a plurality of inputs x (in FIG. 2, as an example, the inputs x1 to x3). Each input x (x1, x2, x3) is multiplied by a weight w (w1, w2, w3) corresponding to this input x. Thereby, the neuron outputs the result y expressed by the following equation (2). The input x, the result y and the weight w are all vectors. Further, in the following formula (2), θ is a bias, and f _k is an activation function.

Referring to FIG. 3, a three-layer neural network configured by combining the neurons shown in FIG. 2 will be described. As shown in FIG. 3, a plurality of inputs x (here, as an example, inputs x1 to x3) are input from the left side of the neural network, and results y (here, as an example, results y1 to input y3). Is output. Specifically, the inputs x1, x2, x3 are input to the three neurons N11 to N13 after being multiplied by corresponding weights. The weights applied to these inputs are collectively labeled W 1.

The neurons N11 to N13 output z11 to z13, respectively. 3, these z11~z13 are collectively is labeled as a feature vector Z 1 and can be considered to have extracts a feature quantity of the input vector vector. The feature vector Z 1 is a feature vector between the weight W 1 and the weight W 2. z11 to z13 are input after being multiplied by corresponding weights for each of the two neurons N21 and N22. The weights applied to these feature vectors are collectively labeled W2 .

The neurons N21 and N22 output z21 and z22, respectively. 3, these z21, Z22 are collectively are labeled as feature vector Z 2. The feature vector Z 2 is a feature vector between the weight W 2 and the weight W 3. z21 and z22 are input to each of the three neurons N31 to N33 after being multiplied by corresponding weights. The weights applied to these feature vectors are collectively labeled W3 .

  Finally, the neurons N31 to N33 output the results y1 to y3, respectively. The operation of the neural network includes a learning mode and a value prediction mode. For example, in the learning mode, the weight W is learned using the learning data set, and the action judgment of the robot is performed in the prediction mode using the parameters. In addition, although written as prediction for convenience, it goes without saying that various tasks such as detection, classification, and inference can be performed.

  Here, the data obtained by actually moving the robot in the prediction mode is immediately learned, and reflected on the next action (online learning), or the learning summarized using the data group collected in advance is performed. After that, it is also possible to perform detection mode with that parameter (batch learning). Alternatively, it is possible to interpolate the learning mode every time the data is accumulated to some extent.

Also, the weights W 1 to W 3 can be learned by an error back propagation method (error reverse propagation method: back propagation). The error information flows from the right side to the left side. The error back propagation method is a method of adjusting (learning) each weight of each neuron so as to reduce the difference between the output y when the input x is input and the true output y (teacher).

  Such neural networks can also have more layers in three or more layers (referred to as deep learning). Further, it is also possible to automatically obtain an arithmetic device which performs input feature extraction stepwise and returns the result from only the teacher data.

  Therefore, the machine learning device 20 according to the present embodiment performs the state quantity observation unit 21, the operation result acquisition unit 26, the learning unit 22, and the decision making unit as illustrated in FIG. It is equipped with 25. However, as described above, the machine learning method applied to the present invention is not limited to Q learning. That is, various methods such as “supervised learning”, “unsupervised learning”, “semi-supervised learning” and “reinforcement learning”, which are methods that can be used in a machine learning apparatus, are applicable. In addition, although these machine learnings (machine learning apparatus 20) may use a general purpose computer or processor, if GPGPU, a large scale PC cluster, etc. are applied, it is possible to process more rapidly.

  That is, according to the present embodiment, the machine learning device learns the operation of the robot 14 that takes out the workpieces 12 by the hand unit 13 from the plurality of workpieces 12 placed in a mess, including the bulked state. The state of the robot 14 including the output data of the three-dimensional measuring instrument 15 which measures the three-dimensional position (x, y, z) for every 12 or three-dimensional position and posture (x, y, z, w, p, r) State amount observation unit 21 for observing the amount, an operation result acquisition unit 26 for acquiring the result of the taking-out operation of the robot 14 for taking out the work 12 by the hand unit 13, an output from the state amount observation unit 21 and the operation result acquisition unit 26 Operation amount including command data for instructing the robot 14 to take out the work 12 from the output from the control unit 12. The operation amount is related to the state quantity of the robot 14 and the result of the take-out operation. Comprising a learning unit 22 for learning and Te.

  Note that the state quantities observed by the state quantity observing unit 21 may include, for example, state variables for setting the position, posture, and taking-out direction of the hand unit 13 when taking out a certain work 12 from the box 11. Further, the operation amount to be learned includes, for example, command values such as torque, speed, and rotational position given from the control device 16 to the respective drive shafts of the robot 14 and the hand unit 13 when the work 12 is taken out of the box 11 May be

  When the learning unit 22 takes out one of the plurality of works 12 stacked in bulk, the learning unit 22 learns the above-mentioned state variable in association with the result of the taking-out operation of the work 12 (output of the operation result acquiring unit 26). . That is, the output data of the three-dimensional measuring instrument 15 (coordinate calculating unit 19) and the command data of the hand unit 13 are set at random by the control device 16 or are set intentionally based on a predetermined rule. The operation of taking out the work 12 by the unit 13 is performed. Here, as the predetermined rule, for example, among the plurality of workpieces 12 stacked in bulk, there is a method in which the workpieces are taken out in order from the workpiece having the high height (z) direction. Thereby, the output data of the three-dimensional measuring instrument 15 and the command data of the hand unit 13 correspond to the action of taking out a certain work. Then, each time the success or failure of taking out the work 12 occurs and such success or failure occurs, the learning unit 22 is a state variable configured from the output data of the three-dimensional measuring instrument 15 and the command data of the hand unit 13 Evaluate the

  Further, the learning unit 22 stores the output data of the three-dimensional measuring instrument 15 when taking out the work 12 and the command data of the hand unit 13 in association with the evaluation of the result of the taking-out operation of the work 12. As a failure example, there is a case where the hand portion 13 can not hold the work 12 or a case where the work 12 collides with or contacts the wall of the box 11 even if the hand 12 can be held. In addition, the success or failure of such removal of the work 12 is determined based on the detection value of the force sensor 17 and the imaging data by the three-dimensional measuring device. Here, the machine learning apparatus 20 can also perform learning using, for example, a part of the command data of the hand unit 13 output from the control device 16.

  Here, it is preferable that the learning unit 22 of the present embodiment includes a reward calculating unit 23 and a value function updating unit 24. For example, the reward calculation unit 23 calculates a reward, for example, a score, based on the success or failure of the retrieval of the work 12 caused by the above-mentioned state variable. The reward is made to be high for success in taking out the work 12 and to be low for failure in taking out the work 12. Alternatively, the reward may be calculated based on the number of successful retrieval of the work 12 within a predetermined time. Furthermore, when calculating this reward, for example, according to each stage of taking out the work 12, such as success in gripping by the hand 13, success in transportation by the hand 13, success in placing the work 12, etc. It may be calculated.

  Then, the value function updating unit 24 has a value function that determines the value of the retrieval operation of the work 12, and updates the value function according to the above-mentioned reward. To update the value function, the update equation of the value Q (s, a) as described above is used. Furthermore, at the time of this updating, it is preferable to create an action value table. The action value table referred to here is a value function updated according to the output data of the three-dimensional measuring instrument 15 when the work 12 is taken out, the command data of the hand unit 13 and the taken out result of the work 12 at that time (ie Evaluation value) is related to each other and recorded.

  It is also possible to use a function approximated using the above-mentioned neural network as this action value table, and it is particularly effective when the amount of information of the state s is enormous as in image data. Also, the above value function is not limited to one type. For example, a value function for evaluating success or failure of gripping of the work 12 by the hand unit 13 or a value function for evaluating time (cycle time) required for gripping and transporting the work 12 by the hand unit 13 can be considered.

  Furthermore, as the above-mentioned value function, a value function that evaluates the interference between the box 11 and the hand unit 13 or the work 12 at the time of taking out the work may be used. In order to calculate a reward used for updating the value function, the state quantity observing unit 21 preferably observes a force acting on the hand unit 13, for example, a value detected by the force sensor 17. Then, when the amount of change in force detected by the force sensor 17 exceeds a predetermined threshold, it can be estimated that the above interference has occurred, so the reward in that case is, for example, a negative value, and the value determined by the value function is low. It is preferable to

  Further, according to the present embodiment, it is also possible to learn the measurement parameter of the three-dimensional measuring instrument 15 as the operation amount. That is, according to the present embodiment, the machine learning device learns the operation of the robot 14 that takes out the workpieces 12 by the hand unit 13 from the plurality of workpieces 12 placed in a mess, including the bulked state. The state of the robot 14 including the output data of the three-dimensional measuring instrument 15 which measures the three-dimensional position (x, y, z) for every 12 or three-dimensional position and posture (x, y, z, w, p, r) State amount observation unit 21 for observing the amount, an operation result acquisition unit 26 for acquiring the result of the taking-out operation of the robot 14 for taking out the work 12 by the hand unit 13, an output from the state amount observation unit 21 and the operation result acquisition unit 26 A learning unit 22 that receives an output from the control unit and learns the operation amount including the measurement parameter of the three-dimensional measuring instrument 15 in association with the state amount of the robot 14 and the result of the extraction operation; Provided.

  Furthermore, in the robot system 10 according to the present embodiment, an automatic hand changer (not shown) may be provided to replace the hand unit 13 attached to the robot 14 with the hand unit 13 of another form. In that case, it is preferable that the value function updating unit 24 have the above-mentioned value function for each of the hand units 13 having different forms, and update the value function of the hand unit 13 after replacement according to the reward. As a result, since the optimum operation of the hand unit 13 can be learned for each of the plurality of hands 13 having different forms, the automatic hand changer can select the hand unit 13 having a higher value function.

  Subsequently, the decision making unit 25 selects output data of the three-dimensional measuring instrument 15 and command data of the hand unit 13 corresponding to the highest evaluation value, for example, with reference to the action value table created as described above. It is preferable to do. After that, the decision making unit 25 outputs the optimum data of the selected hand unit 13 or three-dimensional measuring instrument 15 to the control device 16.

  Then, the control device 16 controls the three-dimensional measuring instrument 15 and the robot 14 to take out the workpiece 12 by using the optimum data of the hand unit 13 and the three-dimensional measuring instrument 15 output by the learning unit 22. For example, the control device 16 operates the drive shafts of the hand unit 13 and the robot 14 based on the state variables which set the optimum position, posture, and take-out direction of the hand unit 13 obtained by the learning unit 22. Is preferred.

  The robot system 10 according to the embodiment described above is provided with one machine learning device 20 for one robot 14 as shown in FIG. 1. However, in the present invention, the number of each of the robot 14 and the machine learning device 20 is not limited to one. For example, the robot system 10 may include a plurality of robots 14 and one or more machine learning devices 20 may be provided corresponding to each robot 14. Then, it is preferable that the robot system 10 share or exchange the optimum state variables of the three-dimensional measuring instrument 15 and the hand unit 13 acquired by the machine learning device 20 of each robot 14 by a communication medium such as a network. Thereby, even if the operation rate of one robot 14 is lower than the operation rate of another robot 14, the optimal operation result acquired by the machine learning device 20 provided to another robot 14 is used for the operation of a robot 14. be able to. In addition, the time taken for learning can be shortened by sharing learning models among a plurality of robots or sharing the operation amount including the measurement parameter of the three-dimensional measuring instrument 15 with the state quantity of the robot 14 and the result of taking out operation. Can.

  Furthermore, the machine learning device 20 may be inside the robot 14 or outside the robot 14. Alternatively, the machine learning device 20 may be in the control device 16 or in a cloud server (not shown).

  When the robot system 10 includes a plurality of robots 14, while the robot 14 transports the work 12 gripped by the hand unit 13, the work of taking out the work 12 to the hand unit of another robot 14 is performed. It is possible to Then, the value function updating unit 24 can also update the value function using the time during which the robot 14 taking out such a work 12 switches. Furthermore, the machine learning device 20 has state variables of a plurality of hand models, performs extraction simulation with a plurality of hand models during the extraction operation of the work 12, and according to the result of the extraction simulation, a plurality of hand models It is also possible to learn in association with the result of the retrieval operation of the work 12.

  In the above-described machine learning apparatus 20, output data of the three-dimensional measuring instrument 15 at the time of acquiring data of the three-dimensional map for each work 12 is transmitted from the three-dimensional measuring instrument 15 to the state quantity observing unit 21. It is supposed to be. Since such transmission data does not necessarily include abnormal data, the machine learning device 20 has a function of filtering abnormal data, that is, data from the three-dimensional measuring instrument 15 as the state quantity observation unit 21. It is possible to have a function capable of selecting whether or not to input. As a result, the learning unit 22 of the machine learning device 20 can efficiently learn the optimal operation of the hand unit 13 by the three-dimensional measuring instrument 15 and the robot 14.

  Furthermore, in the above-described machine learning apparatus 20, although the output data from the learning unit 22 is input to the control device 16, it is assumed that the output data from the learning unit 22 does not include any abnormal data. However, the function of filtering abnormal data, that is, a function capable of selecting whether to output the data from the learning unit 22 to the control device 16 may be provided. As a result, the control device 16 can make the robot 14 execute the optimum operation of the hand unit 13 more safely.

  In addition, the above-mentioned abnormal data can be detected by the following procedures. In other words, the probability distribution of input data is estimated, and the probability distribution is used to derive the occurrence probability of a new input, and if the occurrence probability is less than or equal to a certain value, it is regarded as abnormal data that deviates significantly from typical behavior Data can be detected.

  Next, an example of the operation of the machine learning device 20 provided in the robot system 10 of the present embodiment will be described. FIG. 4 is a flow chart showing an example of the operation of the machine learning device shown in FIG. As shown in FIG. 4, in the machine learning apparatus 20 shown in FIG. 1, when the learning operation (learning process) starts, the three-dimensional measuring device 15 performs three-dimensional measurement and outputs it (step S11 in FIG. 4). . That is, in step S11, for example, a three-dimensional map (output data of the three-dimensional measuring instrument 15) for each of the randomly placed workpieces 12 including the bulk-stacked state is acquired and output to the state quantity observation unit 21. At the same time, the coordinate calculation unit 19 receives the three-dimensional map of each work 12 and calculates the three-dimensional position (x, y, z) of each work 12 to obtain the state quantity observation unit 21, the operation result acquisition unit 26 and the control device 16. Output to Here, the coordinate calculation unit 19 may calculate and output the posture (w, p, r) of each work 12 from the output of the three-dimensional measuring instrument 15.

  As described with reference to FIG. 5, the output (three-dimensional map) of the three-dimensional measuring instrument 15 is subjected to state amount observation via the preprocessing unit 50 that is processed before being input to the state amount observation unit 21. It may be input to the unit 21. Further, as described with reference to FIG. 7, only the output of the three-dimensional measuring instrument 15 may be input to the state quantity observing unit 21, and further, only the output of the three-dimensional measuring instrument 15 may be the preprocessing unit 50. It may be input to the state quantity observation unit 21 via Thus, the implementation and output of the three-dimensional measurement in step S11 can include various things.

  Specifically, in the case of FIG. 1, the state quantity observation unit 21 has a three-dimensional map for each work 12 from the three-dimensional measuring instrument 15 and a three-dimensional position for each work 12 from the coordinate calculation unit 19 (x , Y, z) and attitudes (w, p, r), and observe state quantities (output data of the three-dimensional measuring instrument 15). The operation result acquisition unit 26 acquires the result of the extraction operation of the robot 14 for extracting the workpiece 12 by the hand unit 13 based on the output data of the three-dimensional measuring instrument 15 (output data of the coordinate calculation unit 19). In addition to the output data of the three-dimensional measuring instrument, the operation result acquisition unit 26 also acquires, for example, the result of the extraction operation such as the degree of achievement when the taken out workpiece 12 is passed to the post process and the breakage of the extracted workpiece 12 be able to.

  Furthermore, for example, the machine learning device 20 determines the optimum operation based on the output data of the three-dimensional measuring instrument 15 (step S12 in FIG. 4), and the control device 16 controls the hand unit 13 (robot 14). The command data (operation amount) is output to carry out the operation of taking out the work 12 (step S13 in FIG. 4). Then, the work extraction result is acquired by the above-described operation result acquisition unit 26 (step S14 in FIG. 4).

  Next, based on the output from the operation result acquisition unit 26, it is determined whether or not the removal of the work 12 is successful (step S15 in FIG. 4), and when the work 12 is successfully removed, a positive reward is set (FIG. 4). Step S16), if the retrieval of the work 12 fails, a negative reward is set (step S17 in FIG. 4), and the action value table (value function) is updated (step S18 in FIG. 4).

  Here, the success or failure of the removal of the workpiece 12 can be determined, for example, based on the output data of the three-dimensional measuring instrument 15 after the removal operation of the workpiece 12. Moreover, the success or failure determination of the removal of the work 12 is not limited to the evaluation of the success or failure of the removal of the work 12. For example, the achievement degree when the taken out work 12 is passed to the post process, breakage of the removed work 12 Alternatively, the time (cycle time) or energy (electric energy) required for gripping and transporting the work 12 by the hand unit 13 may be evaluated as to whether there is a change in state.

  The calculation of the value of the reward based on the judgment of success or failure of taking out the work 12 is performed by the reward calculation unit 23, and the update of the action value table is performed by the value function update unit 24. That is, when the retrieval unit 22 succeeds in taking out the work 12, a positive reward is set to the reward in the update formula of the value Q (s, a) described above (S16), and the taking out of the work 12 fails. If it has, the negative reward is set to the reward in the renewal formula (S17). Then, the learning unit 22 updates the aforementioned action value table each time the work 12 is taken out (S18). By repeating the above steps S11 to S18, the learning unit 22 continues (learning) updating of the action value table.

  In the above, the data input to the state quantity observation unit 21 is not limited to the output data of the three-dimensional measuring instrument 15. For example, the data such as the output of another sensor may be included. It is also possible to use part of the command data from. In this manner, the control device 16 uses the command data (operation amount) output from the machine learning device 20 to cause the robot 14 to take out the work 12. The learning performed by the machine learning apparatus 20 is not limited to the taking-out operation of the work 12. For example, as described above, the measurement parameter of the three-dimensional measuring instrument 15 may be used.

  As described above, according to the robot system 10 including the machine learning device 20 of the present embodiment, the robot that takes out the workpieces 12 from the plurality of randomly placed workpieces 12 including the bulked state by the hand unit 13 It is possible to learn 14 actions. As a result, the robot system 10 can learn the selection of the optimal operation of the robot 14 for taking out the workpieces 12 stacked in bulk without human intervention.

  FIG. 5 is a block diagram showing a conceptual configuration of a robot system according to another embodiment of the present invention, showing a robot system to which supervised learning is applied. As apparent from the comparison between FIG. 5 and FIG. 1 described above, the robot system 10 ′ to which the supervised learning shown in FIG. 5 is applied is different from the robot system 10 to which Q learning (reinforcement learning) shown in FIG. Further, a result (labeled) data recording unit 40 is provided. The robot system 10 ′ shown in FIG. 5 further includes a preprocessing unit 50 that preprocesses the output data of the three-dimensional measuring instrument 15. Needless to say, the preprocessing unit 50 can also be provided, for example, for the robot system 10 shown in FIG.

  As shown in FIG. 5, the machine learning device 30 in the robot system 10 ′ to which supervised learning is applied includes the state quantity observing unit 31, the operation result acquiring unit 36, the learning unit 32, and the decision making unit 35. Equipped with The learning unit 32 includes an error calculating unit 33 and a learning model updating unit 34. Also in the robot system 10 'of the present embodiment, the machine learning device 30 learns an operation amount such as command data for instructing the robot 14 to take out the work 12 or a measurement parameter of the three-dimensional measuring instrument 15. Output.

  That is, in the robot system 10 'to which supervised learning shown in FIG. 5 is applied, the error calculating unit 33 and the learning model updating unit 34 respectively calculate the reward calculating unit 23 and the reward calculating unit 23 in the robot system 10 to which Q learning shown in FIG. It corresponds to the value function update unit 24. The other configurations, for example, the configurations of the three-dimensional measuring instrument 15, the control device 16, the robot 14 and the like are the same as those of FIG. 1 described above, and the description thereof will be omitted.

  The error calculation unit 33 calculates an error between the result (label) output from the operation result acquisition unit 36 and the output of the learning model implemented in the learning unit. Here, the result (label) attached data recording unit 40 is a result (label obtained by the day before the predetermined day of causing the robot 14 to perform the work, for example, when the shape of the work 12 and the processing by the robot 14 are the same) ) The attached data can be held, and the result (labeled) data held in the result (labeled) data recording unit 40 can be provided to the error calculating unit 33 on the predetermined date. Alternatively, data obtained by simulation or the like performed outside the robot system 10 'or data (labeled) with other robot system results can be calculated by the error of the robot system 10' using a memory card or a communication line. It is also possible to provide to the unit 33. Furthermore, the result (label) attached data recording unit 40 is configured by a non-volatile memory such as a flash memory (Flash Memory), and the result (label) attached data recording unit (nonvolatile memory) 40 is incorporated in the learning unit 32 The result (labeled) data stored in the result (labeled) data recording unit 40 can be used as it is by the learning unit 32.

  FIG. 6 is a view for explaining an example of processing of the pre-processing unit in the robot system shown in FIG. 5, and FIG. 6 (a) shows three-dimensional positions of a plurality of works 12 stacked in bulk in box 11. 6B shows an example of output data of the three-dimensional measuring instrument 15. In FIG. 6B to FIG. 6D, pre-processing is performed on the workpieces 121 to 123 in FIG. Shows an example of image data after image processing.

  Here, a cylindrical metal part is assumed as the workpiece 12 (121 to 123), and for the hand (13), for example, the cylindrical workpiece 12 is not held by two claws. It is assumed that the suction pad sucks the longitudinal central portion of the under pressure with a negative pressure. Therefore, for example, if the position of the longitudinal central portion of the work 12 is known, the work 12 can be taken out by moving and suctioning the suction pad (13) to the position. The numerical values in FIGS. 6A to 6D are represented by [mm], and indicate the x direction, the y direction, and the z direction, respectively. In the z direction, the height (depth) of the image data captured by the three-dimensional measuring instrument 15 (for example, having two cameras) provided above the box 11 in which a plurality of works 12 are stacked in bulk. Correspond to the direction.

  As apparent from the comparison between FIG. 6 (a) and FIGS. 6 (b) to 6 (d), a three-dimensional measuring device is an example of the processing of the preprocessing unit 50 in the robot system 10 ′ shown in FIG. From the 15 output data (three-dimensional image), the target work 12 (for example, three works 121 to 123) is rotated and processed so that the height of the center becomes "0".

  That is, the output data of the three-dimensional measuring instrument 15 includes, for example, information on the three-dimensional position (x, y, z) and the attitude (w, p, r) of the longitudinal central portion of each work 12 . At this time, as shown in FIGS. 6 (b), 6 (c) and 6 (d), the three works 121, 122 and 123 to be noticed are rotated by -r respectively and subtracted by z. All in line with the same conditions. By performing such pre-processing, the load on the machine learning device 30 can be reduced.

  Here, also the three-dimensional drawing shown in FIG. 6A is not the output data of the three-dimensional measuring instrument 15 itself, but, for example, from an image obtained by a program defining the taking out order of the work 12 implemented before. The threshold value for selection is lowered, and this processing itself can also be performed by the pre-processing unit 50. It is needless to say that the processing by the pre-processing unit 50 can be variously changed depending on various conditions including the shape of the work 12 and the type of the hand 13 and the like.

  Thus, the output data (three-dimensional map for each work 12) of the three-dimensional measuring instrument 15 processed by the pre-processing unit 50 before input to the state quantity observation unit 31 is input to the state quantity observation unit 31 It will be done. Again, referring to FIG. 5, error calculation unit 33 receiving the result (label) output from operation result acquisition unit 36 is, for example, actually when the output of the neural network shown in FIG. 3 is y as a learning model. If there is an error of -log (y) if the work 12 is taken out successfully and if there is an error, it is considered that there is an error of -log (1-y) to minimize this error. Process with the goal. In addition, as an input of the neural network shown in FIG. 3, for example, image data of the objects 121 to 123 of interest after pre-processing as shown in FIG. 6 (b) to FIG. 6 (d), and Data of the three-dimensional position and posture (x, y, z, w, p, r) of each of the works 121 to 123 to be noted will be given.

  FIG. 7 is a block diagram showing a modification of the robot system shown in FIG. As apparent from the comparison between FIG. 7 and FIG. 1, in the modified example of the robot system 10 shown in FIG. 7, the coordinate calculation unit 19 is eliminated, and the state quantity observation unit 21 is three-dimensional from the three-dimensional measuring instrument 15. It receives only the map and observes the state quantity of the robot 14. It goes without saying that a configuration corresponding to the coordinate calculation unit 19 can be provided to the control device 16. The configuration shown in FIG. 7 can also be applied to, for example, a robot system 10 ′ to which the supervised learning described with reference to FIG. 5 is applied. That is, in the robot system 10 'shown in FIG. 5, the preprocessing unit 50 may be deleted, and the state quantity observing unit 31 may receive only the three-dimensional map from the three-dimensional measuring instrument 15 and observe the state quantity of the robot 14. It is possible. Thus, various modifications and variations can be made to the embodiments described above.

  As described above in detail, according to the present embodiment, a machine learning apparatus capable of learning an optimal operation of the robot when taking out a randomly placed work including a bulked state without human intervention. It becomes possible to provide a robot system and a machine learning method. The machine learning devices 20 and 30 in the present invention are not limited to those to which reinforcement learning (for example, Q learning) or supervised learning is applied, and various machine learning algorithms can be applied.

  Although the embodiments have been described above, all the examples and conditions described herein are for the purpose of assisting the understanding of the concept of the invention applied to the invention and the technology, and the examples and conditions described are particularly It is not intended to limit the scope of the invention. Also, such descriptions in the specification do not show the advantages and disadvantages of the invention. While the embodiments of the invention have been described in detail, it should be understood that various changes, substitutions, and alterations can be made without departing from the spirit and scope of the invention.

10, 10 'Robot system 11 Box 12 Work 13 Hand part 14 Robot 15 Three-dimensional measuring instrument 16 Control device 17 Force sensor 18 Support part 19 Coordinate calculation part 20, 30 Machine learning device 21, 31 State quantity observation part 22, 32 Learning Part 23 Reward calculation part 24 Value function update part 25, 35 Decision making part 26, 36 Operation result acquisition part 33 Error calculation part 34 Learning model update part 40 Result (labeled) data storage part 50 Pre-processing part

Claims (20)

A machine learning device that learns the operation of a robot (14) for taking out the work (12) by a hand unit (13) from a plurality of randomly placed works (12), including a bulk-stacked state,
Output data of a three-dimensional measuring instrument (15) which measures a plurality of workpieces (12) stacked in bulk and outputs three-dimensional position information of the surface of the plurality of workpieces (12) and the hand of the workpiece (12) A state quantity observing section (21, 31) for observing a state quantity which sets the position, posture and taking-out direction of the hand section (13) respectively for the taking-out operation by the section (13);
An operation result acquisition unit (26, 36) for acquiring the result of the taking out operation of the robot (14) for taking out the work (12) by the hand unit (13);
The state quantities observed by the state quantity observation unit (21, 31) and the determination result of the success or failure of the extraction operation of the robot (14) acquired by the operation result acquisition unit (26, 36) are received, A learning unit (22, 32) for learning the takeout operation of the work (12);
The learning unit (22)
A reward calculation unit (23) that calculates a reward based on the result of the success or failure of the removal of the work from the operation result acquisition unit (26);
A value function updating unit (24) which has a value function that determines the value of the extraction operation of the work (12), and updates the value function according to the reward, and is acquired by the learning unit (22) A learning is performed to update the state variable for setting the optimum position, posture and take-out direction of the hand portion (13) based on the value function, and the hand portion (13) is updated based on the updated state variable. 13) and operating each drive axis of the robot (14),
A machine learning apparatus characterized by A machine learning device that learns the operation of a robot (14) for taking out the work (12) by a hand unit (13) from a plurality of randomly placed works (12), including a bulk-stacked state,
Output data of a three-dimensional measuring instrument (15) which measures a plurality of workpieces (12) stacked in bulk and outputs three-dimensional position information of the surface of the plurality of workpieces (12) and the hand of the workpiece (12) A state quantity observing section (21, 31) for observing a state quantity which sets the position, posture and taking-out direction of the hand section (13) respectively for the taking-out operation by the section (13);
An operation result acquisition unit (26, 36) for acquiring the result of the taking out operation of the robot (14) for taking out the work (12) by the hand unit (13);
The state quantities observed by the state quantity observation unit (21, 31) and the determination result of the success or failure of the extraction operation of the robot (14) acquired by the operation result acquisition unit (26, 36) are received, A learning unit (22, 32) for learning the takeout operation of the work (12);
The learning unit (32) implements a learning model for learning the extraction operation of the work (12);
An error calculation unit (33) that calculates an error based on the label output from the operation result acquisition unit (36) and the output of the learning model implemented in the learning unit (32);
A learning model updating unit (34) for updating the learning model according to the error, and the optimum position, posture, and taking-out direction of the hand unit (13) obtained by the learning unit (32) Operate the drive shafts of the hand unit (13) and the robot (14) based on the state variable to be set;
A machine learning apparatus characterized by further,
Based on the state variable for setting the optimum position, posture and take-out direction of the hand unit (13) obtained by the learning unit (22, 32), the robot ( 14) comprising a decision making unit (25, 35) for determining command data to be commanded to
The machine learning apparatus according to claim 1 or 2, wherein A machine learning device that learns the operation of a robot (14) for taking out the work (12) by a hand unit (13) from a plurality of randomly placed works (12), including a bulk-stacked state,
Output data of a three-dimensional measuring instrument (15) which measures a plurality of workpieces (12) stacked in bulk and outputs three-dimensional position information of the surface of the plurality of workpieces (12); the hand of the workpiece (12) the hand part take-out operation by the unit (13) the position (13), sets the posture and extraction direction, respectively, as well as the containing measurement parameters of the three-dimensional measuring instrument (15), the state quantity observation for observing state variables Part (21, 31),
An operation result acquisition unit (26, 36) for acquiring the result of the taking out operation of the robot (14) for taking out the work (12) by the hand unit (13);
The state quantities observed by the state quantity observation unit (21, 31) and the determination result of the success or failure of the extraction operation of the robot (14) acquired by the operation result acquisition unit (26, 36) are received, A learning unit (22, 32) for learning the takeout operation of the work (12);
The learning unit (22)
A reward calculation unit (23) that calculates a reward based on the result of the success or failure of the removal of the work from the operation result acquisition unit (26);
A value function updating unit (24) which has a value function that determines the value of the extraction operation of the work (12), and updates the value function according to the reward, and is acquired by the learning unit (22) A learning is performed to update the state variable for setting the optimum position, posture and take-out direction of the hand portion (13) based on the value function, and the hand portion (13) is updated based on the updated state variable. 13) and operating each drive axis of the robot (14),
A machine learning apparatus characterized by A machine learning device that learns the operation of a robot (14) for taking out the work (12) by a hand unit (13) from a plurality of randomly placed works (12), including a bulk-stacked state,
Output data of a three-dimensional measuring instrument (15) which measures a plurality of workpieces (12) stacked in bulk and outputs three-dimensional position information of the surface of the plurality of workpieces (12); the hand of the workpiece (12) the hand part take-out operation by the unit (13) the position (13), sets the posture and extraction direction, respectively, as well as the containing measurement parameters of the three-dimensional measuring instrument (15), the state quantity observation for observing state variables Part (21, 31),
An operation result acquisition unit (26, 36) for acquiring the result of the taking out operation of the robot (14) for taking out the work (12) by the hand unit (13);
The state quantities observed by the state quantity observation unit (21, 31) and the determination result of the success or failure of the extraction operation of the robot (14) acquired by the operation result acquisition unit (26, 36) are received, A learning unit (22, 32) for learning the takeout operation of the work (12);
The learning unit (32) implements a learning model for learning the extraction operation of the work (12);
An error calculation unit (33) that calculates an error based on the label output from the operation result acquisition unit (36) and the output of the learning model implemented in the learning unit (32);
A learning model updating unit (34) for updating the learning model according to the error, and the optimum position, posture, and taking-out direction of the hand unit (13) obtained by the learning unit (32) Operate the drive shafts of the hand unit (13) and the robot (14) based on the state variable to be set;
A machine learning apparatus characterized by further,
Based on the state variable for setting the optimum position, posture and take-out direction of the hand unit (13) obtained by the learning unit (22, 32), the robot ( 14) command data for commanding the operation and a decision making unit (25, 35) for determining the measurement parameter of the three-dimensional measuring instrument (15),
The machine learning device according to claim 4 or 5, characterized in that: The state quantity observation unit (21, 31) is further calculated by the coordinate calculation unit (19) based on three-dimensional position information of the surface of the work (12) output from the three-dimensional measuring instrument (15). It also receives three-dimensional position information for each work (12),
The machine learning device according to any one of claims 1 to 6, characterized in that: The coordinate calculation unit (19) further
Calculating posture data for each work (12) and outputting the calculated three-dimensional position data and posture data for each work (12);
The machine learning device according to claim 7, wherein The operation result acquisition unit (26, 36) acquires the result of the extraction operation of the robot (14) using the output data of the three-dimensional measuring instrument (15).
The machine learning device according to any one of claims 1 to 8, characterized in that. further,
Pre-processing to process output data of the three-dimensional measuring instrument (15) so that the state quantity observing unit (21, 31) can be easily processed before input to the state quantity observing unit (21, 31) Equipped with a part (50)
The state quantity observation unit (21, 31) receives output data of the pre-processing unit (50),
The machine learning device according to any one of claims 1 to 9, characterized in that: The pre-processing unit (50) rotates and positions the three-dimensional position data (x, y, z) and attitude data (w, p, r) included in the output data of the three-dimensional measuring instrument (15). Perform alignment to make the height of the center of each work (12) uniform,
The machine learning device according to claim 10, characterized in that: The operation result acquisition unit (26, 36)
In addition to the success or failure of the removal of the work (12), the damaged state of the work (12) and the time and energy required to hold and transport the work (12) by the hand portion (13), or Obtaining at least one of the time and an estimate of the energy,
The machine learning device according to any one of claims 1 to 11, characterized in that: The machine learning device
The machine learning device according to any one of claims 1 to 12, comprising a neural network. A robot system (10, 10 ') comprising the machine learning device (20, 30) according to any one of claims 1 to 13, comprising:
The robot (14),
The three-dimensional measuring instrument (15);
And a control device (16) for controlling the robot (14) and the three-dimensional measuring instrument (15), respectively.
A robot system characterized by The robot system (10, 10 ') comprises a plurality of the robots (14)
The machine learning device (20, 30) is provided for each of the robots (14).
The plurality of machine learning devices (20) provided in the plurality of robots (14) share or exchange data with each other via a communication medium.
The robot system according to claim 14, characterized in that: The machine learning device (20, 30) exists on a cloud server,
The robot system according to claim 15, characterized in that: A machine learning method by a machine learning device for learning an operation of a robot (14) for taking out the work (12) by a hand unit (13) from a plurality of randomly placed works (12) including a bulk-stacked state There,
Output data of a three-dimensional measuring instrument (15) which measures a plurality of workpieces (12) stacked in bulk and outputs three-dimensional position information of the surface of the plurality of workpieces (12) and the hand of the workpiece (12) Observing a state quantity for setting the position, posture, and taking-out direction of the hand part (13) with respect to the taking-out operation by the part (13);
Obtaining the result of the taking-out operation of the robot (14) for taking out the work (12) by the hand unit (13);
Receiving the observed state quantity and the acquired determination result of the success or failure of the extraction operation of the robot (14), and learning the extraction operation of the work (12);
The learning of the extraction operation is
Calculate the reward based on the acquired judgment result of success or failure of taking out the work,
Having a value function that determines the value of the retrieval operation of the work (12), and updating the value function according to the reward;
The state variable is updated based on the value function, and the state variable for setting the optimum position, posture, and take-out direction of the hand unit (13) obtained by learning the take-out operation is carried out. Operate the drive shafts of the hand unit (13) and the robot (14) based on
Machine learning method characterized by A machine learning method by a machine learning device for learning an operation of a robot (14) for taking out the work (12) by a hand unit (13) from a plurality of randomly placed works (12) including a bulk-stacked state There,
Output data of a three-dimensional measuring instrument (15) which measures a plurality of workpieces (12) stacked in bulk and outputs three-dimensional position information of the surface of the plurality of workpieces (12) and the hand of the workpiece (12) Observing a state quantity for setting the position, posture, and taking-out direction of the hand part (13) with respect to the taking-out operation by the part (13);
Obtaining the result of the taking-out operation of the robot (14) for taking out the work (12) by the hand unit (13);
Receiving the observed state quantity and the acquired determination result of the success or failure of the extraction operation of the robot (14), and learning the extraction operation of the work (12);
The learning of the extraction operation implements a learning model for learning the extraction operation of the work (12),
Calculate the error based on the given label and the output of the implemented learning model,
Updating the learning model according to the error;
Each driving shaft of the hand unit (13) and the robot (14) is obtained based on the state variable which sets the optimum position, posture and taking-out direction of the hand unit (13) obtained by learning of the taking-out operation. To operate,
Machine learning method characterized by A machine learning method by a machine learning device for learning an operation of a robot (14) for taking out the work (12) by a hand unit (13) from a plurality of randomly placed works (12) including a bulk-stacked state There,
Output data of a three-dimensional measuring instrument (15) which measures a plurality of workpieces (12) stacked in bulk and outputs three-dimensional position information of the surface of the plurality of workpieces (12); the hand of the workpiece (12) Setting the position, posture and taking-out direction of the hand unit (13) for the taking-out operation by the unit (13), and observing the state quantities including the measurement parameters of the three-dimensional measuring instrument (15);
Obtaining the result of the taking-out operation of the robot (14) for taking out the work (12) by the hand unit (13);
Receiving the observed state quantity and the acquired determination result of the success or failure of the extraction operation of the robot (14), and learning the extraction operation of the work (12);
The learning of the extraction operation is
Calculate the reward based on the acquired judgment result of success or failure of taking out the work,
Having a value function that determines the value of the retrieval operation of the work (12), and updating the value function according to the reward;
The state variable is updated based on the value function, and the state variable for setting the optimum position, posture, and take-out direction of the hand unit (13) obtained by learning the take-out operation is carried out. Operate the drive shafts of the hand unit (13) and the robot (14) based on
Machine learning method characterized by A machine learning method by a machine learning device for learning an operation of a robot (14) for taking out the work (12) by a hand unit (13) from a plurality of randomly placed works (12) including a bulk-stacked state There,
Output data of a three-dimensional measuring instrument (15) which measures a plurality of workpieces (12) stacked in bulk and outputs three-dimensional position information of the surface of the plurality of workpieces (12); the hand of the workpiece (12) Setting the position, posture and taking-out direction of the hand unit (13) for the taking-out operation by the unit (13), and observing the state quantities including the measurement parameters of the three-dimensional measuring instrument (15);
Obtaining the result of the taking-out operation of the robot (14) for taking out the work (12) by the hand unit (13);
Receiving the observed state quantity and the acquired determination result of the success or failure of the extraction operation of the robot (14), and learning the extraction operation of the work (12);
The learning of the extraction operation implements a learning model for learning the extraction operation of the work (12),
Calculate the error based on the given label and the output of the implemented learning model,
Updating the learning model according to the error;
Each driving shaft of the hand unit (13) and the robot (14) is obtained based on the state variable which sets the optimum position, posture and taking-out direction of the hand unit (13) obtained by learning of the taking-out operation. To operate,
Machine learning method characterized by