JP3806133B2 - Data transmission apparatus, data transmission method and program - Google Patents

Description

  The present invention transmits an encoded moving image / still image using a wired communication network such as ISDN (Integrated Services Digital Network) or the Internet, or a wireless communication network such as PHS (Personal Handy-phone System) or satellite communication. The present invention relates to an information transmission method, a data transmission apparatus to which the method is applied, and a program thereof.

  In recent years, with the development of digital coding technology for various information including images and broadband network technology, the development of applications using these has become popular. Transmission systems have been developed.

  For example, as shown in FIG. 13, the image signal 131 input from the image input unit 101 is encoded by the image encoding unit 102, and the obtained encoded data 132 is given to the encoded data transmission unit 103 as transmission data 133. Send to the transmission line. In this way, images are encoded and transmitted. In recent years, with the spread of the Internet and Intranet, applications and systems that packetize and transmit / receive data are increasing. Packetization is a very effective means for efficiently sharing the bandwidth of a communication path among a plurality of users.

  By the way, TCP / IP (Transmission Control Protocol / Internet Protocol), UDP / IP (User Datagram Protocol / Internet Protocol), and the like exist as protocols for transmitting and receiving packet data on the Internet / Intranet.

  Among these, TCP / IP is an Internet standard protocol, and is a protocol that is applied not only to the Internet but also to intranets and LANs as the most popular protocol in the world. TCP works in the fourth layer transport layer of the OSI basic reference model, and IP works in the third layer network layer.

  Further, UDP / IP is one of the transport layer protocols of the TCP / IP suite, and TCP is a connection type protocol, whereas UDP is a connectionless type protocol. Similar to TCP, the host application is identified using the port number. Often used to carry short data for network supervisory control and real-time oriented data such as audio or video data.

  Of these, TCP / IP is resistant to errors because it incorporates a framework such as retransmission, and is effective in download-type applications that want to receive data correctly even if it takes some time. However, it is ineffective for applications that require real-time performance.

  On the other hand, UDP / IP does not have a retransmission framework, but has no delay for retransmission and is very effective for an application that requires real-time performance.

  A typical example of an application that requires real-time performance is the transmission of moving images. However, in the case of normal moving image communication, the amount of image data is very large, and in most cases it does not fit in the network bandwidth. is there. In that case, a method of encoding image data and transmitting the data after reducing the data amount is used. As compression coding techniques for moving image signals, techniques such as motion compensation, discrete cosine transform (DCT), subband coding, pyramid coding, variable length coding, etc., and methods combining these techniques have been developed.

  As international standard systems for moving picture encoding, ISO MPEG-1, MPEG-2, ITU-T H.264, etc. 261, H.H. 262, H.C. H.263, and ISO MPEG system, ITU-T H.264, and the like as international standard systems for multiplexing moving images, code strings compressed audio / audio signals, and other data. 221, H.M. 223 exists.

  The Internet and the like are connected through a myriad of networks, and it is normal not to know which network is in what state. In addition, since the amount of data flowing there fluctuates from moment to moment, a mechanism for determining how much data can be communicated in real time is necessary.

  In view of this, an application using a packet format called RTP (Real-time Transport Protocol), which advances one step further from a real-time application using UDP / IP and adds time information to a packet for transmission, is increasing.

  This RTP is a protocol for real-time transfer of audio data and video data defined by RFC1889, and is usually carried on UDP (User Datagram Protocol). It is intended to be applied to multimedia systems such as video conferencing, and can exchange data in real-time conversation format. However, there is no quality assurance function for sound quality and image quality. Real-time operation is supported by adding packet sequence numbers and time stamps (time display data) in the RTP header.

  RFC is an abbreviation for Request for Comments, and refers to a document of technical proposals and comments published by the Internet Engineering Task Force (IETF), which is an Internet technology development organization. Many de facto standards are described in the RFC, such as the various protocols of the TCP / IP suite.

  By using this RTP, time information and a packet number are added to the packet, and the receiving side can display voice and images using the correct time information, or the packet whose order has been changed on the network, etc. Judgment and detection of packet loss by looking at the packet number can be performed.

  In addition, the RTP also has a mechanism (RTCP) for notifying a jitter (delay information), a packet loss rate, and the like as supplementary information from the transmission side and the reception side.

  However, how to use this RTCP information depends on the application and is not determined by the standard.

  In the case of an image, it is not possible to secure a bandwidth for raw image transmission in the network bandwidth, so that it is necessary to send the image signal after being compressed by an encoding method such as MPEG as described above. This is effective in reducing the amount of data, but on the other hand, it becomes very fragile against packet loss and mixing errors caused by flowing data over the unstable Internet. This is because a moving image encoding method transmits only a difference from the previous frame, and a part of the data is lost. When UDP or RTP is used, since data is not basically retransmitted, it is necessary to take measures against this problem.

  By the way, there are usually two modes in image coding. There are an inter-frame coding mode (P Picture) in which the difference from the previous frame is sent, and an intra-frame coding mode (I Picture) in which encoding is performed by closing within one frame. Normally, as shown in FIG. 19, an intra-frame coding mode (I-Pic in the figure) is provided at an appropriate timing, and an inter-frame coding mode (P-Pic in the figure) is provided between them. Provided. The interval at which frames are encoded in this intra-frame encoding mode is called a GOP (Group of Pictures) interval. Here, as shown in FIG. 20, when a decoded image (for example, 1000 in the figure) in which data loss has occurred is broken, it is assumed that encoding is performed in the interframe encoding mode thereafter. Since decoding is performed based on the broken image, all subsequent decoded images are affected and cannot be decoded correctly. Therefore, as shown in FIG. 21, a means for cutting off and recovering the propagation of the influence of an error by inserting a frame (for example, 1001 in the figure) encoded in the intraframe coding mode in the middle is used. .

  In the prior art, it was possible to determine that there was an error in the network, but it was unclear how to use it. In addition, although a mechanism for notifying network information to the transmission side in an original form was considered, it became a specification unique to the application and was not versatile.

  For example, as shown in FIG. 14, the transmission data 133 is received using the reception side information reception unit 1401, and the image encoding process of the image encoding unit 102 is controlled within the range of information received from the network side through this transmission path. Degree.

  In the conventional network state determination, it has been considered assuming network congestion. However, as the Internet spreads into the mobile environment, it becomes necessary to consider errors in the wireless environment. However, such measures are not taken at all with the current technology.

  Furthermore, even when dealing with errors, no consideration has been given to cases where the network status changes and the error rate is not constant. If the GOP interval is set to be narrow in order to cope with a case where there are many errors, encoding in the intra-frame encoding mode increases and encoding efficiency deteriorates. This may be necessary if there are a lot of errors on a regular basis, but in a network where there are few errors in the normal state and errors occur only at a certain moment, this is very large. It becomes useless. On the contrary, when the GOP interval is set to be long, there is a problem that the influence when an error occurs becomes large.

  Therefore, attention is paid to the use of real-time transfer using RTP.

  As described above, by using RTP, which is a protocol for transferring audio data and video data in real time, time information and a packet number are added to the packet, and the receiving side uses the correct time information for voice. Or images can be displayed, packets whose order has been changed in the network can be determined, or the packet number can be detected by looking at the packet number.

  In addition, the RTP has a mechanism (RTCP) for notifying the jitter, the packet loss rate, and the like as supplementary information from the transmission side and the reception side.

  The present invention has been made in consideration of the above circumstances, and based on the notification of jitter, packet loss rate, etc. as supplementary information obtained from the receiving side on the transmitting side, using such RTP features, Even for transmissions that require real-time performance by adjusting the transmission rate according to the transmission status of the transmission path and performing control such as changing error resilience so that data can be transmitted as efficiently as possible. It is an object to provide a data transmission apparatus, a data transmission method, and a program that can be sufficiently used.

  The present invention is a data transmission apparatus for transmitting image data to a receiving side, and selectively supplies one of a plurality of encoded image data encoded with different bit rates for the same image content. Supply means; network status determination means for obtaining an optimum transmission rate for the current transmission path based on at least one of delay information and packet loss rate as supplementary information obtained from the receiving side; and the supply means Selecting means for selecting image encoded data having a bit rate corresponding to the determined optimum transmission rate from among the plurality of image encoded data supplied by the image data, and the image selected and supplied from the supply means Transmitting means for transmitting the encoded data to the receiving side, wherein the network state determining means is the image code selected by the selecting means. Comprising: first storage means for storing the encoding parameter information relating to the encoded data; and second storage means for storing the supplementary information obtained from the receiving side, and the supplementary information obtained from the receiving side and the first Determining a network state from past coding parameter information output from one storage means and past supplemental information obtained from the second storage means, and obtaining an optimum transmission rate for the current transmission path. Features.

  Further, the present invention is a data transmission method in a data transmission apparatus for transmitting image data to a receiving side, and is any one of a plurality of image encoded data encoded with different bit rates for the same image content. Network condition determination to determine the optimum transmission rate for the current transmission path based on the supply step of selectively supplying the information and the notification of at least delay information or packet loss rate as supplementary information obtained from the receiving side A selection step of selecting image encoded data having a bit rate corresponding to the optimum transmission rate obtained from the plurality of image encoded data supplied in the supplying step; A transmission step of transmitting the image encoded data supplied in the step to the reception side, The network state determination step stores the encoding parameter information related to the encoded image data selected in the selection step in the first storage unit, and the supplementary information obtained from the reception side in the second storage unit. A network state based on supplementary information obtained from the receiving side, past coding parameter information output from the first storage means, and past supplementary information obtained from the second storage means And determining the optimum transmission rate for the current transmission path.

  According to the present invention, there is provided a program for causing a computer to function as a data transmission device that transmits image data to a receiving side, wherein the program encodes the same image content with different bit rates. Optimal transmission for the current transmission path based on a supply step for selectively supplying one of the encoded data and at least notification of delay information or packet loss rate as supplementary information obtained from the receiving side A network state determination step for obtaining a rate, and a selection step for selecting image encoded data having a bit rate corresponding to the obtained optimum transmission rate from among the plurality of image encoded data supplied in the supplying step. And the encoded image data selected and supplied in the supplying step And transmitting the transmission step to the reception side, and the network state determination step stores in the first storage means the encoding parameter information related to the image encoded data selected in the selection step. And the step of storing the supplementary information obtained from the reception side in the second storage means, the supplementary information obtained from the reception side and the past encoding parameters output from the first storage means A network state is determined from the information and past supplementary information obtained from the second storage means, and an optimum transmission rate for the current transmission path is obtained.

[1] Further, the present invention provides an image input unit that captures an image, an image encoding unit that encodes an image signal output from the image input unit, and an image encoded data output from the image encoding unit. Transmitting means for transmitting to the network, network information receiving means for receiving network information relating to the network status, and network status determining means for determining the network status from the network information received by the network information receiving means. The image encoding means is controlled by network state information determined by the network state determining means.
[2] The present invention is also characterized in that, in the apparatus of [1], the encoding parameter information determined by the image encoding means is input to the network state determining means.
[3] Further, the present invention provides an encoding parameter information storage unit that stores the encoding parameter information input from the image encoding unit in the network state determination unit of the image transmission apparatus according to the second invention, Network information storage means for storing network information input from the network information reception means, and network information input from the network information reception means and past encoding parameters output from the encoding parameter information storage means The network state is determined from the information and past network information output from the network information storage means.
[4] The present invention further includes an encoding parameter determining unit that determines an encoding parameter from the network state information input from the network state determining unit in the image encoding unit in the apparatus of [1]. Features.
[5] Further, according to the present invention, when the encoding parameter determination means in the apparatus of [4] determines the encoding parameter using the network state information, the next frame is forcibly encoded by intra-frame encoding. Intra-frame encoding determination means for determining whether to encode is provided.
[6] Further, the present invention provides a transmission unit that transmits encoded data to a network, a network information reception unit that receives network information related to a network state, and the network that is received by the network information reception unit. A network status judging means for judging a network status from information; a coded data switching means for selecting and outputting one of a plurality of encoded data input; and a network outputted from the network status judging means And encoding data selection means for selecting encoded data to be transmitted from the transmission means from state information and outputting switching information to the encoded data switching means.
[7] Further, the present invention is the encoded data switching means of the apparatus of [6], further comprising a switching position detecting means for detecting a switchable position of the encoded data, and output from the encoded data selecting means. When an encoded data switching signal is received according to the switching information, the switching position is detected by the switching position detecting means, and the encoded data is switched at the switching position.

  The present invention relating to the apparatus is also established as an invention relating to a method, and the present invention relating to a method is also established as an invention relating to an apparatus.

  Further, the present invention relating to an apparatus or a method has a function for causing a computer to execute a procedure corresponding to the invention (or for causing a computer to function as a means corresponding to the invention, or for a computer to have a function corresponding to the invention. It is also established as a program (for realizing) and also as a computer-readable recording medium on which the program is recorded.

  The present invention uses the characteristics of RTP, and RTP includes a mechanism (RTCP) for notifying delay information (jitter), packet loss rate, and the like as supplementary information from the transmission side or the reception side. Based on notifications such as jitter and packet loss rate as supplementary information obtained from the reception side, the transmission data bit rate on the transmission side is adjusted according to the transmission status of the transmission line, and the error resilience level is changed. It performs control such as.

  Therefore, according to the present invention, it is possible to realize a data transmission that can be efficiently used to a maximum extent and can be sufficiently used for a transmission that requires real-time performance.

  According to the present invention, it is possible to determine a network state and optimally set encoding parameters. Further, when packet loss due to congestion and loss such as a radio error coexist in the network, it is possible to determine this and set parameters.

  Therefore, according to the present invention, a data transmission apparatus and a data transmission method capable of realizing data transmission that enables data transmission to the maximum extent and that can be sufficiently utilized even for transmission that requires real-time performance. Can provide.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  Hereinafter, embodiments of the invention will be described with reference to the drawings.

  The present invention uses RTP and RTCP which are protocols for transferring audio data and video data in real time, and includes jitter (delay information) as supplementary information obtained from the receiving side on the transmitting side. Based on notification of packet loss rate, etc., it is possible to adjust the transmission rate according to the transmission status of the transmission line, and to perform control such as changing error resilience so that data can be transmitted efficiently with maximum efficiency. The details will be described below.

(First embodiment)
FIG. 1 is a basic configuration diagram of an image transmission apparatus according to the first embodiment of the present invention. In the figure, reference numeral 101 denotes an image input unit, and reference numeral 102 denotes an image encoding unit, which encodes an image signal 131 input from the image input unit 101.

  Reference numeral 103 denotes an encoded data transmission unit that processes the encoded data 132 encoded by the image encoding unit 102 into a form suitable for the network and transmits the processed data to the reception side. A network information receiving unit 104 receives the network information 134 sent from the receiving side and outputs it to the network state determining unit 105. The network state determination unit 105 has a function of determining the network state from the network information 135 output from the network information receiving unit 104 and notifying the result to the image encoding unit 102 as network state information 136.

  The image encoding unit 102 encodes the image signal 131 using the network state information 136. This is performed as follows. That is, in the case of RTP and RTCP, a packet loss rate and delay information are given. Thereby, in the network state determination unit 105, if the packet loss rate is not “0” or the delay time is larger than a certain value, some load is applied to the network, and the assumed amount of data cannot be flowed. Judge that Therefore, an instruction (network state information 136) for lowering the bit rate is issued to the image encoding unit 102, and a process of resetting the target bit rate for encoding is performed again.

  In this apparatus having such a configuration, when the image signal 131 is input from the image input unit 101, the input image signal 131 is encoded by the image encoding unit 102. The encoded data 132 encoded by the image encoding unit 102 is input to the encoded data transmission unit 103.

  The encoded data transmission unit 103 processes the encoded data 132 into a form suitable for the network and transmits it to the receiving side. The network information receiving unit 104 receives the network information 134 sent from the receiving side and outputs it to the network state determining unit 105.

  The network state determination unit 105 determines the network state from the network information 135 output from the network information reception unit, and notifies the image encoding unit 102 of the result as network state information 136.

  The image encoding unit 102 encodes the image signal 131 using the notified network state information 136.

  As described above, in the case of RTP and RTCP, a packet loss rate and delay information are given. Thereby, in the network state determination unit 105, if the packet loss rate is not “0” or the delay time is larger than a certain value, some load is applied to the network, and the assumed amount of data cannot be flowed. Judge that Therefore, in such a case, the network state determination unit 105 issues an instruction (network state information 136) to lower the bit rate to the image encoding unit 102, and sets the target bit rate to be low when encoding. Set again.

  FIG. 10 is a flowchart showing a very simple example of the network state determination method in the network state determination unit 105. That is, in step S1001, the packet loss rate of the network information is checked to determine whether it is “0” or not. As a result, if there is a packet loss, the effective bit rate is calculated from the packet loss rate in step S1002. This can be calculated by, for example, the following equation (1).

b '= b * (1-r) (1)
Where b is the current bit rate, b 'is the new bit rate, and r is the packet loss rate.

  In step S1003, the image encoding unit 102 is notified of the parameters (encoding parameters) obtained in step S102. Here, the above parameters are for controlling the rate of encoding processing in the image encoding unit 102 and determining the level of error resilience. In the image encoding unit 102, the encoding processing speed of the input image signal 131 is determined. Is used for adjustments such as changing the bit rate, changing the frame interval, and changing the level of error resilience.

  On the other hand, if the packet loss rate is “0” in step S1001, no parameter is changed in step S1004. This is an example of a determination method, and other cases such as determining the bit rate based on a mathematical expression other than the above equation (1) when determining with a threshold value that is not “0” or other than the determination criterion of step S1001 It is also possible to take the determination method.

  As described above, in the first embodiment, an image input unit that captures an image, an image encoding unit that encodes an image signal output from the image input unit, and an image output from the image encoding unit. Transmitting means for transmitting encoded data to the network; network information receiving means for receiving network information relating to the state of the network; network state determining means for determining the network state from the network information received by the network information receiving means; And the image encoding means is controlled by the network status information determined by the network status determination means.

  Therefore, by using this embodiment, even if the bandwidth of the transmission network is unknown or changes in the middle, the parameters etc. are reset based on the network information on the receiving side, and the parameters suitable for the current network are used. Images can be encoded and transmitted. As a result, even when the network is congested, the bit rate is automatically reduced, and the image quality is reduced, but it is possible to communicate the image without causing a phenomenon such as a delay that occurs or an image being corrupted due to packet loss. It becomes. This is a very effective method for real-time image transmission.

The first embodiment can be implemented by being modified as follows. An example is shown in FIG. FIG. 11 shows an example of a network state determination method in the network state determination unit 105 other than that in FIG. What is characteristic here is that, in the flowchart of FIG. 10, the parameters are set only by the packet loss rate, but in the method of FIG. 11, the delay amount is checked even in the case where there is no packet loss in step S1104. If it is delayed more than a certain value, it is determined that the network is still congested. If it is determined that the network is congested, in step S1105, for example, using the following equation (2), the bit rate is calculated and set as a new parameter.
b ′ = b × {(T _N −T _S ) + (d−d _th )} / (T _N −T _S ) (2)
Here, b is the current bit rate, b 'is the new bit rate, T _N is the current time, T _S is the start time, d is the delay time, d _th is the threshold of the delay time.

  FIG. 2 shows an example in which the configuration of FIG. 1 is developed so that the network state determination unit 105 has a function of generating network state information 136 and is provided to the image encoding unit 102.

  That is, in the case of the configuration illustrated in FIG. 2, the network state determination unit 105 obtains the current parameter of the current image encoding unit 102, the current parameter acquired thereby, and the network information 135. The network state information 136 indicating what parameter settings should be generated is generated and given to the image encoding unit 102.

  In the case of this configuration, it means that the network state determination unit 105 can know the image parameters, that is, the state of the encoding process currently performed in the image encoding unit 102. It is possible to calculate and grasp by This is advantageous when the network information 135 and the parameter information 137 are compared to determine the network status. Then, it is possible to realize a mechanism in which parameters can be changed so as to obtain an optimum bit rate in accordance with the state of the network and can be given to the image encoding unit 102 to adjust the encoding process.

  As described above, the present invention adjusts the transmission according to the transmission state of the transmission line or changes the error tolerance. The transmission state is adjusted by changing the bit rate or changing the frame interval. In response to fluctuations in the quality of the transmission line, the state of the transmission line is changed by switching the error resistance according to the transmission state of the transmission line (for example, changing the error resistance by changing the interval of the MPEG4 synchronization signal). Therefore, it is possible to perform data transmission in such a manner that data transmission can be performed to a maximum extent and efficiently so that even a transmission requiring real-time property can be used sufficiently.

  Another example is shown in FIG.

  In FIG. 3, the configuration of FIG. 2 is further developed, and the network information storage unit 301 that stores the past network information in the network state determination unit 105 and the encoding applied by the current image encoding unit 102. The encoding parameter information storage unit 302 stores parameter information. The network state determination unit 303 knows the past network information and the current encoding parameter from these, determines the temporal transition of the network state from these, determines the optimum parameter to be given to the image encoding unit 102, This is given to the image encoding unit 102.

  According to this configuration, it is possible to obtain a feature that makes it possible to more accurately determine the state of the network by looking at the transition of the network information over time. FIG. 12 shows a flowchart of a network state determination method when this method is used.

  The processing of the network state determination unit 303 will be described with reference to this flowchart. First, the network state determination unit 303 determines whether or not there is a packet loss in step S1201. Then, it is determined whether or not packet loss has occurred in the previous time. As a result, when packet loss has occurred in the previous time, the loss rates are compared. If the current packet loss rate exceeds the previous packet loss rate, it can be determined that the previous correction was not effective.

  It is judged that this means that there is a high possibility that data such as noise is generated on the line and the data is destroyed, rather than the transmission data being over bandwidth such as network congestion. it can. Therefore, the network state determination unit 303 sets parameters that are resistant to errors without changing the bit rate in step S1204, and notifies the image encoding unit 102 in step S1205.

  If the current packet loss rate is lower than the previous packet loss rate in step S1203, the network state determination unit 303 re-establishes the effective bit rate from the packet loss rate in step S1206, assuming that the previous change has worked effectively. calculate. If no packet loss has occurred in the previous step in step S1202, the current loss is determined as line noise, and the process proceeds to step 1204.

  On the other hand, if there is no packet loss in step S1201, the network state determination unit 303 proceeds to step S1207 and checks the delay amount. As a result, when the delay amount exceeds the threshold Th, an effective bit rate is calculated from the delay amount in step S1208, and the image encoding unit 102 is notified in step S1205.

  Finally, the network state determination unit 303 determines that good communication is being performed when the delay amount does not exceed the threshold Th in step S1207, and does not particularly change the parameter in step S1209.

  By causing the network state determining unit 303 to perform such processing, the network state determining unit 105 causes the packet loss occurring on the transmission path to be network congestion or the state of the line such as wireless to be bad and noise to occur. It becomes possible to have a function that can determine whether it is happening.

  This is an example. For example, when there is no packet loss in the previous step S1202, it is not determined as noise, and conversely, it is considered that the network is congested, or the bit rate is reduced, or the previous packet loss is performed in step S1203. If the packet loss rate of this time is improved from the rate, it is possible to investigate what kind of change was made last time and change the same parameter. Thus, in this method, various determination methods can be set according to the network.

  The above is an example in which the image encoding unit 102 performs encoding processing based on the given encoding parameter determined by the network state determination unit 105. However, the image encoding unit 102 is not the network state determination unit 105. The parameter may be determined by An example is shown in FIG.

  FIG. 4 is a block diagram including a configuration when the image encoding unit 102 determines encoding parameters. In this example, the image encoding unit 102 includes an encoding parameter determination unit 401 and a signal processing unit 402.

  In the case of this configuration, the network state determination unit 105 outputs the network state information 136 from the network state determination unit 303 and supplies the network state information 136 to the image encoding unit 102.

  In such a configuration, the network state determination unit 105 gives the network state information 136 output from the network state determination unit 303 to the image encoding unit 102. Then, in the image encoding unit 102, the network state information 136 is first input to the encoding parameter determination unit 401, and the encoding parameter determination unit 401 knows the network state from the network state information 136 and has a form adapted to the network state. The encoding parameter 431 is generated.

  The generated encoding parameter 431 is input to the signal processing unit 402. Then, the signal processing unit 402 encodes the image signal 131 input from the image input unit 101 using the encoding parameter 431.

  Encoding information 432 such as the amount of code after encoding is input to the encoding parameter determination unit 401 and is used when determining the next encoding parameter. The encoded data 132 encoded by the signal processing unit 402 is output to the encoded data transmission unit 103.

  FIG. 5 is a block diagram illustrating a configuration including an intra-frame coding determination unit 501 for forcibly setting intra-frame coding in the coding parameter determination unit 401 illustrated in FIG. As shown in the figure, the encoding parameter determination unit 401 includes an intra-frame encoding determination unit 501 and an encoding parameter determination unit 502.

  In the case of this configuration, the network state information 136 output from the network state determination unit 105 is input to the encoding parameter determination unit 502 included in the encoding parameter determination unit 401, and the network state information 136 is input from the encoding parameter determination unit 502. Corresponding encoding parameter information 531 is output.

  The encoding parameter information 531 is input to the intra-frame encoding determination unit 501 and determines whether to perform inter-frame encoding as it is or to force intra-frame encoding. If it is determined that the coding is intraframe coding, the coding parameter information 531 is updated so as to perform the intraframe coding, and the signal processing unit 402 of the image coding unit 102 in FIG. Is output.

  In this way, even if the previous data is not correctly transmitted to the receiving side, a correct image can be reproduced if it can be received correctly after the parameter change.

  It is also possible to configure an image encoding unit capable of determining a network state having both functions of the image encoding unit 102 and the network state determination unit 105, and to have the functions described above in one block. .

  The encoding parameter determination unit 401 and the network state determination unit 105 make a determination related to the next encoding parameter setting. On the other hand, if the image quality changes suddenly due to temporary instability of the network, the video is difficult to see. It may become. In order to suppress such a phenomenon, it is possible to include a mechanism for suppressing the fluctuation of the encoding parameter within a certain range, and it functions effectively in the above situation.

  The bit rate setting and error resilience parameter setting may be set as calculated. For example, a pattern may be determined in advance, and the closest one may be selected. Is possible. Also, it is possible to use a method such as selecting the closest bit rate that is equal to or lower than the calculated bit rate instead of the closest one. As a result, it is possible to prevent an image with unexpected image quality from being produced by various parameter combinations, and it is possible to prepare combinations of encoding parameters that have been tested to some extent.

  In the above example, when the packet loss rate is not “0” or the delay exceeds the threshold, the parameters such as the bit rate and error resilience are changed, and control is performed so as to reset the parameters to the current conditions. I went.

  However, if the network condition recovers and packet loss does not occur or the delay becomes small, it is possible to increase the bit rate or weaken the error resilience parameter. .

  Such an example will be described next as a second embodiment.

(Second Embodiment)
FIG. 6 is a basic configuration diagram of an image transmission apparatus according to the second embodiment of the present invention. In FIG. 6, 601 is a storage medium, 602 is an encoded data switching unit, 603 is an encoded data selecting unit, 103 is an encoded data transmitting unit, 104 is a network information receiving unit, and 105 is a network state determining unit. It is.

  Of these, the storage medium 601 is for storing encoded data 631 of content encoded in advance, and there are a plurality of storage media. Each storage medium 601 stores encoded data of the same content encoded with different encoding parameters.

  The encoded data switching unit 602 selects one of the plurality of storage media 601 and supplies the encoded data 631 held in the selected storage medium 601 to the encoded data transmission unit 103. The storage medium 601 is selectively switched, and this selection is performed according to switching information 632 from the encoded data selection unit 603.

  The network information receiving unit 104 receives transmission data from the receiving side, receives network information 134 such as packet loss rate and delay information sent from the receiving side and the network side, and receives the network information 134 from the network state. The determination unit 105 has a function of giving it as network information 135.

  The network state determination unit 105 determines the state of the network from the input network information 135 and outputs the result as the network state information 136 to the encoded data selection unit 603. The encoded data selection unit 603 The network state information 136 is used to estimate which encoding data of the encoding parameter is optimal for the current network, and to switch to select the encoded data based on the estimated optimal encoding parameter as an output. It has a function of outputting the switching information 632 to the encoded data switching unit 602.

  The encoded data switching unit 602 selects one encoded data from the input encoded data 631 based on the switching information 632 and outputs the encoded data 132.

  In such a configuration, encoded data 631 that has been encoded and stored in the storage medium 601 or the like in advance is input to the encoded data switching unit 602 when reproduced from the storage medium 601. That is, each of the plurality of storage media 601 holds respective encoded data encoded with different encoding parameters, and as a result of reproducing these, the encoded data switching unit 602 has different encoding data. Encoded data encoded with parameters is input. Then, the encoded data switching unit 602 selects one of them and outputs it to the encoded data transmission unit 103.

  This selection is performed according to the switching information 632 from the encoded data selection unit 603.

  On the other hand, network information 134 such as packet loss rate and delay information from the receiving side is received by the network information receiving unit 104 and sent to the network state determining unit 105. The network state determination unit 105 determines the network state from the input network information 135 and outputs the result to the encoded data selection unit 603 as the network state information 136.

  The encoded data selection unit 603 infers from the network state information 136 which encoding parameter encoding data is optimal for the current network, and outputs the switching information 632 to the encoded data switching unit 602.

  The encoded data switching unit 602 selects one encoded data from the input encoded data 631 based on the switching information 632 and outputs the encoded data 132.

  This embodiment will be described with a specific example. Now, as the storage medium 601, for example, five units in which five encoded data encoded at 384 [kbps], 128 [kbps], 64 [kbps], 32 [kbps], and 16 [kbps] are stored are stored. It is assumed that the storage media 601a to 601e exist.

  That is, for example, the storage medium storing the encoded data of the content encoded at 384 [kbps] is the storage medium 601a, and the storage medium storing the encoded data of the content encoded at 128 [kbps] is the storage medium The storage medium storing the encoded data of the content encoded at 601b and 64 [kbps] is the storage medium 601c, and the storage medium storing the encoded data of the content encoded at 32 [kbps] is the storage medium 601d, A storage medium 601e is a storage medium that stores encoded data of content encoded at 16 [kbps].

  When a transmission path having a quality that can be transmitted at 384 [kbps] is used, first, the reproduction encoded data by the storage medium 601a which is a storage medium storing the encoded data of the content encoded at 384 [kbps] is selected. Then, transmission is started at 384 [kbps], and reception of network information from the receiving side is also started at the same time.

  Network information is received at a certain interval, but an effective bit rate is calculated when it is found that the information has some trouble in the network and packet loss or delay occurs. Then, encoded data having a bit rate close to the calculated bit rate is selected by the encoded data selection unit 603, switched by the encoded data switching unit 602, and transmitted.

  For example, if the calculated bit rate is 140 [kbps], the value close to this bit rate is 128 [kbps]. In this case, the encoded data of the content encoded at 128 [kbps] If the calculated bit rate is 100 [kbps] and the calculated bit rate is 100 [kbps], the bit rate below this is 64 [ kbps], and in this case, the encoded data reproduced from the storage medium 601c storing the encoded data of the content encoded at 64 [kbps] is selected and switched. .

  As described above, in the present embodiment, contents are prepared by encoding in advance corresponding to the type of transmission bit rate, and network information such as packet loss rate and delay information from the receiving side, which is a function of RTP. The current optimum transmission rate is calculated based on the transmission rate, and the encoded data that obtains the bit rate corresponding to this transmission rate is selectively switched to the reproduced encoded data output of the storage medium that is stored and transmitted. Thus, it is not necessary to perform real-time encoding, and the load on the server can be reduced. This is a very effective means for distributing pre-made videos such as broadcasts.

  Here, a modification of the second embodiment will be described. This is a configuration as shown in FIG. 7, in which a block diagram of a method for controlling the switching timing of input encoded data in the encoded data switching unit 602 is shown.

  The encoded data 631 is input to the switching position detection unit 701 and the switching unit 702. In the switching position detection unit 701, switching information 632 is input. When the switching information 632 makes it necessary to perform switching from the current encoded data to another encoded data, the switching position detection unit 701 analyzes the encoded data and detects a switchable position.

  This means, for example, searching for a frame encoded by intra-frame encoding (I-Picture).

  When the switchable position is reached, the switching unit 702 is instructed to switch the encoded data using the switching instruction information 731. FIG. 8 is a diagram showing this state. Assume that the A encoded data is currently selected, and it is necessary to switch to the B encoded data by the switching information 632.

  At that time, it is assumed that the frame at the time of “switching instruction” in FIG. 8 has been processed. In that case, if the encoded data is switched here, the image will be inconsistent. This is because, in inter-frame coding (P-Picture), the difference from the previous image is coded. Therefore, if the encoded data is switched at the time of P-Picture, the next image (the image P11 of the encoded data (B)) is used using the immediately preceding image (the decoded image I11 of the encoded data (A)). Will end up trying to play.

  In this case, there is a problem that a correct decoded image cannot be obtained and the image is broken.

  Therefore, it is necessary to perform switching at the timing of an intra-frame coding (I-Picture) image in which coding is performed only on the frame, not on a difference from the previous image. This is the timing at the “switch execution” mark position in FIG.

  In addition, as shown in FIG. 9, the number of frames and the time position may be different for each encoded data. At that time, similarly, the switchable position of the encoded data of the switching destination is detected, and the encoded data is switched when it reaches the possible position.

  In the switching position detection unit 701 in FIG. 7, when the switching information 632 needs to be switched, the output 132 to the encoded data transmission unit 103 is temporarily stopped, and a new encoding is performed when the switching position comes. A method of outputting data to the encoded data transmission unit 103 is also possible.

  This is effective when it is necessary to suppress buffer overflow or the like by switching encoded data. Conversely, if underflow becomes a problem, it can be dealt with by forcibly inserting a stuffing bit or the like.

(Third embodiment)
Next, a third embodiment will be described.

  In the present embodiment, in the encoding parameter determination unit 401 of the image encoding unit 102 of FIG. 4 of the first embodiment, based on the network state information 136 output from the network state determination unit 105, the intraframe encoding mode is used. A configuration example when the encoding time interval (GOP interval) is adjusted will be described. Here, the description will focus on the differences from the first embodiment.

  FIG. 15 shows a configuration example of the encoding parameter determination unit 401 in this case. As described above, the GOP interval calculation unit 1501 and the encoding parameter determination unit 502 are provided.

  The network state information 136 output from the network state determination unit 105 (here, at least the packet loss rate information is included) is input to the encoding parameter determination unit 502. The parameter determination unit 502 inputs the packet loss rate information 1531 of the network state information 136 to the GOP interval calculation unit 1501. The GOP interval calculation unit 1501 calculates the GOP interval from the packet loss rate information 1531. The GOP interval information 1532 is notified to the encoding parameter determination unit 401. The encoding parameter determination unit 401 outputs the encoding parameter information 431 including the input GOP interval information 1532 (in the encoding parameter determination unit 401, if there is internally generated parameter information, the GOP interval The encoded parameter information 431 including the information 1532 and the parameter information generated therein is output).

  FIG. 16 is a flowchart showing an example of a method for determining the GOP interval in the GOP interval calculator.

In step S1601, it is determined from the network state information 136 whether a packet loss has occurred. If packet loss has occurred, the GOP interval is estimated from the following equation (3) in step S1602, for example.
gop = (T _N −T _L ) / {(F _N −F _L ) × r} (3)
Here, gop the GOP interval, r is the packet loss rate, F _N is the current total number of frames, F _L is total number of frames in the previous calculation, T _N is the current time, T _L is the time of the previous calculation is there.

  The GOP interval obtained in step S1602 is notified to the signal processing unit 402 in step S1603, and encoding is performed based on this value.

  On the other hand, if no packet loss has occurred in step S1601, the default GOP interval value is read in step S1604. In step S1603, the signal processing unit 402 is notified of this value.

  According to the present embodiment, the GOP interval can be dynamically changed to a value suitable for the network. As a result, when there are few errors, it is possible to widen the GOP interval and reduce the useless intra-frame coding mode. On the other hand, when there are many errors, the GOP interval is narrowed and the frame in the intra-frame coding mode appears earlier so that the recovery can be accelerated. In the present embodiment, it is possible to select an efficient intraframe coding mode.

  FIG. 17 is a flowchart showing another example of a method for determining the GOP interval in the GOP interval calculator.

  In this example, even when there is no packet loss compared with the method of FIG. 16, the network state is estimated from the past history of packet loss in step S1704, and the optimum GOP interval is calculated. This method does not immediately return the GOP interval to the default value as soon as there is no packet loss at a certain moment, but examines the past history and whether the network is completely error-free, or an error still occurs It is possible to estimate whether there is a possibility of doing. This makes it possible to more accurately determine the state of the network.

  FIG. 18 is a flowchart showing still another example of the method for determining the GOP interval in the GOP interval calculator.

  In this example, the GOP interval is not determined based on the presence / absence of packet loss, but is determined by a unique calculation method. This can be used when the GOP interval is determined by one equation from the current packet loss rate, or when the most suitable GOP interval is determined by examining past history.

The following equation (4) shows an example of a calculation equation for determining the GOP interval.
gop = [(T _N −T _L ) / {(F _N −F _L ) × r}] × α (4)
Here, gop is the GOP interval, r is the packet loss rate, α is the sensitivity coefficient, F _N is the current total number of frames, F _L is the total number of frames at the previous calculation, T _N is the current time, and T _L is the previous time The time of calculation.

  In this example, by multiplying the GOP interval calculated from the actual value by the sensitivity coefficient, if there is an error, there is a request to recover quickly at the expense of efficiency, or a request that the influence of the error may be somewhat prolonged. Etc. can be satisfied. For example, when the sensitivity coefficient α is set to be smaller than 1, a GOP interval shorter than the GOP interval calculated from the actual measurement value is output. This sacrifices efficiency, but allows for a much faster recovery if an error occurs.

  In addition, this embodiment is not limited to Formula (3) and (4). For example, these formulas calculate the value using an increment from the previous calculation, but it is also possible to calculate by taking a width such as an increment from the previous n times. As a result, it is possible to cope with a gradual change hidden in a fine change in state or to remove a temporary change to some extent. Similarly, the algorithm is not limited to that shown in this embodiment.

  In the above description, the encoding parameter determination unit 401 of the image encoding unit 102 of FIG. 4 of the first embodiment adjusts the time interval (GOP interval) for encoding in the intraframe encoding mode based on the network state information 136. In the network state determination unit 105 (FIG. 1, FIG. 2, or FIG. 3) of the first embodiment, information from the network information reception unit 104 or from the network information reception unit 104 has been described. A time interval (GOP interval) for encoding in the intra-frame encoding mode is determined based on the information of the above and the information from the image encoding unit 102, and may be provided to the image encoding unit 102. .

  In addition, as described in the first embodiment, an image encoding unit capable of determining the network state having both functions of the image encoding unit 102 and the network state determining unit 105 is configured, and the above-described image encoding unit is included in one block. It is also possible to have the functions as described above.

  Although various embodiments have been described above, in short, the present invention uses the characteristics of RTP, and the RTP is notified of jitter, packet loss rate, and the like as supplementary information from the transmission side and the reception side ( RTCP), and adjusts the bit rate of transmission data on the transmission side according to the transmission state of the transmission line based on notifications such as jitter and packet loss rate as supplementary information obtained from the reception side on the transmission side And control such as changing the error resilience level. Therefore, according to the present invention described above, it is possible to realize data transmission that can be efficiently transmitted to the maximum extent and can be sufficiently used for transmission that requires real-time performance.

  The present invention is not limited to the above-described embodiment, and can be implemented with various modifications. In the present invention, the above embodiment includes inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiment, at least one of the problems described in the column of the problem to be solved by the invention can be solved, and is described in the column of the effect of the invention. When at least one of the effects is obtained, a configuration in which this configuration requirement is deleted can be extracted as an invention.

  In addition, the method described in the embodiment of the present invention uses a magnetic disk (flexible disk, hard disk, etc.), optical disk (CD-ROM, CD-R, CD-RW, DVD, MO) as programs that can be executed by a computer. Etc.), can be stored and distributed in a recording medium such as a semiconductor memory, or can be distributed by transmission via a network.

The figure which shows the basic structural example of the 1st Embodiment of this invention. The figure which shows the example of 1 structure in the embodiment The figure which shows the structural example of the network state determination part in the embodiment The figure which shows the structural example of the image coding part in the embodiment The figure which shows the structural example of the encoding parameter determination part in the embodiment The figure which shows the example of a basic composition in the 2nd Embodiment of this invention. The figure which shows the structural example of the encoding data switching part in the embodiment The figure for demonstrating the encoding data switching timing in the encoding data switching part in the embodiment The figure for demonstrating the switching timing of the encoding data from which the frame interval differs in the encoding data switching part in the embodiment The flowchart which shows the basic example of the network state determination method in the network state determination part in the 1st Embodiment of this invention. The flowchart which shows the basic example which made the delay correspondence of the network state determination method in the network state determination part in the embodiment The flowchart which shows the basic example which considered the time direction fluctuation | variation with the network state determination method in the network state determination part in the embodiment The figure which shows the structure of the conventional image transmission apparatus. The figure which shows the structure of the image transmission apparatus using the conventional original network information The figure which shows the structural example which concerns on the 3rd Embodiment of this invention. The flowchart which shows an example of the GOP space | interval calculation method in the GOP space | interval calculation part in the embodiment The flowchart which shows the other example of the GOP space | interval calculation method in the GOP space | interval calculation part in the embodiment The flowchart which shows the further another example of the GOP space | interval calculation method in the GOP space | interval calculation part in the embodiment. Diagram for explaining GOP structure and GOP interval The figure for demonstrating the influence on the decoded image by an error The figure for demonstrating the influence on the decoded image by the error at the time of making GOP space | interval small.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 101 ... Image input part, 102 ... Image encoding part, 103 ... Encoded data transmission part, 104 ... Network information receiving part, 105 ... Network state determination part, 131 ... Image signal, 132 ... Encoded data, 133 ... Transmission data , 134 ... received data, 135 ... network information, 136 ... network state information, 137 ... encoding parameter information, 301 ... network information storage unit, 302 ... encoding parameter storage unit, 303 ... network state determination unit, 331 ... network information 332... Encoding parameter information, 401... Encoding parameter determination unit, 402... Signal processing unit, 431... Encoding parameter information, 432. , 531 ... encoding parameter information, 601 ... storage medium, 6 2 ... Encoded data switching unit, 603 ... Encoded data selection unit, 631 ... Encoded data, 632 ... Switching information, 701 ... Switching position detection unit, 702 ... Switching unit, 731 ... Switching instruction information, 1501 ... GOP interval calculation Part

Claims (9)

A data transmission device for transmitting image data to a receiving side,
Supply means for selectively supplying any one of a plurality of encoded image data encoded with different bit rates for the same image content;
Based on the notification of at least delay information or packet loss rate as supplementary information obtained from the receiving side, network state determination means for obtaining an optimal transmission rate for the current transmission path;
Selecting means for selecting image encoded data having a bit rate corresponding to the optimum transmission rate obtained from the plurality of image encoded data supplied by the supplying means;
A transmission unit that transmits the encoded image data selected and supplied from the supply unit to a reception side;
The network state determination unit includes a first storage unit that stores encoding parameter information related to the encoded image data selected by the selection unit, and a second storage unit that stores supplementary information obtained from the receiving side. And
The network state is determined from the supplementary information obtained from the receiving side, the past encoding parameter information output from the first storage means, and the past supplementary information obtained from the second storage means, A data transmission apparatus characterized by obtaining an optimum transmission rate for a transmission line. The selection means includes
GOP interval calculation means for calculating the GOP interval from the supplementary information obtained from the receiving side is further provided.
The data transmission apparatus according to claim 1, wherein selection of image encoded data is performed in accordance with GOP interval information indicating a GOP interval output from the GOP interval calculating means. The network state determining means determines at least a packet loss rate as the state of the network;
3. The data transmission apparatus according to claim 2, wherein the GOP interval calculation unit obtains the GOP interval based on at least the packet loss rate. A data transmission method in a data transmission apparatus for transmitting image data to a receiving side,
A supply step of selectively supplying any one of a plurality of encoded image data encoded with different bit rates for the same image content;
A network state determination step for obtaining an optimum transmission rate for the current transmission path based on at least notification of delay information or packet loss rate as supplementary information obtained from the receiving side;
A selection step of selecting image encoded data having a bit rate corresponding to the optimum transmission rate obtained from the plurality of encoded image data supplied in the supplying step;
A transmission step of transmitting the image encoded data selected and supplied in the supply step to a reception side;
The network state determination step includes a step of storing the encoding parameter information relating to the encoded image data selected in the selection step in a first storage unit, and a supplementary information obtained from the receiving side as a second storage unit And storing in the
The network state is determined from the supplementary information obtained from the receiving side, the past encoding parameter information output from the first storage means, and the past supplementary information obtained from the second storage means, A data transmission method characterized by obtaining an optimum transmission rate for a transmission line. The selection step includes
GOP interval calculation second storage means for calculating the GOP interval from the supplementary information obtained from the receiving side,
5. The data transmission apparatus according to claim 4, wherein selection of image encoded data is performed according to GOP interval information indicating the GOP interval output in the GOP interval calculation step. The network state determining step determines at least a packet loss rate as the state of the network,
6. The data transmission apparatus according to claim 5, wherein the GOP interval calculation step obtains the GOP interval based on at least the packet loss rate. In a program for causing a computer to function as a data transmission device that transmits image data to a receiving side,
The program is
A supply step of selectively supplying any one of a plurality of encoded image data encoded with different bit rates for the same image content;
A network state determination step for obtaining an optimum transmission rate for the current transmission path based on at least notification of delay information or packet loss rate as supplementary information obtained from the receiving side;
A selection step of selecting image encoded data having a bit rate corresponding to the optimum transmission rate obtained from the plurality of encoded image data supplied in the supplying step;
A transmission step of transmitting the encoded image data selected and supplied in the supply step to the receiving side, and
The network state determination step includes a step of storing the encoding parameter information relating to the encoded image data selected in the selection step in a first storage unit, and a supplementary information obtained from the receiving side as a second storage unit And storing in the
The network state is determined from the supplementary information obtained from the receiving side, the past encoding parameter information output from the first storage means, and the past supplementary information obtained from the second storage means, A program for obtaining an optimum transmission rate for a transmission line. The selection step includes
GOP interval calculation second storage means for calculating the GOP interval from the supplementary information obtained from the receiving side,
The program according to claim 7, wherein the encoded image data is selected according to GOP interval information indicating the GOP interval output in the GOP interval calculation step. The network state determining step determines at least a packet loss rate as the state of the network,
9. The program according to claim 8, wherein the GOP interval calculation step obtains the GOP interval based on at least the packet loss rate.

Images