[go: nahoru, domu]

CN110581812A - Data message processing method and device - Google Patents

Data message processing method and device Download PDF

Info

Publication number
CN110581812A
CN110581812A CN201810587646.2A CN201810587646A CN110581812A CN 110581812 A CN110581812 A CN 110581812A CN 201810587646 A CN201810587646 A CN 201810587646A CN 110581812 A CN110581812 A CN 110581812A
Authority
CN
China
Prior art keywords
fragment
field
message
identification field
messages
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201810587646.2A
Other languages
Chinese (zh)
Inventor
韩建飞
刘海锋
鲍永成
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Jingdong Shangke Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to CN201810587646.2A priority Critical patent/CN110581812A/en
Publication of CN110581812A publication Critical patent/CN110581812A/en
Pending legal-status Critical Current

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L49/00Packet switching elements
    • H04L49/90Buffering arrangements
    • H04L49/9057Arrangements for supporting packet reassembly or resequencing
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/16Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
    • H04L69/166IP fragmentation; TCP segmentation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00Traffic control in data switching networks
    • H04L47/10Flow control; Congestion control
    • H04L47/12Avoiding congestion; Recovering from congestion
    • H04L47/125Avoiding congestion; Recovering from congestion by balancing the load, e.g. traffic engineering

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The invention discloses a method and a device for processing a data message, and relates to the technical field of computers. One embodiment of the method comprises: receiving each data message sent by a client, wherein each data message comprises an identification field and a mark field; identifying the fragment message from each data message according to the mark field; for any identification field, when all the fragment messages with the identification field are received, recombining all the fragment messages with the identification field into a complete data message; and segmenting the complete data message according to the maximum transmission unit value of the network, and forwarding the segmented complete data message to a corresponding back-end service, wherein the back-end service corresponding to the complete data message is selected according to a preset model. The embodiment can ensure that all data messages are still forwarded to the same back-end service after the UDP large packet exceeding the MTU passes through the load balancer, thereby ensuring the load balancing normal operation of the UDP large packet.

Description

Data message processing method and device
Technical Field
the present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for processing a data packet.
background
the load balancing is to uniformly distribute the requests sent from the outside to each server of an application server set by a load balancing sharing technology, thereby realizing the uniform distribution and distributed response of the external request flow and solving the problem of massive concurrent access service.
The existing load balancing scheme is based on a commonly used software LVS (Linux virtual server), and the LVS as a four-layer load balancing does not support a UDP (user Datagram protocol) under a FULLNAT (network Address translation) model. In the traditional load balancing of the TCP (transmission control protocol) or the HTTP protocol (hypertext transfer protocol), because the TCP has a protocol stack guarantee, it is ensured that MSS (maximum network transmission data volume) for packet transmission is basically negotiated in one MTU (maximum transmission unit), and thus IP (internet protocol address) fragmentation is not generated, and when the length of UDP is greater than 1500(1500 is the MTU (maximum transmission unit) value of a common ethernet), a plurality of IP fragmentation is generated, and a main UDP packet carries IP + Port information (i.e., source IP Port information), but the fragmentation packet does not carry Port-related information, so that under the IP + Port Hash (Hash) model of fullmat, there is not one backend server calculated after Hash of the main UDP packet, and thus load balancing abnormality is caused.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:
Existing load balancers do not support such UDP packet (i.e., UDP large packet) load balancing techniques that exceed the MTU.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for processing a data packet, so that after a UDP packet exceeding an MTU passes through a load balancer, all data packets are still forwarded to a same backend service, thereby ensuring that the load balancing of the UDP packet normally operates.
to achieve the above object, according to an aspect of the embodiments of the present invention, a method for processing a data packet is provided.
a method for processing data messages comprises the following steps: receiving each data message sent by a client, wherein each data message comprises an identification field and a mark field; identifying the fragment message from each data message according to the mark field; for any identification field, when all the fragment messages with the identification field are received, recombining all the fragment messages with the identification field into a complete data message; and segmenting the complete data message according to the maximum transmission unit value of the network, and forwarding the segmented complete data message to a corresponding back-end service, wherein the back-end service corresponding to the complete data message is selected according to a preset model.
Optionally, after the step of identifying the fragment packet from each data packet according to the flag field, the method includes: and respectively taking the identification field of each fragment message as a main key, and storing each fragment message in a preset memory area.
optionally, the step of reassembling all the fragmented packets with the identification field into a complete data packet when all the fragmented packets with the identification field are received completely includes: and when all the fragment messages with the identification fields are received, recombining all the fragment messages in the preset memory area by using the identification fields as main keys into a complete data message.
Optionally, the data packet further includes a fragment offset field, and it is determined that all fragment packets having an identification field are received by the following method: determining that the last fragmented message with the identification field is received completely, and acquiring a fragment offset field value of the last fragmented message; acquiring the number of received fragment messages with the identification field, wherein the received fragment messages are all fragment messages in a preset memory region with the identification field as a main key; obtaining the total number of the fragmented messages with the identification field according to the fragment offset field value of the last fragmented message; and if the number of the received fragment messages is consistent with the total number of the fragment messages, determining that all the fragment messages with the identification field are received completely.
Optionally, the flag field includes an MF field, where the MF field is used to indicate whether there is a subsequent fragment in the fragment packet, and it is determined that the last fragment packet with an identification field is received through the following method: if a certain fragmented message with the identification field is received, and the MF field value of the fragmented message indicates that the fragmented message has no subsequent fragmentation, the last fragmented message with the identification field is determined to be received completely.
Optionally, if the number of the received fragment packets is inconsistent with the total number of the fragment packets, then: setting timing time, and retrying the operation of determining that all the fragmentation messages with the identification field are received completely when the timing time is reached; and if the number of the received fragment messages is not consistent with the total number of the fragment messages when the retry times reach a preset threshold value, deleting all the fragment messages with the identification field in the preset memory area so as to discard all the fragment messages.
Optionally, the preset model is a source IP port hash model or a polling hash model.
Optionally, the data packet is a UDP/IP packet.
according to another aspect of the embodiments of the present invention, a device for processing a data packet is provided.
a device for processing data packets, comprising: the receiving module is used for receiving each data message sent by the client, and each data message comprises an identification field and a mark field; the identification module is used for identifying the fragment message from each data message according to the mark field; a restructuring module, configured to, for any one of the identifier fields, when all the fragmented packets having the identifier field are received completely, restructure all the fragmented packets of the identifier field into a complete data packet; the selecting module is used for selecting the back-end service corresponding to the complete data message according to a preset model; and the forwarding module is used for segmenting the complete data message according to the maximum transmission unit value of the network and forwarding the segmented complete data message to the back-end service.
Optionally, the system further comprises a storage module, configured to: and respectively taking the identification field of each fragment message as a main key, and storing each fragment message in a preset memory area.
Optionally, the restructuring module is further configured to: and when all the fragment messages with an identification field are received, recombining all the fragment messages taking the identification field as a main key in the preset memory area into a complete data message.
Optionally, the data packet further includes a slice offset field, and the apparatus further includes a checking module configured to: determining that the last fragmented message with an identification field is received completely, and acquiring a fragment offset field value of the last fragmented message; acquiring the number of received fragment messages with the identification field, wherein the received fragment messages are all fragment messages in a preset memory region with the identification field as a main key; obtaining the total number of the fragmented messages with the identification field according to the fragment offset field value of the last fragmented message; and if the number of the received fragment messages is consistent with the total number of the fragment messages, determining that all the fragment messages with the identification field are received completely.
Optionally, the flag field includes an MF field, where the MF field is used to indicate whether a subsequent fragment exists in the fragment packet, and the checking module includes a packet checking sub-module, configured to: if a certain fragmented message with an identification field is received, and the MF field value of the fragmented message indicates that the fragmented message has no subsequent fragmentation, the last fragmented message with the identification field is determined to be received completely.
Optionally, a retry module is further included for: if the number of the received fragment messages with an identification field is not consistent with the total number of the fragment messages with the identification field, then: setting timing time, and retrying the operation of determining that all the fragmentation messages with the identification field are received completely when the timing time is reached; and if the number of the received fragment messages is not consistent with the total number of the fragment messages when the retry times reach a preset threshold value, deleting all the fragment messages with the identification field in the preset memory area so as to discard all the fragment messages.
Optionally, the selecting module is further configured to: and selecting the back-end service corresponding to the complete data message according to a source IP port hash model or a polling hash model.
Optionally, the data packet is a UDP/IP packet.
According to yet another aspect of an embodiment of the present invention, an electronic device is provided.
An electronic device, comprising: one or more processors; the memory is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors realize the data message processing method provided by the invention.
According to yet another aspect of an embodiment of the present invention, a computer-readable medium is provided.
a computer-readable medium, on which a computer program is stored, which, when executed by a processor, implements the method for processing data packets provided by the present invention.
One embodiment of the above invention has the following advantages or benefits: identifying the fragment message from each data message according to the mark field of the data message; for any identification field, when all the fragment messages with the identification field are received, recombining all the fragment messages with the identification field into a complete data message; and segmenting the complete data message according to the maximum transmission unit value of the network, and forwarding the segmented complete data message to the corresponding back-end service, wherein the back-end service corresponding to the complete data message is selected according to a preset model. After the UDP large packets exceeding the MTU pass through the load balancer, all the data messages are still forwarded to the same back-end service, and the load balancing normal operation of the UDP large packets is guaranteed.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
the drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
Fig. 1 is a schematic diagram illustrating main steps of a data packet processing method according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating key fields of an IP packet format according to an embodiment of the present invention;
FIG. 3 is a flow diagram illustrating a preferred implementation of data message processing according to an embodiment of the present invention;
Fig. 4 is a schematic diagram of main modules of a data packet processing apparatus according to an embodiment of the present invention;
FIG. 5 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
FIG. 6 is a schematic block diagram of a computer system suitable for use with a server implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram illustrating main steps of a data packet processing method according to an embodiment of the present invention
As shown in fig. 1, the method for processing a data packet according to the embodiment of the present invention mainly includes the following steps S101 to S105. The method for processing the Data packet is applied to a four-layer load balancer, and this embodiment can be realized based on a DPDK, and the DPDK, i.e., a Data Plane Development Kit, is a tool for quickly receiving and transmitting a packet, aims at quickly processing a Data packet, and can greatly improve Data processing performance and throughput. Besides DPDK, other fast package receiving and sending tools may be used in the embodiments of the present invention. The data message in the embodiment of the invention is a UDP/IP message, and the embodiment of the invention can also be called as a UDP message.
Step S101: and receiving each data message sent by the client.
Each data packet includes an identification field and a flag field.
Step S102: and identifying the fragment message from each received data message according to the mark field.
A fragmented packet is a data packet in a fragmented form. Fragmentation is IP fragmentation, which is a technical means for transmitting IP packets over a network. When the IP protocol transmits a data packet, the data packet is divided into a plurality of fragments for transmission, and the fragments are recombined in a system for receiving the packet. Different link types specify link layer data frames of different maximum lengths, called link layer MTUs (maximum transmission units). The MTU of the common ethernet is 1500, and if the length of the IP packet is greater than the MTU of the forwarding interface when the IP protocol is transmitting a data packet, the data packet is divided into a plurality of fragments for transmission, and the fragments are reassembled by the receiver when the fragments reach the receiver.
The data packet received from the client may or may not be in a fragmented form. The 32 bits in the IP packet format are the key fields used in the embodiment of the present invention to process the UDP load balancing identifier, which is specifically shown in fig. 2. In fig. 2, the identification (identification) is 16 bits, and is a counter used to generate the identification field of the data packet. A flag (flag) is 3 bits, and only the first two bits are significant at present, wherein the lowest bit of a flag field is MF (more fragment), the MF field is used to indicate whether the located fragment message has subsequent fragments, MF ═ 1 indicates that the located fragment message is followed by "further fragments", MF ═ 0 indicates that the located fragment message is the last fragment, that is, there is no subsequent fragment; one bit in the middle of the flag field is DF (Don't Fragment) for indicating whether fragmentation is allowed for the data packet in which the flag field is located, and fragmentation is allowed only when DF is equal to 0. The slice offset (offset) is 13 bits, the slice offset field indicates the offset length of the data packet, and among multiple slice packets with the same identifier field, the offset of the first slice packet is 0, the offset of the second slice packet is 1, which are accumulated one by one, and assuming that there are 5 slice packets in total, the offset of the fifth slice packet is 4.
and identifying the fragment message according to the DF field value of the mark field of the received data message. That is, DF is 0, which indicates that the datagram text itself is in a fragment form; DF is 1, which means that the data packet itself is not in a fragmented form.
For a data packet that is not in a fragmentation form (that is, DF is 1), in the embodiment of the present invention, according to the existing DPDK fullmat forwarding flow, a backend service is selected according to a source IP port hash model or a polling hash model, and the data packet is forwarded to the selected backend service. The source IP port information is port information of a client that sends a data packet, and is carried by one of the fragmentation packets, which is called a main UDP packet.
After step S102, the identifier field of each segment message may be used as a primary key to store each segment message in a preset memory area. The preset memory area is an applied memory which is specially used for storing the IP fragment message, and when the IP fragment message arrives, the preset memory area is stored in the memory area. That is, if DF is 0, it indicates that the data packet is a fragment packet, the identification bit (i.e. the identification field) in the fragment packet is used as the main Key (Key) of the fragment packet, and the offset (fragment offset field) and the length of the data packet are temporarily stored in the preset memory area, where DF is 0 and MF is 0 or 1.
Step S103: and for any identification field, when all the fragment messages with the identification field are received, recombining all the fragment messages with the identification field into a complete data message.
All the fragmentation messages with a certain identification field refer to all the fragmentation messages with the identification field sent by the client, and the fragmentation messages are not necessarily received at the same time.
Specifically, when all the fragment messages with a certain identification field are received, all the fragment messages in the preset memory area using the identification field as the primary key are recombined into a complete data message. The identification field is also the identification field of the complete data message obtained by recombination.
Wherein, it can be determined that all the fragmentation messages with an identification field are received and completed by the following method: determining that the last fragmentation message with the identification field is received completely, and acquiring a fragment offset field value of the last fragmentation message; acquiring the number of received fragment messages with the identification field, wherein the received fragment messages are all fragment messages in a preset memory region with the identification field as a main key; obtaining the total number of the fragmented messages with the identification field according to the fragment offset field value of the last fragmented message; and if the number of the received fragment messages is consistent with the total number of the fragment messages, determining that all the fragment messages with the identification field are received completely. .
For example, if the number of received fragment packets having a certain identification field is 7 and the fragment offset field value of the last fragment packet is 6, then it can be known from the fragment offset field value (offset) that the total number of the fragment packets having the identification field sent by the client is 7, and the number of received fragment packets is consistent with the total number of the fragment packets having the identification field sent by the client, it is determined that the last fragment packet having the identification field is received completely. If not, the last fragmentation message with the identification field is not received completely.
It should be noted that the last fragment packet with a certain identifier field in the embodiment of the present invention refers to the last fragment packet in all fragment packets with the identifier field sent by the client, rather than the last fragment packet that reaches the load balancer in the embodiment of the present invention, and the last fragment packet is determined during IP fragmentation.
The completion of the reception of the last fragmented packet with an identification field may be determined by:
If a certain fragmented message with the identification field is received, and the MF field value of the fragmented message indicates that the fragmented message has no subsequent fragments, the last fragmented message with the identification field is determined to be received completely. For example, if the value of the MF field of a certain fragmentation message with a certain identification field is 0(MF ═ 0), it indicates that the fragmentation message has no subsequent fragmentation, that is, the fragmentation message is the last fragmentation message with the identification field sent by the client.
if the number of the received fragment messages with a certain identification field is inconsistent with the total number of the fragment messages with the identification field sent by the client, it indicates that all the fragment messages with the identification field sent by the client have not been completely received, then: the setting of the timing time is realized by setting a timer in a DPDK specifically, so as to wait for receiving a fragment packet received in a delayed manner (i.e., delayed to reach a load balancer) in the network transmission process. And when the timing time is reached (that is, the timer is overtime), retrying the operation of determining that all the fragment messages with the identification field are completely received, if the fragment messages with the identification field are still not completely received after the retrying is executed (that is, the number of the received fragment messages with the identification field is not consistent with the total number of the fragment messages), resetting the timing time so as to perform the retrying operation again, and if the retrying times reach a preset threshold value, and the fragment messages with the identification field are still not completely received, deleting all the fragment messages with the identification field in a preset memory area so as to discard all the fragment messages with the identification field.
Step S104: and selecting the back-end service corresponding to the complete data message according to a preset model.
The preset model may be a source IP Port Hash (IP + Port Hash) model or a Round Robin Hash (Round Robin Hash) model. The polling distributes the data messages of the client to different backend servers (also called backend services) in turn. The IP + Port Hash calculates a Hash value of the IP and Port in the data packet (referred to as the main UDP packet) sent from the client, and the load balancer sends the fragment packet having the same identifier field as the main UDP packet to the same backend service.
Step S105: and segmenting the complete data message according to the maximum transmission unit value of the network, and forwarding the segmented complete data message to the selected back-end service.
The method can adopt a general IP message segmentation method, and sets identification, flag and offset of IP header bytes. And sending out the plurality of data messages obtained by segmentation through a network card, and simultaneously emptying the memory in which all IP fragment messages of the complete data message are located in a preset memory region (namely emptying the preset memory region taking the identification field of the complete data message as a main key).
It should be noted that, in step S104 and step S105 of the embodiment of the present invention, the complete data packet may be segmented according to the maximum transmission unit value of the network, then the backend service corresponding to the complete data packet is selected according to the preset model, and then the segmented complete data packet is forwarded to the selected backend service.
The invention mainly comprises a technical method and a means for processing data messages at a load balancing end under the condition of ensuring that UDP/IP messages are fragmented according to RFC standards (RFC records a series of files of Internet protocol standards), recombines the fragmented messages, implements a message forwarding strategy, and then decouples the data messages to be forwarded according to IP fragmentation specifications and forwards the data messages to the same back end for service.
Fig. 3 is a flow chart illustrating a preferred implementation of data message processing according to an embodiment of the present invention.
as shown in fig. 3, the preferred execution flow of the datagram processing according to the embodiment of the present invention includes steps S301 to S312 as follows. For each data message sent by the client, the following process is executed.
Step S301: receiving a data message sent by the client.
the data packet includes a flag field, an identification field, and a slice offset field, and for convenience of description, the identification field of the packet is referred to as a target identification field. The flags field includes an MF field and a DF field. These fields have been described in detail above and will not be described further here.
step S302: and judging whether the data message is a fragment message or not according to the flag field of the data message, if so, executing step S303, otherwise, executing step S312.
and judging whether the data message is a fragment message or not according to the DF field value of the mark field of the data message. That is, DF is 0, which indicates that the data packet is a fragment packet; DF is 1, which means that the data message is not a fragment message.
Step S303: and judging whether the fragmentation message is the last fragmentation message with the target identification field sent by the client or not according to the MF field value of the fragmentation message, if so, executing the step S304, otherwise, executing the step S305.
MF ═ 0 indicates that the fragmentation message is the last fragmentation, i.e., there is no subsequent fragmentation.
Step S304: and storing the fragment message into a preset memory area with the target identification field as a main key, and judging whether all the fragment messages with the target identification field sent by the client exist in the preset memory area, if so, executing a step S306, otherwise, executing a step S307.
specifically, the method for determining whether all the fragment messages with the target identification field sent by the client have been stored in the preset memory area includes: acquiring the number of received fragmented messages with the target identification field, wherein the received fragmented messages are all fragmented messages in a preset memory area with the target identification field as a main key; and obtaining the total number of the fragment messages with the target identification field according to the fragment offset field value of the fragment message, if the number of the received fragment messages is consistent with the total number of the fragment messages, storing all the fragment messages with the target identification field sent by the client, otherwise, storing the fragment messages in the preset memory area in the fragment messages with the target identification field sent by the client.
step S305: and storing the fragment message into a preset memory area with the target identification field as a main key.
Step S306: and recombining all the fragmentation messages with the target identification field stored in the preset memory area into a complete data message.
after step S306 is executed, step S311 is executed.
Step S307: and setting timing time so as to store other fragment messages of the target identification field which are received in a delayed manner into the preset memory area.
Step S308: and when the timing time is up, retrying to judge whether all the fragment messages with the target identification field sent by the client exist in the preset memory area, if so, executing the step S306, otherwise, executing the step S309.
Step S309: and judging whether the maximum retry number is reached, if so, executing the step S310, otherwise, returning to the step S307 to reset the timing time.
Step S310: and deleting all the fragmentation messages with the target identification field in a preset memory area so as to discard all the fragmentation messages.
step S311: and selecting a back-end service corresponding to the complete data message according to a preset model, forwarding the complete data message to the back-end service, and deleting all the fragment messages with the target identification field in a preset memory area.
Step S312: and selecting a back-end service corresponding to the data message according to a preset model, and forwarding the data message to the back-end service.
the embodiment of the invention provides a processing flow for carrying out load balancing on more than one UDP fragmented data message in a load balancing application scene. Through the embodiment of the invention, the UDP large packet can be transmitted to the corresponding back-end service after being sub-packaged by the load balancer.
Fig. 4 is a schematic diagram of main modules of a device for processing data packets according to an embodiment of the present invention.
As shown in fig. 4, the apparatus 400 for processing a data packet according to the embodiment of the present invention mainly includes: the system comprises a receiving module 401, an identifying module 402, a reorganizing module 403, a selecting module 404 and a forwarding module 405. The data packet processing apparatus 400 is applied to a four-layer load balancer, and this embodiment may be implemented based on a DPDK (fast packet forwarding and receiving tool), and besides the DPDK, other fast packet forwarding and receiving tools may also be used in the embodiment of the present invention.
The receiving module 401 is configured to receive data packets sent from a client, where each data packet includes an identification field and a flag field, and may further include a slice offset field. The flag field may include an MF field, and the MF field may be used to indicate whether there is a subsequent fragment in the fragmented packet.
The identifying module 402 is configured to identify a fragment message from each data message according to the flag field, where the fragment message is a data message in a fragment form.
A reassembly module 403, configured to, for any identifier field, when all the fragment packets with the identifier field are received completely, reassemble all the fragment packets with the identifier field into a complete data packet.
The restructuring module 403 may specifically be configured to: when all the fragment messages with an identification field are received, all the fragment messages in the preset memory area with the identification field as the main key are recombined into a complete data message.
A selecting module 404, configured to select a backend service corresponding to the complete data packet according to a preset model.
the selection module may specifically be configured to: and selecting the back-end service corresponding to the complete data message according to the source IP port hash model or the polling hash model.
the forwarding module 405 is configured to segment the complete data packet according to the maximum transmission unit value of the network, and forward the segmented complete data packet to a back-end service.
the apparatus 400 for processing data packets may further include a storage module configured to: and respectively taking the identification field of each fragment message as a main key, and storing each fragment message in a preset memory area.
The apparatus 400 for processing data packets may further include an inspection module configured to: determining that the last fragmentation message with an identification field (which may be any identification field in the identification field of the received data message) is received completely, and acquiring a fragment offset field value of the last fragmentation message; acquiring the number of received fragment messages with the identification field, wherein the received fragment messages are all fragment messages in a preset memory region with the identification field as a main key; obtaining the total number of the fragmented messages with the identification field according to the fragment offset field value of the last fragmented message; and if the number of the received fragment messages is consistent with the total number of the fragment messages, determining that all the fragment messages with the identification field are received completely.
The inspection module may include a message inspection sub-module to: if a certain fragmentation message with an identification field (which may be any identification field in the identification fields of the received data messages) is received, and the MF field value of the fragmentation message indicates that the fragmentation message has no subsequent fragmentation, it is determined that the last fragmentation message with the identification field is received completely.
The apparatus 400 for processing data packets may further include a retry module configured to: if the number of received fragment messages with an identification field (which may be any identification field in the identification field of the received data message) is not consistent with the total number of fragment messages with the identification field, then: setting timing time, and retrying to determine that all the fragmentation messages with the identification field are received completely when the timing time is reached; and if the number of the received fragment messages is not consistent with the total number of the fragment messages when the retry times reach a preset threshold value, deleting all the fragment messages with the identification field in a preset memory area so as to discard all the fragment messages.
in addition, in the embodiment of the present invention, the detailed implementation content of the processing apparatus for data packets has been described in detail in the above-mentioned processing method for data packets, so that repeated content herein is not described again.
fig. 5 illustrates an exemplary system architecture 500 of a method or apparatus for processing data packets to which embodiments of the present invention may be applied.
As shown in fig. 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 serves to provide a medium for communication links between the terminal devices 501, 502, 503 and the server 505. Network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 501, 502, 503 to interact with a server 505 over a network 504 to receive or send messages or the like. The terminal devices 501, 502, 503 may have various communication client applications installed thereon, such as a shopping application, a web browser application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.
The terminal devices 501, 502, 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 505 may be a server that provides various services, such as a background management server that supports shopping websites browsed by users using the terminal devices 501, 502, 503. The background management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (e.g., target push information and product information) to the terminal device.
It should be noted that the method for processing the data packet provided by the embodiment of the present invention is generally executed by the server 505, and accordingly, the processing device for the data packet is generally disposed in the server 505.
It should be understood that the number of terminal devices, networks, and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 6, shown is a block diagram of a computer system 600 suitable for use in implementing a server according to embodiments of the present application. The server shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU)601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
the following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the present application are executed when the computer program is executed by the Central Processing Unit (CPU) 601.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor comprises a receiving module 401, an identifying module 402, a reorganizing module 403, a selecting module 404, and a forwarding module 405. The names of these modules do not form a limitation on the modules themselves in some cases, for example, the receiving module 401 may also be described as a "module for receiving each data packet sent by the client.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: receiving each data message sent by a client, wherein each data message comprises an identification field and a mark field; identifying the fragment message from each data message according to the mark field; for any identification field, when all the fragment messages with the identification field are received, recombining all the fragment messages with the identification field into a complete data message; and segmenting the complete data message according to the maximum transmission unit value of the network, and forwarding the segmented complete data message to a corresponding back-end service, wherein the back-end service corresponding to the complete data message is selected according to a preset model.
according to the technical scheme of the embodiment of the invention, the fragment message is identified from each data message according to the mark field of the data message; for any identification field, when all the fragment messages with the identification field are received, recombining all the fragment messages with the identification field into a complete data message; and segmenting the complete data message according to the maximum transmission unit value of the network, and forwarding the segmented complete data message to the corresponding back-end service, wherein the back-end service corresponding to the complete data message is selected according to a preset model. After the UDP large packets exceeding the MTU pass through the load balancer, all the data messages are still forwarded to the same back-end service, and the load balancing normal operation of the UDP large packets is guaranteed.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (18)

1. A method for processing data messages is characterized by comprising the following steps:
Receiving each data message sent by a client, wherein each data message comprises an identification field and a mark field;
Identifying the fragment message from each data message according to the mark field;
For any identification field, when all the fragment messages with the identification field are received, recombining all the fragment messages with the identification field into a complete data message;
And segmenting the complete data message according to the maximum transmission unit value of the network, and forwarding the segmented complete data message to a corresponding back-end service, wherein the back-end service corresponding to the complete data message is selected according to a preset model.
2. the method according to claim 1, wherein the step of identifying fragmented packets from the data packets according to the flag field comprises:
and respectively taking the identification field of each fragment message as a main key, and storing each fragment message in a preset memory area.
3. The method according to claim 2, wherein the step of reassembling all the fragmented packets with the identification field into a complete data packet when all the fragmented packets with the identification field are completely received comprises:
And when all the fragment messages with the identification fields are received, recombining all the fragment messages in the preset memory area by using the identification fields as main keys into a complete data message.
4. The method of claim 2, wherein the data packet further comprises a slice offset field,
Determining that all the fragmentation messages with an identification field are received completely by the following method:
Determining that the last fragmented message with the identification field is received completely, and acquiring a fragment offset field value of the last fragmented message;
Acquiring the number of received fragment messages with the identification field, wherein the received fragment messages are all fragment messages in a preset memory region with the identification field as a main key;
obtaining the total number of the fragmented messages with the identification field according to the fragment offset field value of the last fragmented message;
And if the number of the received fragment messages is consistent with the total number of the fragment messages, determining that all the fragment messages with the identification field are received completely.
5. the method according to claim 4, wherein the flag field comprises an MF field, and the MF field is used for indicating whether there is a subsequent slice in the located slice packet,
Determining that the last fragment message with an identification field is received completely by the following method:
If a certain fragmented message with the identification field is received, and the MF field value of the fragmented message indicates that the fragmented message has no subsequent fragmentation, the last fragmented message with the identification field is determined to be received completely.
6. the method according to claim 4, wherein if the number of received fragmented packets is not consistent with the total number of fragmented packets, then:
Setting timing time, and retrying the operation of determining that all the fragmentation messages with the identification field are received completely when the timing time is reached;
And if the number of the received fragment messages is not consistent with the total number of the fragment messages when the retry times reach a preset threshold value, deleting all the fragment messages with the identification field in the preset memory area so as to discard all the fragment messages.
7. The method of claim 1, wherein the predetermined model is a source IP port hash model or a polling hash model.
8. The method of claim 1, wherein the data packet is a UDP/IP packet.
9. A device for processing data packets, comprising:
the receiving module is used for receiving each data message sent by the client, and each data message comprises an identification field and a mark field;
the identification module is used for identifying the fragment message from each data message according to the mark field;
A restructuring module, configured to, for any one of the identifier fields, when all the fragmented packets having the identifier field are received completely, restructure all the fragmented packets of the identifier field into a complete data packet;
The selecting module is used for selecting the back-end service corresponding to the complete data message according to a preset model;
And the forwarding module is used for segmenting the complete data message according to the maximum transmission unit value of the network and forwarding the segmented complete data message to the back-end service.
10. The apparatus of claim 9, further comprising a storage module to:
And respectively taking the identification field of each fragment message as a main key, and storing each fragment message in a preset memory area.
11. the apparatus of claim 10, wherein the reassembly module is further configured to:
And when all the fragment messages with an identification field are received, recombining all the fragment messages taking the identification field as a main key in the preset memory area into a complete data message.
12. The apparatus of claim 10, wherein the data packet further comprises a slice offset field,
the apparatus further comprises a checking module for:
Determining that the last fragmented message with an identification field is received completely, and acquiring a fragment offset field value of the last fragmented message;
acquiring the number of received fragment messages with the identification field, wherein the received fragment messages are all fragment messages in a preset memory region with the identification field as a main key;
Obtaining the total number of the fragmented messages with the identification field according to the fragment offset field value of the last fragmented message;
And if the number of the received fragment messages is consistent with the total number of the fragment messages, determining that all the fragment messages with the identification field are received completely.
13. The apparatus according to claim 12, wherein the flag field includes an MF field, and the MF field is used for indicating whether there is a subsequent slice in the located slice packet,
The checking module comprises a message checking submodule for:
If a certain fragmented message with an identification field is received, and the MF field value of the fragmented message indicates that the fragmented message has no subsequent fragmentation, the last fragmented message with the identification field is determined to be received completely.
14. The apparatus of claim 12, further comprising a retry module to:
If the number of the received fragment messages with an identification field is not consistent with the total number of the fragment messages with the identification field, then:
Setting timing time, and retrying the operation of determining that all the fragmentation messages with the identification field are received completely when the timing time is reached;
and if the number of the received fragment messages is not consistent with the total number of the fragment messages when the retry times reach a preset threshold value, deleting all the fragment messages with the identification field in the preset memory area so as to discard all the fragment messages.
15. The apparatus of claim 9, wherein the selecting module is further configured to:
And selecting the back-end service corresponding to the complete data message according to a source IP port hash model or a polling hash model.
16. The apparatus of claim 9, wherein the data packet is a UDP/IP packet.
17. an electronic device, comprising:
one or more processors;
A memory for storing one or more programs,
The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-8.
18. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-8.
CN201810587646.2A 2018-06-08 2018-06-08 Data message processing method and device Pending CN110581812A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810587646.2A CN110581812A (en) 2018-06-08 2018-06-08 Data message processing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810587646.2A CN110581812A (en) 2018-06-08 2018-06-08 Data message processing method and device

Publications (1)

Publication Number Publication Date
CN110581812A true CN110581812A (en) 2019-12-17

Family

ID=68809528

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810587646.2A Pending CN110581812A (en) 2018-06-08 2018-06-08 Data message processing method and device

Country Status (1)

Country Link
CN (1) CN110581812A (en)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112565381A (en) * 2020-11-30 2021-03-26 杭州迪普科技股份有限公司 Data packet forwarding method, device and equipment
CN112583822A (en) * 2020-12-09 2021-03-30 海信视像科技股份有限公司 Communication apparatus and communication method
CN112688879A (en) * 2020-12-29 2021-04-20 联想未来通信科技(重庆)有限公司 Data limit value determining method and device
WO2021185208A1 (en) * 2020-03-17 2021-09-23 华为技术有限公司 Packet processing method and apparatus, device, and storage medium
CN113660295A (en) * 2021-10-20 2021-11-16 深圳市龙信信息技术有限公司 Message processing device
WO2022001287A1 (en) * 2020-07-03 2022-01-06 华为技术有限公司 Message processing method and device
CN114679425A (en) * 2022-03-24 2022-06-28 深圳震有科技股份有限公司 Message processing method, network equipment and storage medium in 5G network
CN114844842A (en) * 2022-05-07 2022-08-02 湖南戎腾网络科技有限公司 Data transmission method and device, electronic equipment and readable storage medium
CN115225593A (en) * 2022-06-30 2022-10-21 中科驭数(北京)科技有限公司 Fragmented message reassembling method, device, equipment and medium
CN115514810A (en) * 2021-06-22 2022-12-23 成都鼎桥通信技术有限公司 Data transmission method and device

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1592215A (en) * 2003-08-27 2005-03-09 中兴通讯股份有限公司 Method for partitioned document recombination and service distribution
CN1863141A (en) * 2005-07-29 2006-11-15 华为技术有限公司 Method for transmission processing IP fragment message
CN101656677A (en) * 2009-09-18 2010-02-24 杭州迪普科技有限公司 Message diversion processing method and device
CN103685030A (en) * 2013-12-24 2014-03-26 大唐移动通信设备有限公司 Method and device for data processing
CN106559413A (en) * 2016-10-19 2017-04-05 深圳众享互联科技有限公司 The message fragment method and its system of P2P network data securities transmission

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1592215A (en) * 2003-08-27 2005-03-09 中兴通讯股份有限公司 Method for partitioned document recombination and service distribution
CN1863141A (en) * 2005-07-29 2006-11-15 华为技术有限公司 Method for transmission processing IP fragment message
CN101656677A (en) * 2009-09-18 2010-02-24 杭州迪普科技有限公司 Message diversion processing method and device
CN103685030A (en) * 2013-12-24 2014-03-26 大唐移动通信设备有限公司 Method and device for data processing
CN106559413A (en) * 2016-10-19 2017-04-05 深圳众享互联科技有限公司 The message fragment method and its system of P2P network data securities transmission

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021185208A1 (en) * 2020-03-17 2021-09-23 华为技术有限公司 Packet processing method and apparatus, device, and storage medium
US11870690B2 (en) 2020-03-17 2024-01-09 Huawei Technologies Co., Ltd. Packet processing method and apparatus, device, and storage medium
WO2022001287A1 (en) * 2020-07-03 2022-01-06 华为技术有限公司 Message processing method and device
CN112565381B (en) * 2020-11-30 2022-05-31 杭州迪普科技股份有限公司 Data packet forwarding method, device and equipment
CN112565381A (en) * 2020-11-30 2021-03-26 杭州迪普科技股份有限公司 Data packet forwarding method, device and equipment
CN112583822A (en) * 2020-12-09 2021-03-30 海信视像科技股份有限公司 Communication apparatus and communication method
CN112688879A (en) * 2020-12-29 2021-04-20 联想未来通信科技(重庆)有限公司 Data limit value determining method and device
CN112688879B (en) * 2020-12-29 2023-08-22 联想未来通信科技(重庆)有限公司 Data limit value determining method and device
CN115514810A (en) * 2021-06-22 2022-12-23 成都鼎桥通信技术有限公司 Data transmission method and device
CN113660295A (en) * 2021-10-20 2021-11-16 深圳市龙信信息技术有限公司 Message processing device
CN114679425A (en) * 2022-03-24 2022-06-28 深圳震有科技股份有限公司 Message processing method, network equipment and storage medium in 5G network
CN114679425B (en) * 2022-03-24 2024-05-31 深圳震有科技股份有限公司 Message processing method, network equipment and storage medium under 5G network
CN114844842A (en) * 2022-05-07 2022-08-02 湖南戎腾网络科技有限公司 Data transmission method and device, electronic equipment and readable storage medium
CN115225593A (en) * 2022-06-30 2022-10-21 中科驭数(北京)科技有限公司 Fragmented message reassembling method, device, equipment and medium
CN115225593B (en) * 2022-06-30 2023-11-28 中科驭数(北京)科技有限公司 Method, device, equipment and medium for reorganizing fragmented messages

Similar Documents

Publication Publication Date Title
CN110581812A (en) Data message processing method and device
CN106850402B (en) Message transmission method and device
US8009672B2 (en) Apparatus and method of splitting a data stream over multiple transport control protocol/internet protocol (TCP/IP) connections
US9130991B2 (en) Processing data packets in performance enhancing proxy (PEP) environment
US9300733B2 (en) System and/or method for client-driven server load distribution
US9185033B2 (en) Communication path selection
CN105939297B (en) A kind of TCP message recombination method and device
CN106612284B (en) Streaming data transmission method and device
US9794354B1 (en) System and method for communication between networked applications
CN110572422B (en) Data downloading method, device, equipment and medium
CN110609746B (en) Method, apparatus and computer readable medium for managing network system
CN113382062B (en) Data transmission method, device and system
CN112671771B (en) Data transmission method, device, electronic equipment and medium
CN110545230B (en) Method and device for forwarding VXLAN message
CN113595927A (en) Method and device for processing mirror flow in bypass mode
CN113259490B (en) Multi-level node network data transmission method based on UDP transmission protocol
CN111385068B (en) Data transmission method, device, electronic equipment and communication system
CN113312355A (en) Data management method and device
CN103873443A (en) Information processing method, local proxy server and network proxy server
CN105100101A (en) Method, apparatus and system based on SSL session
US8819107B2 (en) Relay apparatus, recording medium storing a relay program, and a relay method
CN111800223A (en) Method, device and system for generating sending message and processing receiving message
EP3408989B1 (en) Detecting malware on spdy connections
CN107612831A (en) A kind of transmission method and device of the data message for accessing source station
CN106209666A (en) A kind of link multiplexing method based on load equalizer and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20191217