CN111427917A - Search data processing method and related product - Google Patents
Search data processing method and related product Download PDFInfo
- Publication number
- CN111427917A CN111427917A CN202010131859.1A CN202010131859A CN111427917A CN 111427917 A CN111427917 A CN 111427917A CN 202010131859 A CN202010131859 A CN 202010131859A CN 111427917 A CN111427917 A CN 111427917A
- Authority
- CN
- China
- Prior art keywords
- data
- search
- consumption
- consumed
- data set
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24573—Query processing with adaptation to user needs using data annotations, e.g. user-defined metadata
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the application discloses a search data processing method and a related product, wherein the method comprises the following steps: receiving a first search data set transmitted by a first data source, and receiving a plurality of search service records transmitted by a second data source; generating a second search data set according to the plurality of search service records, and asynchronously filing the second search data set and the first search data set as data to be consumed to a preset message queue; extracting the data to be consumed from the preset message queue and executing consumption operation to obtain a third search data set; and storing the third search data set to a preset first database and a preset second database. The embodiment of the application has the advantage of high user experience.
Description
Technical Field
The present application relates to the field of data storage, and in particular, to a search data processing method and related products.
Background
With the rapid development of internet technology, search becomes the basic function of the current internet products, the search is a very basic core component for traditional internet era B-end applications or mobile internet C-end applications, and whether the search function of an application program is powerful or not plays a great role in enhancing user experience and improving user stickiness.
In the searching workflow, the key point is that the storage of the search metadata falls to the ground, and the traditional search metadata filing basically adopts synchronous processing, namely primary collection and storage, but the processing mode may not realize data timely recovery in case of emergency, and the filing operation has poor concurrency performance in processing the process service data, and may occupy physical resources of the system in a large scale, thereby causing service blockage of other systems, causing delay of the whole network interaction of the application program, and low user experience.
Disclosure of Invention
The embodiment of the application provides a search data processing method and a related product, a first search data set and a second search data set are asynchronously filed to a preset message queue and then consumption operation is executed to obtain a third search data set, and the third search data set is stored to a first database and a second database, so that data decoupling is realized, the occupancy rate of system physical resources is favorably reduced, network interaction delay is reduced, and user experience is improved.
In a first aspect, an embodiment of the present application provides a search data archiving method, including:
receiving a first search data set transmitted by a first data source, and receiving a plurality of search service records transmitted by a second data source;
generating a second search data set according to the plurality of search service records, and asynchronously storing the second search data set and the first search data set as data to be consumed to a preset message queue;
extracting the data to be consumed from the preset message queue and executing consumption operation to obtain a third search data set;
and storing the third search data set to a preset first database and a preset second database.
In a second aspect, an embodiment of the present application provides a search data archiving apparatus, including:
the receiving unit is used for receiving a first search data set transmitted by a first data source and receiving a plurality of search service records transmitted by a second data source;
the generating unit is used for generating a second search data set according to the plurality of search service records, and asynchronously storing the second search data set and the first search data set as data to be consumed to a preset message queue;
the consumption unit is used for extracting the data to be consumed from the preset message queue and executing consumption operation to obtain a third search data set;
and the storage unit is used for storing the third search data set to a preset first database and a preset second database.
In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, a communication device, and one or more programs, where the processor, the memory, and the communication device are connected to each other, where the communication device is configured to perform information interaction with an external device, and the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing the steps in any of the methods of the first aspect of the embodiments of the present application.
In a fourth aspect, the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program for electronic data exchange, where the computer program makes a computer perform some or all of the steps described in the method according to the first aspect of the present application.
In a fifth aspect, the present application provides a computer program product, where the computer program product includes a non-transitory computer-readable storage medium storing a computer program, where the computer program is operable to cause a computer to perform some or all of the steps described in the method according to the first aspect of the present application. The computer program product may be a software installation package.
It can be seen that, in the embodiment of the present application, the electronic device receives a first search data set transmitted by a first data source, and receives a plurality of search service records transmitted by a second data source; generating a second search data set according to the plurality of search service records, and asynchronously storing the second search data set and the first search data set as data to be consumed to a preset message queue; extracting the data to be consumed from the preset message queue and executing consumption operation to obtain a third search data set; and storing the third search data set to a preset first database and a preset second database. Therefore, the first search data set and the second search data set are asynchronously filed to the preset message queue and then consumption operation is executed to obtain a third search data set, and the third search data set is stored in the first database and the second database, so that data decoupling is realized, the physical resource occupancy rate of a system is favorably reduced, network interaction delay is reduced, and user experience is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic structural diagram of a search data processing method according to an embodiment of the present application;
FIG. 2 is a schematic flow chart of another search data processing method provided in the embodiments of the present application;
FIG. 3 is a schematic flowchart of another search data processing method provided in an embodiment of the present application;
FIG. 4 is a schematic flowchart of another search data processing method provided in an embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;
fig. 6 is a block diagram of functional units of a search data processing apparatus according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "first," "second," "third," and "fourth," etc. in the description and claims of the invention and in the accompanying drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
Reference herein to "an embodiment" means that a particular feature, result, or characteristic described in connection with the embodiment can be included in at least one embodiment of the invention. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
Electronic devices may include various handheld devices, vehicle-mounted devices, wearable devices (e.g., smartwatches, smartbands, pedometers, etc.), computing devices or other processing devices connected to wireless modems, as well as various forms of User Equipment (UE), Mobile Stations (MS), terminal Equipment (terminal device), and so forth, having wireless communication capabilities. For convenience of description, the above-mentioned devices are collectively referred to as electronic devices.
Data archiving is the process of moving data that is no longer frequently used to a separate storage device for long term storage. A data archive consists of old data, but it is the data that is necessary and important for later reference, which data must be preserved in compliance with the rules. The data archive has indexing and searching functions so that files can be easily found.
The following describes embodiments of the present application in detail.
Referring to fig. 1, fig. 1 is a schematic flowchart of a search data processing method provided in an embodiment of the present application, and the search data processing method is applied to an electronic device, and the search data processing method includes:
optionally, before receiving the first search data set transmitted by the first data source and receiving the plurality of search service records transmitted by the second data source, the method further includes: setting a timing controller, wherein the timing controller is used for controlling data transmission of the first data source and the second data source, when the time of the timing controller is equal to preset time, generating a first data acquisition request and a search service acquisition request, sending the first data acquisition request to the first data source, and sending the search service acquisition request to the second data source, the first data acquisition request is used for requesting the first data source to transmit a first search data set within the preset time, and the search service acquisition request is used for requesting the second data source to transmit a plurality of search service records within the preset time.
Optionally, before receiving the first search data set transmitted by the first data source and receiving the plurality of search service records transmitted by the second data source, the method further includes: when a data acquisition instruction of a target user is detected, the target user is verified, if the verification is successful, a first data acquisition request and a search service acquisition request are generated, the first data acquisition request is sent to a first data source, the search service acquisition request is sent to a second data source, and if the verification is failed, the data acquisition instruction is rejected.
Further, authenticating the target user comprises: receiving verification data in the data acquisition instruction, verifying the target user according to the verification data, for example, if the verification data is fingerprint data, acquiring a preset fingerprint template set, comparing the fingerprint data in the fingerprint template set, and if the comparison is successful, successfully verifying; if the verification data is face data, a preset face template set is obtained, the face data is compared in the face template set, and if the comparison is successful, the verification is successful.
wherein any one of the plurality of search service records comprises: the search traffic is pipelined.
Wherein the first search data set includes at least one first search metadata and the second search data set includes at least one second search metadata.
In the embodiment of the present application, data to which a search is applied is abstracted into data of various data mechanisms because of structural differences and association differences thereof, which are collectively referred to as metadata, and since the information structure of the search to which the metadata is applied is different, the index structure in which they are stored is also different.
In the embodiment of the application, asynchronous archiving refers to a process that different metadata are acquired from different sources, data are written into a message queue by different synchronous information, are decoupled by a message queue middleware asynchronous mechanism, are distributed to different server instances for consumption, and are persisted to a database for storage, wherein different consumption instances need to be provided for asynchronous consumption due to different metadata mechanisms and different information associations; and there are differences in the format and fields of the data table stored in the database, so that separate services may be provided.
103, extracting the data to be consumed from the preset message queue and executing consumption operation to obtain a third search data set;
optionally, a preset data extraction rule is obtained, the data to be consumed in the preset message queue is obtained according to the data extraction rule, and a consumption operation is performed on the data to be consumed to obtain the third search data set.
Wherein the data extraction rule may include: first-in first-out rules, first-in last-out rules, etc., and are not limited herein.
In the embodiment of the application, the data consumption is a consumption activity taking a statistical data product and a statistical data service as consumption objects, is a relatively special information consumption form, and is an important component of social consumption. It is the purpose and motive power of statistical data production, and is the embodiment of the final purpose of statistical work. The quantity, quality, structure and mode of data consumption not only affect the production and transmission of statistical data, directly reflect the quality of statistical data use, reflect the exertion of the whole statistical function, but also affect the quality and level of data user consumption, and are related to the development of the whole society and economy due to statistical career. The trend of modern scientific development shows that almost all disciplines and all fields accumulate data, all need to speak by using the data, and the rules of the data need to be discovered through analysis of the data. And with the development of computer science and network technology, the situation is more and more obvious and the speed is faster and faster. Therefore, the data consumption has great practical significance for promoting the development of various social careers, especially the statistical career and the information service industry, promoting the consumption upgrade, improving the quality and the level of life of people and realizing the sustainable development of economy.
And 104, storing the third search data set to a preset first database and a preset second database.
The first database is used for searching and retrieving, and the second database is used for recording backtracking.
In a possible example, the generating a second search data set from the plurality of search service records comprises: extracting a plurality of search service pipelines from the plurality of search service records, wherein any one of the plurality of search service pipelines comprises at least one search keyword; executing search keyword extraction operation on the plurality of search service streams to obtain a plurality of search keyword sets corresponding to the plurality of search service streams; and synthesizing the plurality of search keyword sets to obtain the second search data set.
Wherein any one of the plurality of search service pipelines comprises: at least one search keyword and at least one search time corresponding to the at least one search keyword.
Optionally, a preset keyword position set is obtained, where the keyword position set includes: at least one keyword position, executing search keyword extraction operation on each search service flow of the plurality of search service flows according to the keyword position set to obtain a plurality of search keyword sets, wherein the plurality of search keyword sets correspond to the plurality of search service flows one by one, extracting a plurality of search keyword sets contained in the plurality of search keyword sets to generate a second search data set, and the plurality of search keywords are second metadata corresponding to the second search data set.
In a possible example, the extracting the data to be consumed from the preset message queue and performing a consumption operation to obtain a third search data set includes: acquiring a plurality of listeners corresponding to a plurality of preset consumption servers, wherein the plurality of listeners are used for monitoring data transmission operations of the first data source and the second data source; when the plurality of listeners monitor data transmission operations of the first data source and/or the second data source, starting the plurality of consumption servers; and executing consumption operation on the data to be consumed according to the plurality of consumption servers to obtain the third search data set.
Wherein, the plurality of listeners correspond to the plurality of consumption servers one to one.
The listeners in the plurality of listeners are used for monitoring a first data source and a second data source, monitoring data transmission operations of the first data source and the second data source, and starting a consumption model corresponding to the listener when the first data source and/or the second data source is monitored to transmit data to the preset message queue.
Optionally, the obtaining of the data to be consumed in the preset message queue and performing the consumption operation to obtain the third search data set further includes: and setting a listener for the preset message queue, and sending a consumption server starting request to the application processor when the listener monitors that the preset message queue is to be written with the consumption data, wherein the consumption server starting request is used for requesting the application processor to start a plurality of consumption servers.
In a possible example, the obtaining the third search data set by performing a consumption operation on the data to be consumed according to the plurality of consumption servers includes: extracting the data to be consumed from the preset message queue according to a preset data extraction rule; acquiring a data format to be consumed of the data to be consumed, and judging whether a preset data format set contains the data format to be consumed; if the data format set contains the data format to be consumed, determining a target consumption server corresponding to the data format to be consumed according to a preset mapping relation between the data format and a consumption server and the plurality of consumption servers; generating a data consumption instruction according to the data to be consumed, and sending the data consumption instruction to the target consumption server, wherein the data consumption instruction is used for indicating the target consumption server to execute consumption operation on the data to be consumed; receiving a data consumption response returned by the target consumption server, and extracting consumption data from the data consumption response; generating the third search data set in dependence on the consumption data.
The format of the data to be consumed may include, but is not limited to, JSON format, XM L format, YAM L format, and the like.
The set of data formats includes a plurality of data formats, wherein any one of the plurality of data formats may include, but is not limited to, JSON format, XM L format, YAM L format, and the like.
The consumption servers correspond to the data formats contained in the data format set one by one.
The data format (data format) described in the embodiments of the present application is a rule describing that data is stored in a file or record. It may be in a text format in the form of characters, or in a compressed format in the form of binary data. The text format in the form of characters occupies a large amount of storage space but has high transparency, and the compressed format in the form of binary numbers occupies a small amount of storage space but lacks transparency.
In the specific implementation process, data to be consumed is extracted from the preset message queue according to a preset data extraction rule (for example, a first-in first-out rule), that is, data to be consumed at the end of the queue in the preset message queue is acquired, a format of the data to be consumed is determined, assuming that the format of the data to be consumed is a text format, the data to be consumed is text data to be consumed, a preset data format set is acquired, whether the data format set contains the text format is judged, if the data format set contains the text format, a mapping relation between the data format and a consumption server is acquired, a text consumption server corresponding to the text format is determined according to the mapping relation between the data format and the consumption server, and a data consumption instruction is generated according to the data to be consumed, that is the text data to be consumed, wherein the data consumption instruction is used for instructing the text consumption server to perform consumption operation on the text data to be consumed, and sending the text data to be consumed to the text consumption server, receiving a data consumption response returned by the text consumption server, extracting the consumption text data corresponding to the text data to be consumed from the data consumption response, and storing the consumption text data into the third search data set.
In a possible example, the method further comprises: if the data format set does not contain the data format to be consumed, generating a data consumption reminder according to the data format to be consumed, wherein the data consumption reminder is used for reminding that a consumption server corresponding to the data format to be consumed is missing; and sending the data consumption prompt to a preset mobile terminal.
The data consumption reminder is used for sending a reminder for missing the data consumption format to the preset mobile terminal. In a specific implementation process, assuming that the data format to be consumed is a text format, and determining that the data format set does not contain the text format, generating a data consumption reminder according to the text format, wherein the data consumption reminder includes: the data consumption reminding is used for reminding that the consumption server corresponding to the format of the data to be consumed is missing, namely for reminding that the text consumption server corresponding to the text format is missing in the current consumption server, and the data consumption reminding is sent to a preset mobile terminal.
In a possible example, after storing the third search data in the preset first database and the preset second database, the method further includes: when input data of a target user are received, comparing the input data with the third search data set in the first database to obtain a plurality of comparison values; determining third search data corresponding to the maximum value in the comparison values as a search result; and sending the search result to the mobile terminal corresponding to the target user.
In a specific implementation process, input data of a target user is received, and the input data may include: text data, audio data, and the like, without limitation; comparing the input data with a third search data set stored in the first database to obtain a plurality of comparison values, determining that the third search data corresponding to the maximum value in the plurality of comparison values is a search result, and sending the search result to a user terminal corresponding to the target user, wherein if the input data is audio data, the method further comprises the following steps before matching the input data with the third search data stored in the first database: and taking the audio data as the input of a preset audio recognition model to obtain audio text data corresponding to the audio data, and updating the input data according to the audio text data.
It can be seen that, in the embodiment of the present application, the electronic device receives a first search data set transmitted by a first data source, and receives a plurality of search service records transmitted by a second data source; generating a second search data set according to the plurality of search service records, and asynchronously storing the second search data set and the first search data set as data to be consumed to a preset message queue; extracting the data to be consumed from the preset message queue and executing consumption operation to obtain a third search data set; and storing the third search data set to a preset first database and a preset second database. Therefore, the first search data set and the second search data set are asynchronously filed to the preset message queue and then consumption operation is executed to obtain a third search data set, and the third search data set is stored in the first database and the second database, so that data decoupling is realized, the physical resource occupancy rate of a system is favorably reduced, network interaction delay is reduced, and user experience is improved.
Referring to fig. 2, fig. 2 is a schematic flowchart of another search data processing method provided in an embodiment of the present application, and the search data processing method is applied to an electronic device, and the search data processing method includes:
and step 206, storing the third search data set to a preset first database and a preset second database.
The specific description of the steps 201 to 206 may refer to the corresponding steps of the search data processing method described in fig. 1, and will not be described herein again.
It can be seen that, in the embodiment of the present application, the electronic device receives a first search data set transmitted by a first data source, and receives a plurality of search service records transmitted by a second data source; extracting a plurality of search service pipelines from the plurality of search service records, wherein any one of the plurality of search service pipelines comprises at least one search keyword; executing search keyword extraction operation on the plurality of search service streams to obtain a plurality of search keyword sets corresponding to the plurality of search service streams; merging the plurality of search keyword sets to obtain a second search data set, and asynchronously archiving the second search data set and the first search data set as data to be consumed to a preset message queue; extracting the data to be consumed from the preset message queue and executing consumption operation to obtain a third search data set; and storing the third search data set to a preset first database and a preset second database. Therefore, metadata decoupling can be realized through asynchronous archiving, and the occupancy rate of system physical resources is reduced, so that network interaction delay is reduced, and user experience is improved.
Referring to fig. 3, fig. 3 is a schematic flowchart of another search data processing method provided in an embodiment of the present application, and the search data processing method is applied to an electronic device, and the search data processing method includes:
305, executing consumption operation on the data to be consumed according to the plurality of consumption servers to obtain a third search data set;
and step 306, storing the third search data set to a preset first database and a preset second database.
For the detailed description of steps 301 to 306, reference may be made to the corresponding steps of the search data processing method described in fig. 1, which are not described herein again.
It can be seen that, in the embodiment of the present application, the electronic device receives a first search data set transmitted by a first data source, and receives a plurality of search service records transmitted by a second data source; generating a second search data set according to the plurality of search service records, and asynchronously filing the second search data set and the first search data set as data to be consumed to a preset message queue; acquiring a plurality of listeners corresponding to a plurality of preset consumption servers, wherein the plurality of listeners are used for monitoring data transmission operations of the first data source and the second data source; when the plurality of listeners monitor data transmission operations of the first data source and/or the second data source, starting the plurality of consumption servers; executing consumption operation on the data to be consumed according to the plurality of consumption servers to obtain a third search data set; and storing the third search data set to a preset first database and a preset second database. Therefore, the first data source and the second data source can be monitored through the monitor, when data transmission occurs, the consumption server is started to execute consumption operation on the data to be consumed to obtain a third search data set, the decoupling of metadata of different data sources is achieved, and the user experience degree is improved.
Referring to fig. 4, fig. 4 is a schematic flowchart of another search data processing method provided in an embodiment of the present application, and the search data processing method is applied to an electronic device, and the search data processing method includes:
For the detailed description of steps 401 to 407, reference may be made to corresponding steps of the search data processing method described in fig. 1, and details are not repeated here.
It can be seen that, in the embodiment of the present application, the electronic device receives a first search data set transmitted by a first data source, and receives a plurality of search service records transmitted by a second data source; generating a second search data set according to the plurality of search service records, and asynchronously filing the second search data set and the first search data set as data to be consumed to a preset message queue; extracting the data to be consumed from the preset message queue and executing consumption operation to obtain a third search data set; storing the third search data set to a preset first database and a preset second database; when receiving input data of a target user, comparing the input data with the third search data set in the first database to obtain a plurality of comparison values; determining third search data corresponding to the maximum value in the comparison values as a search result; and sending the search result to the mobile terminal corresponding to the target user. Therefore, a third search data set can be obtained by asynchronously filing the first search data set and the second search data set to a preset message queue and then executing consumption operation, the third search data set is stored in the first database and the second database, data decoupling is achieved, the system physical resource occupancy rate is reduced, network interaction delay is reduced, comparison is conducted in the first database when input data are received, the search data can be cleaned, and user experience is improved.
Referring to fig. 5, fig. 5 is a schematic structural diagram of an electronic device 500 according to an embodiment of the present application, and as shown in the drawing, the electronic device 500 includes an application processor 510, a memory 520, a communication interface 530, and one or more programs 521, where the one or more programs 521 are stored in the memory 520 and configured to be executed by the application processor 510, and the one or more programs 521 include instructions for performing the following steps:
receiving a first search data set transmitted by a first data source, and receiving a plurality of search service records transmitted by a second data source;
generating a second search data set according to the plurality of search service records, and asynchronously storing the second search data set and the first search data set as data to be consumed to a preset message queue;
extracting the data to be consumed from the preset message queue and executing consumption operation to obtain a third search data set;
and storing the third search data set to a preset first database and a preset second database.
It can be seen that, in the embodiment of the present application, the electronic device receives a first search data set transmitted by a first data source, and receives a plurality of search service records transmitted by a second data source; generating a second search data set according to the plurality of search service records, and asynchronously storing the second search data set and the first search data set as data to be consumed to a preset message queue; extracting the data to be consumed from the preset message queue and executing consumption operation to obtain a third search data set; and storing the third search data set to a preset first database and a preset second database. Therefore, the first search data set and the second search data set are asynchronously filed to the preset message queue and then consumption operation is executed to obtain a third search data set, and the third search data set is stored in the first database and the second database, so that data decoupling is realized, the physical resource occupancy rate of a system is favorably reduced, network interaction delay is reduced, and user experience is improved.
In a possible example, the generating of the second search data set from the plurality of search traffic records is performed by instructions in the program being specifically configured to: extracting a plurality of search service pipelines from the plurality of search service records, wherein any one of the plurality of search service pipelines comprises at least one search keyword; executing search keyword extraction operation on the plurality of search service streams to obtain a plurality of search keyword sets corresponding to the plurality of search service streams; and combining the plurality of search keyword sets to obtain the second search data set.
In a possible example, the extracting the data to be consumed from the preset message queue and performing a consumption operation obtain a third search data set, and the instructions in the program are specifically configured to perform the following operations: acquiring a plurality of listeners corresponding to a plurality of preset consumption servers, wherein the plurality of listeners are used for monitoring data transmission operations of the first data source and the second data source; when the plurality of listeners monitor data transmission operations of the first data source and/or the second data source, starting the plurality of consumption servers; and executing consumption operation on the data to be consumed according to the plurality of consumption servers to obtain the third search data set.
In a possible example, the third search data set is obtained by performing, by the plurality of consumption servers, a consumption operation on the data to be consumed, and the instructions in the program are specifically configured to perform the following operations: extracting the data to be consumed from the preset message queue according to a preset data extraction rule; acquiring a data format to be consumed of the data to be consumed, and judging whether a preset data format set contains the data format to be consumed; if the data format set contains the data format to be consumed, determining a target consumption server corresponding to the data format to be consumed according to a preset mapping relation between the data format and a consumption server and the plurality of consumption servers; generating a data consumption instruction according to the data to be consumed, and sending the data consumption instruction to the target consumption server, wherein the data consumption instruction is used for indicating the target consumption server to execute consumption operation on the data to be consumed; receiving a data consumption response returned by the target consumption server, and extracting consumption data from the data consumption response; generating the third search data set in dependence on the consumption data.
In one possible example, the instructions in the program are further to perform the following operations: if the data format set does not contain the data format to be consumed, generating a data consumption reminder according to the data format to be consumed, wherein the data consumption reminder is used for reminding that a consumption server corresponding to the data format to be consumed is missing; and sending the data consumption prompt to a preset mobile terminal.
In a possible example, after storing the third search data in the preset first database and the preset second database, the instructions in the program are further configured to: when input data of a target user are received, comparing the input data with the third search data set in the first database to obtain a plurality of comparison values; determining third search data corresponding to the maximum value in the comparison values as a search result; and sending the search result to the mobile terminal corresponding to the target user.
The above description has introduced the solution of the embodiment of the present application mainly from the perspective of the method-side implementation process. It is understood that the electronic device comprises corresponding hardware structures and/or software modules for performing the respective functions in order to realize the above-mentioned functions. Those of skill in the art would readily appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiment of the present application, the electronic device may be divided into the functional units according to the method example, for example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one control unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit. It should be noted that the division of the unit in the embodiment of the present application is schematic, and is only a logic function division, and there may be another division manner in actual implementation.
Fig. 6 is a block diagram of functional units of a search data processing apparatus 600 according to an embodiment of the present application. The search data processing apparatus 600 is applied to an electronic device, and the search data processing apparatus 600 includes a receiving unit 601, a generating unit 602, a consuming unit 603, and a storing unit 604, in which:
a receiving unit 601, configured to receive a first search data set transmitted by a first data source, and receive multiple search service records transmitted by a second data source;
a generating unit 602, configured to generate a second search data set according to the multiple search service records, and asynchronously archive the second search data set and the first search data set as data to be consumed to a preset message queue;
the consumption unit 603 is configured to extract the data to be consumed from the preset message queue, and perform a consumption operation to obtain a third search data set;
the storage unit 604 is configured to store the third search data set in a preset first database and a preset second database.
It can be seen that, in the embodiment of the present application, the electronic device receives a first search data set transmitted by a first data source, and receives a plurality of search service records transmitted by a second data source; generating a second search data set according to the plurality of search service records, and asynchronously storing the second search data set and the first search data set as data to be consumed to a preset message queue; extracting the data to be consumed from the preset message queue and executing consumption operation to obtain a third search data set; and storing the third search data set to a preset first database and a preset second database. Therefore, the first search data set and the second search data set are asynchronously filed to the preset message queue and then consumption operation is executed to obtain a third search data set, and the third search data set is stored in the first database and the second database, so that data decoupling is realized, the physical resource occupancy rate of a system is favorably reduced, network interaction delay is reduced, and user experience is improved.
In a possible example, the generating unit 602 is specifically configured to generate the second search data set according to the plurality of search service records, and to: extracting a plurality of search service pipelines from the plurality of search service records, wherein any one of the plurality of search service pipelines comprises at least one search keyword; executing the keyword extraction operation on the plurality of search service streams to obtain a plurality of search keyword sets corresponding to the plurality of search service streams; and combining the plurality of search keyword sets to obtain the second search data set.
In a possible example, the extracting the data to be consumed from the preset message queue and performing a consumption operation to obtain a third search data set, and the consumption unit 603 is specifically configured to: acquiring a plurality of listeners corresponding to a plurality of preset consumption servers, wherein the plurality of listeners are used for monitoring data transmission operations of the first data source and the second data source; when the plurality of listeners monitor data transmission operations of the first data source and/or the second data source, starting the plurality of consumption servers; and executing consumption operation on the data to be consumed according to the plurality of consumption servers to obtain the third search data set.
In a possible example, the consuming unit 603 is specifically configured to, according to the third search data set obtained by performing a consuming operation on the data to be consumed by the consuming servers, specifically: extracting the data to be consumed from the preset message queue according to a preset data extraction rule; acquiring a data format to be consumed of the data to be consumed, and judging whether a preset data format set contains the data format to be consumed; if the data format set contains the data format to be consumed, determining a target consumption server corresponding to the data format to be consumed according to a preset mapping relation between the data format and a consumption server and the plurality of consumption servers; generating a data consumption instruction according to the data to be consumed, and sending the data consumption instruction to the target consumption server, wherein the data consumption instruction is used for indicating the target consumption server to execute consumption operation on the data to be consumed; receiving a data consumption response returned by the target consumption server, and extracting consumption data from the data consumption response; generating the third search data set in dependence on the consumption data.
In a possible example, the consuming unit 603 is further configured to: if the data format set does not contain the data format to be consumed, generating a data consumption reminder according to the data format to be consumed, wherein the data consumption reminder is used for reminding that a consumption server corresponding to the data format to be consumed is missing; and sending the data consumption prompt to a preset mobile terminal.
In a possible example, after storing the third search data in the preset first database and the preset second database, the storage unit 604 is further configured to: when input data of a target user are received, comparing the input data with the third search data set in the first database to obtain a plurality of comparison values; determining third search data corresponding to the maximum value in the comparison values as a search result; and sending the search result to the mobile terminal corresponding to the target user.
Embodiments of the present application also provide a computer storage medium, where the computer storage medium stores a computer program for electronic data exchange, the computer program enabling a computer to execute part or all of the steps of any one of the methods described in the above method embodiments, and the computer includes an electronic device.
Embodiments of the present application also provide a computer program product comprising a non-transitory computer readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods as described in the above method embodiments. The computer program product may be a software installation package, the computer comprising an electronic device.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present application is not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.
In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus may be implemented in other manners. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the above-described division of the units is only one type of division of logical functions, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of some interfaces, devices or units, and may be an electric or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit may be stored in a computer readable memory if it is implemented in the form of a software functional unit and sold or used as a stand-alone product. Based on such understanding, the technical solution of the present application may be substantially implemented or a part of or all or part of the technical solution contributing to the prior art may be embodied in the form of a software product stored in a memory, and including several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the above-mentioned method of the embodiments of the present application. And the aforementioned memory comprises: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable memory, which may include: flash Memory disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.
The foregoing detailed description of the embodiments of the present application has been presented to illustrate the principles and implementations of the present application, and the above description of the embodiments is only provided to help understand the method and the core concept of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
1. A search data processing method, comprising:
receiving a first search data set transmitted by a first data source, and receiving a plurality of search service records transmitted by a second data source;
generating a second search data set according to the plurality of search service records, and asynchronously filing the second search data set and the first search data set as data to be consumed to a preset message queue;
extracting the data to be consumed from the preset message queue and executing consumption operation to obtain a third search data set;
and storing the third search data set to a preset first database and a preset second database.
2. The method of claim 1, wherein generating a second search data set from the plurality of search service records comprises:
extracting a plurality of search service pipelines from the plurality of search service records, wherein any one of the plurality of search service pipelines comprises at least one search keyword;
executing search keyword extraction operation on the plurality of search service streams to obtain a plurality of search keyword sets corresponding to the plurality of search service streams;
and combining the plurality of search keyword sets to obtain the second search data set.
3. The method of claim 1, wherein the extracting the data to be consumed from the preset message queue and performing a consumption operation to obtain a third search data set comprises:
acquiring a plurality of listeners corresponding to a plurality of preset consumption servers, wherein the plurality of listeners are used for monitoring data transmission operations of the first data source and the second data source;
when the plurality of listeners monitor data transmission operations of the first data source and/or the second data source, starting the plurality of consumption servers;
and executing consumption operation on the data to be consumed according to the plurality of consumption servers to obtain the third search data set.
4. The method of claim 3, wherein the obtaining the third search data set by performing the consumption operation on the data to be consumed according to the plurality of consumption servers comprises:
extracting the data to be consumed from the preset message queue according to a preset data extraction rule;
acquiring a data format to be consumed of the data to be consumed, and judging whether a preset data format set contains the data format to be consumed;
if the data format set contains the data format to be consumed, determining a target consumption server corresponding to the data format to be consumed according to a preset mapping relation between the data format and a consumption server and the plurality of consumption servers;
generating a data consumption instruction according to the data to be consumed, and sending the data consumption instruction to the target consumption server, wherein the data consumption instruction is used for indicating the target consumption server to execute consumption operation on the data to be consumed;
receiving a data consumption response returned by the target consumption server, and extracting consumption data from the data consumption response;
generating the third search data set in dependence on the consumption data.
5. The method of claim 4, further comprising:
if the data format set does not contain the data format to be consumed, generating a data consumption reminder according to the data format to be consumed, wherein the data consumption reminder is used for reminding that a consumption server corresponding to the data format to be consumed is missing;
and sending the data consumption prompt to a preset mobile terminal.
6. The method according to claim 1, wherein after storing the third search data in the preset first database and the preset second database, further comprising:
when input data of a target user are received, comparing the input data with the third search data set in the first database to obtain a plurality of comparison values;
determining third search data corresponding to the maximum value in the comparison values as a search result;
and sending the search result to the mobile terminal corresponding to the target user.
7. A search data processing apparatus, characterized by comprising:
the receiving unit is used for receiving a first search data set transmitted by a first data source and receiving a plurality of search service records transmitted by a second data source;
the generating unit is used for generating a second search data set according to the plurality of search service records, and asynchronously archiving the second search data set and the first search data set to a preset message queue as data to be consumed;
the consumption unit is used for extracting the data to be consumed from the preset message queue and executing consumption operation to obtain a third search data set;
and the storage unit is used for storing the third search data set to a preset first database and a preset second database.
8. The apparatus according to claim 7, wherein, in said generating a second search data set from the plurality of search service records, the generating unit is specifically configured to:
extracting a plurality of search service pipelines from the plurality of search service records, wherein any one of the plurality of search service pipelines comprises at least one search keyword;
executing search keyword extraction operation on the plurality of search service streams to obtain a plurality of search keyword sets corresponding to the plurality of search service streams;
and combining the plurality of search keyword sets to obtain the second search data set.
9. An electronic device comprising a processor, a memory, a communication device and one or more programs, the processor, the memory and the communication device being interconnected, wherein the communication device is configured for information interaction with an external device, the one or more programs being stored in the memory and configured for execution by the processor, the programs comprising instructions for performing the steps of the method of any of claims 1-6.
10. A computer-readable storage medium, characterized in that a computer program for electronic data exchange is stored, wherein the computer program causes a computer to perform the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010131859.1A CN111427917A (en) | 2020-02-29 | 2020-02-29 | Search data processing method and related product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010131859.1A CN111427917A (en) | 2020-02-29 | 2020-02-29 | Search data processing method and related product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111427917A true CN111427917A (en) | 2020-07-17 |
Family
ID=71551968
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010131859.1A Pending CN111427917A (en) | 2020-02-29 | 2020-02-29 | Search data processing method and related product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111427917A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111966636A (en) * | 2020-08-20 | 2020-11-20 | 中国农业银行股份有限公司 | Data file fusion method and device |
| CN115102672A (en) * | 2022-06-15 | 2022-09-23 | 珠海格力电器股份有限公司 | Data processing method, system, computer device and storage medium |
-
2020
- 2020-02-29 CN CN202010131859.1A patent/CN111427917A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111966636A (en) * | 2020-08-20 | 2020-11-20 | 中国农业银行股份有限公司 | Data file fusion method and device |
| CN115102672A (en) * | 2022-06-15 | 2022-09-23 | 珠海格力电器股份有限公司 | Data processing method, system, computer device and storage medium |
| CN115102672B (en) * | 2022-06-15 | 2024-04-05 | 珠海格力电器股份有限公司 | Data processing method, system, computer device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106936441B (en) | Data compression method and device | |
| CN106657213B (en) | File transmission method and device | |
| CN103970793B (en) | Information query method, client and server | |
| CN112162965B (en) | Log data processing method, device, computer equipment and storage medium | |
| US9552382B2 (en) | Reference counter integrity checking | |
| CN111813756B (en) | Log retrieval system, method and device, electronic equipment and storage medium | |
| CN104246767A (en) | Telemetry system for a cloud synchronization system | |
| US8752204B2 (en) | Identifying and redacting privileged information | |
| CN111935227A (en) | Method for uploading file through browser, browser and electronic equipment | |
| US20200204688A1 (en) | Picture book sharing method and apparatus and system using the same | |
| CN113010542B (en) | Service data processing method, device, computer equipment and storage medium | |
| CN111737443B (en) | Answer text processing method and device and key text determining method | |
| CN108846098B (en) | Information flow abstract generating and displaying method | |
| CN111427917A (en) | Search data processing method and related product | |
| CN112181835A (en) | Automatic testing method and device, computer equipment and storage medium | |
| CN110618999A (en) | Data query method and device, computer storage medium and electronic equipment | |
| CN113672748A (en) | Multimedia information playing method and device | |
| CN111475388A (en) | Data push test method and device, computer equipment and storage medium | |
| CN110442439B (en) | Task process processing method and device and computer equipment | |
| US20130325908A1 (en) | Systems and methods for storing data and eliminating redundancy | |
| CN103870476A (en) | Retrieval method and device | |
| CN115801765A (en) | File transmission method, device, system, electronic equipment and storage medium | |
| CN112416875B (en) | Log management method, device, computer equipment and storage medium | |
| CN114615325A (en) | Message pushing method and device, computer equipment and storage medium | |
| CN111507758B (en) | Investigation method, device, system and server based on semantic analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |