CN109800258B - Data file deployment method, device, computer equipment and storage medium - Google Patents
Data file deployment method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN109800258B CN109800258B CN201811503111.9A CN201811503111A CN109800258B CN 109800258 B CN109800258 B CN 109800258B CN 201811503111 A CN201811503111 A CN 201811503111A CN 109800258 B CN109800258 B CN 109800258B
- Authority
- CN
- China
- Prior art keywords
- data
- preset
- basic
- file
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000003860 storage Methods 0.000 title claims abstract description 13
- 238000013515 script Methods 0.000 claims abstract description 122
- 238000011161 development Methods 0.000 claims abstract description 66
- 238000012795 verification Methods 0.000 claims abstract description 57
- 238000004519 manufacturing process Methods 0.000 claims abstract description 49
- 238000010606 normalization Methods 0.000 claims abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 17
- 230000018109 developmental process Effects 0.000 claims description 64
- 238000004458 analytical method Methods 0.000 claims description 28
- 230000011218 segmentation Effects 0.000 claims description 17
- 238000001514 detection method Methods 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 15
- 230000005540 biological transmission Effects 0.000 claims description 11
- 238000006243 chemical reaction Methods 0.000 claims description 11
- 238000013075 data extraction Methods 0.000 claims description 7
- 238000007619 statistical method Methods 0.000 claims description 7
- 238000004806 packaging method and process Methods 0.000 claims description 6
- 238000013500 data storage Methods 0.000 claims description 2
- 238000013524 data verification Methods 0.000 claims description 2
- 230000006870 function Effects 0.000 description 16
- 238000010586 diagram Methods 0.000 description 4
- 238000007689 inspection Methods 0.000 description 4
- 238000012360 testing method Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data file deployment method, a device, computer equipment and a storage medium, wherein the method comprises the following steps: when a data file deployment request sent by a client is received, a file to be processed contained in the data file deployment request is acquired, data in the file to be processed is extracted through a development library to serve as basic data, a database script corresponding to the basic data is generated according to preset conditions, normalization verification is carried out on the database script, when a verification result is qualified, pre-deployment processing is carried out on the basic data to obtain target data, then a connection between the development library and a production library is established, and the target data and the database script are deployed into the production library.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a data file deployment method, apparatus, computer device, and storage medium.
Background
With the development of social economy, the business scope of some group companies is wider and wider, data to be stored in the database is also more and more, and the data is generally stored in a development library and a production library respectively, namely, the development library and the production library are separated, the development library is a database used by a developer for developing test, the production library is a database used by business operation, the production library generally performs a lot of business data processing, which often needs to consume a lot of system resources, and is generally required to be kept on in non-preset maintenance time.
When operators deploy data files to a production library, data import and data inspection are required to be performed in the production library, database scripts are generated according to data association after the inspection is passed, and because the data volume of the production library is large and the operation state needs to be kept always on, a great deal of time is required to be consumed for the operations, so that the efficiency of deploying the data files to the production library is low, meanwhile, the production library resources are easily caused to be insufficient under the conditions of large data volume and non-uniform data format, the deployment failure is caused, and the deployment stability is seriously influenced.
Disclosure of Invention
The embodiment of the invention provides a data file deployment method, a data file deployment device, computer equipment and a storage medium, which are used for solving the problems of low deployment efficiency and poor stability of the current data file deployment to a production library.
A data file deployment method, comprising:
if a data file deployment request sent by a client is received, acquiring a file to be processed contained in the data file deployment request;
writing the file to be processed into a preset development library, and extracting data in the file to be processed from the preset development library as basic data;
generating a database script corresponding to the basic data according to preset conditions;
performing normalization verification on the database script to obtain a verification result;
if the verification result is qualified, pre-deploying the basic data to obtain target data;
establishing a connection between the preset development library and a preset production library, and deploying the target data and the database script with qualified verification results into the preset production library through a preset data transmission script.
A data file deployment apparatus, comprising:
The file receiving module is used for acquiring a file to be processed contained in a data file deployment request sent by a client if the data file deployment request is received;
the data extraction module is used for writing the file to be processed into a preset development library, and extracting data in the file to be processed from the preset development library as basic data;
the script generation module is used for generating a database script corresponding to the basic data according to preset conditions;
the script verification module is used for carrying out normalization verification on the database script to obtain a verification result;
the target data acquisition module is used for performing pre-deployment processing on the basic data to obtain target data if the verification result is qualified;
the data deployment module is used for establishing the connection between the preset development library and the preset production library, and deploying the target data and the database script with qualified verification results into the preset production library through a preset data transmission script.
A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the data file deployment method described above when executing the computer program.
A computer readable storage medium storing a computer program which when executed by a processor implements the data file deployment method described above.
According to the data file deployment method, the device, the computer equipment and the storage medium, when a data file deployment request sent by a client is received, a file to be processed contained in the data file deployment request is acquired, the file to be processed is further written into a development library, data in the file to be processed is extracted through the development library to serve as basic data, then a database script corresponding to the basic data is generated according to preset conditions, and the database script is subjected to normative verification to obtain a verification result, when the verification result is qualified, the basic data is subjected to pre-deployment processing to obtain target data, further connection between the development library and a production library is established, the target data and the database script are deployed into the preset production library through a preset data transmission script, the data file is led into the development library for processing, and because the development library does not need to be connected with application data in real time, system resources in the development library are occupied less, the development library can be used for carrying out data processing, the efficiency of the data processing on the data file is improved, and the data is effectively prevented from being subjected to the verification of the database or the data is not sufficient, the data file is subjected to the corresponding data file is deployed into the database, the database is not required to be deployed, the stability is improved, and the stability of the data file is improved, and the data file is deployed in the corresponding data is deployed in the production database, and the data is deployed through the database is deployed.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments of the present invention will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic view of an application environment of a data file deployment method according to an embodiment of the present invention;
FIG. 2 is a flowchart of an implementation of a data file deployment method provided by an embodiment of the present invention;
FIG. 3 is a flowchart illustrating implementation of step S20 in the data file deployment method according to the embodiment of the present invention;
FIG. 4 is a flowchart illustrating implementation of step S21 in the data file deployment method according to the embodiment of the present invention;
FIG. 5 is a flowchart illustrating implementation of step S30 in the data file deployment method according to the embodiment of the present invention;
FIG. 6 is a flowchart illustrating implementation of step S40 in the data file deployment method according to the embodiment of the present invention;
FIG. 7 is a schematic diagram of a data file deployment apparatus according to an embodiment of the present invention;
fig. 8 is a schematic diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, fig. 1 illustrates an application environment of a data file deployment method according to an embodiment of the present invention. The data file deployment method is applied to a data file deployment scene of deploying data in the business data file to the production library. The data file deployment scene comprises a client and a server, wherein the client and the server are connected through a network, the client sends a data file deployment request containing a file to be processed to the server, the server receives the file to be processed and extracts data, the extracted data is analyzed and processed through a development library to generate a database script, the processed data and the database script are deployed into a production library, the client can be a notebook computer, a personal computer, a mobile phone, a portable handheld intelligent device, a wearable intelligent device and the like, and the server can be realized by a server cluster formed by an independent server or a plurality of servers.
Referring to fig. 2, fig. 2 shows a data file deployment method according to an embodiment of the present invention, and the method is applied to the server in fig. 1 for illustration, and is described in detail as follows:
s10: and if the data file deployment request sent by the client is received, acquiring a file to be processed contained in the data file deployment request.
Specifically, a service person sends a data file deployment request to a server through a client, and when the server receives the data file deployment request, the server acquires a file to be processed contained in the data file deployment request through a network transmission protocol.
The data file deployment request refers to a request for deploying a data file, in this embodiment, data contained in a file to be processed is deployed to a production library, and a request for generating a database script that can be used for performing a correlation query between data in the file to be processed is generated in the production library.
The files to be processed are business data files which are collected and input by business personnel and are required to be deployed into a production library.
Among them, network transport protocols include, but are not limited to: internet control message protocol (Internet Control Message Protocol, ICMP), simple network management protocol (Simple Network Management Protocol, SNMP) address resolution protocol (ARP Address Resolution Protocol, ARP), reverse address resolution protocol (Reverse ARP Address Resolution Protocol, RARP), and file transfer protocol (File Transfer Protocol, FTP), among others.
S20: writing the file to be processed into a preset development library, and extracting data in the file to be processed from the preset development library as basic data.
Specifically, after receiving a file to be processed contained in a data file deployment request, the server writes the file to be processed into a preset development library, and performs data extraction on the file to be processed in the preset development library to obtain data contained in the file to be processed as basic data.
The preset development library is used in the process of programming and testing programs by a developer, and can be used for programming and storing various functions and functional components to be used in advance, the development library is started when the task is carried out, and is not required to be started when the task is not carried out, namely, the development library is in an idle state most of the time, and a large amount of available system resources are possessed.
In this embodiment, the file write script may be written and stored in the development library in advance, the file to be processed may be read by using the file write script, and the file to be processed may be written into the development library.
The script type of the file writing script may be, but is not limited to: shell script, javaScript, VBScript, SQL script, lua script, python script, etc.
Preferably, the script type of the file writing script adopted in the embodiment is a shell script.
It should be noted that, after reading the data in the file to be processed, the data needs to be formatted uniformly, that is, the data with a non-preset format is converted in format, and the data after the format conversion is used as the basic data, so as to ensure that the database script can be generated through the basic data later.
S30: and generating a database script corresponding to the basic data according to preset conditions.
Specifically, at present, data in a file to be processed is mainly imported into a production library, and then a database script file is generated in the production library according to the association between the data, so that the data can be quickly inquired and classified through the database script file, and in order to avoid the situation that the production library resources are tense due to the fact that the operation is executed in the production library, the production library is unstable, in the preset development library, the embodiment executes basic data extracted from the file to be processed, and generates a database script corresponding to the basic data.
The preset condition may be attribute information of the basic data, or may be association between the basic data, which may be specifically determined according to actual situations, and is not limited herein.
S40: and carrying out normalization verification on the database script to obtain a verification result.
Specifically, after the database script is generated, checking the normalization of the database script to obtain a check result, wherein the check result comprises pass and fail.
Among the ways in which the normalization of the database scripts may be verified include, but are not limited to: batch processing, regular expressions, database query analyzers, triggers, preset script files, etc., may be selected according to actual needs, and are not particularly limited herein.
It should be noted that, the normalization verification of the database script file may be implemented not only in the above manner, but also using a third party tool, for example, using database script normalization format tool SQLinForm to perform normalization inspection on the database script.
Preferably, the database query analyzer is used to verify the normalization of the database script in this embodiment, and the specific process may refer to the descriptions of step S41 to step S43, so that the description is omitted here for avoiding repetition.
S50: and if the verification result is qualified, pre-deploying the basic data to obtain target data.
Specifically, when the normalization check result of the database script is qualified, pre-deploying the basic data to obtain target data.
The target data is data meeting the data storage standard in the production library, and may specifically be a data table containing basic data, or may be spatial data generated by the basic data and capable of reflecting the data relationship between the basic data, or may be set according to actual needs, which is not limited herein.
In this embodiment, since the data in the production library is stored in the form of a data table, in this embodiment, as a preferable mode, the target data is set in the form of a data table having the same data table template as that in the production library.
Wherein, the pre-deployment processing comprises: acquiring a pre-stored data table template; storing each basic data into a corresponding position in a database template according to the attribute of each basic data; and obtaining target data according to the database template and the basic data stored in the database template.
S60: establishing a connection between a preset development library and a preset production library, and deploying the database script with qualified target data and verification results into the preset production library through a preset data transmission script.
Specifically, after target data is obtained, a connection between a preset development library and a preset production library is established, and the target data and the database script are deployed into the preset production library through a preset data transmission script.
The preset script is used for rapidly deploying the target data generated in the development library to a corresponding position in the production library, for example, in this embodiment, the preset script may specifically be:
copy from epciscde/cde6953@EPCISDVP append AHS_MBANK_INFO_TMP using select*from AHS_MBANK_INFO_TMP
wherein, "epciscde" is an access user name of the development library, "cde6953" is an access password corresponding to the access user name "epciscde" of the development library, "epcildbp" is an instance name of the development library, "ahs_mbank_info_tmp" is an identification of the target data, and in this example, is a table name corresponding to the target data.
In this embodiment, when a data file deployment request sent by a client is received, a to-be-processed file contained in the data file deployment request is acquired, the to-be-processed file is further written into a development library, data in the to-be-processed file is extracted as basic data through the development library, a database script corresponding to the basic data is generated according to preset conditions, normalization verification is performed on the database script, a verification result is obtained, when the verification result is qualified, pre-deployment processing is performed on the basic data to obtain target data, a connection between the development library and a production library is further established, and through a preset data transmission script, the database script with the qualified target data and the verification result is deployed into a preset production library, so that the data file is led into the development library for processing.
The specific implementation method for extracting the data in the file to be processed from the preset development library mentioned in step S20 as the basic data will be described in detail below by using a specific embodiment based on the corresponding embodiment of fig. 2.
Referring to fig. 3, fig. 3 shows a specific implementation flow of step S20 provided in the embodiment of the present invention, which is described in detail below:
s21: and extracting data in the file to be processed from a preset development library as initial data.
Specifically, in a preset development library, extracting data of a file to be processed to obtain data contained in the file to be processed as initial data.
S22: and performing data segmentation operation on the initial data according to a preset data segmentation method to obtain at least one piece of segmented data.
Specifically, according to a preset data segmentation method, performing data segmentation operation on the initial data to obtain one or more pieces of segmented data.
The preset data segmentation method includes, but is not limited to: the data partitioning method, regular expression, score algorithm and the like can be specifically set according to actual application requirements, and are not particularly limited.
Preferably, the data dividing method adopted in the present embodiment is a data dividing method, and the text content is divided into the data identifiers and the data content corresponding to each data identifier by acquiring the boundary identifiers in the text content with the boundary identifiers as intervals.
For example, in one embodiment, the acquired data is "name: thirdly, stretching; gender: a male; age: 22", using the semicolons as the division marks of the data division, traversing the semicolons in the data to obtain two semicolons, and dividing the data into three groups of names respectively by using the two semicolons: zhang Sanj "," sex: male "and" age: 22", the three sets of data are three split data.
S23: and carrying out normalization verification on each piece of segmentation data according to a preset data standard, and selecting the segmentation data meeting the preset data standard as basic data.
Specifically, according to a preset data standard, carrying out normalization verification on each piece of divided data, and selecting the divided data meeting the preset data standard as basic data according to a verification result.
The normalization check refers to checking the value range and the data type of the split data, and taking the split data as basic data when the value range and the data type of the split data meet preset data standards.
In this embodiment, data in a file to be processed is extracted from a preset development library and is used as initial data, then data segmentation operation is performed on the initial data according to a preset data segmentation method to obtain at least one segmented data, normalization verification is performed on each segmented data according to a preset data standard, segmented data meeting the preset data standard is selected as basic data, so that basic data meeting requirements is analyzed from the file to be processed, and the efficiency of generating a database script through the basic data is improved.
Based on the corresponding embodiment of fig. 3, the specific implementation method for extracting the data in the file to be processed in the preset development library mentioned in step S21 as the initial data is described in detail below by using a specific embodiment.
Referring to fig. 4, fig. 4 shows a specific implementation flow of step S21 provided in the embodiment of the present invention, which is described in detail below:
s211: and extracting data in the file to be processed from a preset development library.
Specifically, the data in the file to be processed is read from a preset development library through the function stored in the preset development library.
The data to be processed may be loaded into a file to be processed, and then read from a memory, or may be directly read from the file to be processed, where the data to be processed may be service data in the insurance field, for example: policy number, premium amount, policy start time, etc.
S212: and judging the data type of the extracted data to obtain the data type of the extracted data.
Specifically, the data type of the data to be processed can be obtained through the typeof function, and the data type of the data can be obtained, so that whether the data type of the data belongs to a preset data type or not can be confirmed.
S213: if the data type of the extracted data does not belong to the preset data type, performing data type conversion on the extracted data according to a preset data type conversion method to obtain initial data belonging to the preset data type, and if the data type of the extracted data belongs to the preset data type, taking the extracted data as the initial data.
Specifically, when the data type of the extracted data does not belong to the preset data type, performing data type conversion on the extracted data according to a preset data type conversion method to obtain initial data belonging to the preset data type, and directly taking the extracted data as the initial data when the data type of the extracted data belongs to the preset data type, so that the data types of the obtained initial data belong to the preset data type, the subsequent identification of the initial data is facilitated, and the generation of a database script is performed by using the initial data.
In this embodiment, data in a file to be processed is extracted from a preset development library, and a data type judgment is performed on the extracted data to obtain a data type of the extracted data, when the data type of the extracted data does not belong to a preset data type, according to a preset data type conversion method, the extracted data is subjected to data type conversion to obtain initial data belonging to the preset data type, when the data type of the extracted data belongs to the preset data type, the extracted data is used as the initial data, so that the data types in the obtained initial data are all preset data types, and the subsequent data segmentation analysis is facilitated.
The specific implementation method for generating the database script corresponding to the basic data according to the preset condition mentioned in step S30 will be described in detail below by means of a specific embodiment on the basis of the corresponding embodiment of fig. 2.
Referring to fig. 5, fig. 5 shows a specific implementation flow of step S30 provided in the embodiment of the present invention, which is described in detail below:
s31: traversing each piece of basic data according to the data identification of the basic data to obtain each piece of field content contained in the basic data, and carrying out statistical analysis on the field content to obtain the field attribute corresponding to each piece of field content.
Specifically, traversing each piece of basic data according to the data identification of the basic data to obtain each piece of field content contained in the basic data, and carrying out statistical analysis on the field content to obtain the field attribute corresponding to each piece of field content.
Wherein Traversal (Traversal) refers to access to information of all nodes in a node tree identified by data in the base data, i.e., each node in the node tree is accessed once and only once in turn. Traversing means include, but are not limited to: preamble traversal, midamble traversal, and postamble traversal, etc.
Preferably, the traversal method used in the embodiment of the present invention is a preamble traversal, where the sequence of the preamble traversal includes NLR or NRL, where N refers to a root (Node), L refers to a Left Node (Left subtree) of the root, and R refers to a Right Node (Right subtree) of the root.
In this embodiment, the server is preset with a plurality of field contents, each field content corresponds to one or more data identifiers, according to the data identifiers, the field content to which the basic data belongs can be determined, for example, one obtained basic data identifier is CX-ZD-512036, according to the preset correspondence between the field content and the data identifier, the field content corresponding to the data identifier with the first two bits being CX is determined to be a risk, and then the field content to which the data identifier belongs can be determined to be a risk.
And carrying out statistical analysis on the field contents to obtain field attributes corresponding to each field content, namely, carrying out analysis and statistics on basic data contained in each field content by using a pointer to obtain the attributes and the associated information of the basic data in the field content.
S32: and generating a basic sentence set containing at least one basic sentence according to preset database sentence generation conditions, each field content and the field attribute corresponding to each field content.
Specifically, a basic sentence set containing at least one basic sentence is generated according to preset database sentence generation conditions, each field content and a field attribute corresponding to each field content.
The preset database statement generation conditions can be set according to actual needs.
For example, in one embodiment, the preset conditions are: inquiring basic data with the value of the field content A being more than 1000, acquiring an inquiry result, and generating related basic sentences as follows:
select field content A from temporary table where value >1000
Wherein value is the value of the basic data in field a, and another preset condition is: querying all repeated basic data, removing redundant repeated parts, and generating basic sentences as follows:
select distint field A from temporary table
Wherein distict indicates that the repeated results only remain one, and the finally obtained results are different.
S33: and carrying out relevance detection on the basic sentences in the basic sentence set, and taking the basic sentences with relevance as a group of relevance sentences.
Specifically, the relevance detection is performed on the basic sentences in the basic sentence set, and the basic sentences with relevance to each other in the detection result are used as a group of relevance sentences.
The relevance refers to the property that two or more basic sentences have one or more common characteristics and do not conflict with each other, for example, one basic sentence is color information of the query field content B, the other basic sentence is sales information of the query field content B, the two basic sentences are attribute information of the query field content B, and the two query information do not conflict, so that the two basic sentences can be used as a group of related sentences, for example, one basic sentence is yellow basic data of the query field content C, the other basic sentence is red basic data of the query field content C, and the two basic sentences have the same characteristics, but the query content has conflict because of no relevance.
S34: and executing database statement stitching instructions for each group of associated statements to obtain a target statement set containing at least one target statement, and deleting the associated statements from the base statement set.
Specifically, for each group of related sentences, the related sentences are spliced through a database sentence splicing instruction to obtain a target sentence set containing at least one target sentence, and after the target sentence is obtained according to the related sentences, the related sentences are deleted from the basic sentence set.
Easily understood, the base sentences with the association are spliced, so that the number of subsequent data processing times can be reduced, and the data processing efficiency can be improved.
Taking the example in step S32 as an example, if it is required to obtain basic data with a value greater than 1000 and not repeated in the field content a, the two basic sentences are spliced into: the select discrete field content A from temporary table where value >1000.
S35: and packaging the basic sentence set and the target sentence set into a database script.
Specifically, the basic sentence set and the target sentence set are packaged into the database script, so that the database script can be directly imported into the production library in the follow-up process, the database script does not need to be generated in the production library with fewer idle resources, and the generation efficiency of the database script is improved.
The package is mainly that a query method in QueryRunner class in a DbUtil component is used for writing a query statement into a List, and finally, the content in the List is stored into an execution statement position of an SQL script.
In this embodiment, according to the data identifier of the basic data, traversing each basic data to obtain each field content contained in the basic data, performing statistical analysis on the field content to obtain a field attribute corresponding to each field content, generating a basic sentence set including at least one basic sentence according to a preset database sentence generation condition, each field content and a field attribute corresponding to each field content, simultaneously performing relevance detection on the basic sentences in the basic sentence set, taking the basic sentences with relevance as a group of associated sentences, executing a database sentence splicing instruction for each group of associated sentences to obtain a target sentence set including at least one target sentence, deleting the associated sentence from the basic sentence set, and packaging the basic sentence set and the target sentence set to obtain a database script, thereby realizing automatic generation of the database script and improving the intelligentization degree of the generated database script.
On the basis of the corresponding embodiment of fig. 2, a detailed description will be given below of a specific implementation method for performing normalization verification on the database script mentioned in step S40 to obtain a verification result according to a specific embodiment.
Referring to fig. 6, fig. 6 shows a specific implementation flow of step S40 provided in the embodiment of the present invention, which is described in detail below:
s41: and reading each statement contained in the database script as a statement to be detected.
Specifically, each sentence contained in the database script is obtained by reading the database script, and each sentence is used as a sentence to be detected.
S42: and adopting a database script analysis tool to perform grammar checking and functional detection on the sentences to be detected to obtain an analysis result.
Specifically, the database script analysis tool performs grammar inspection and functional detection on the sentences to be detected to obtain analysis results, wherein the analysis results can be that the sentences to be detected are qualified, or one or more error prompts.
Preferably, the database script analysis tool employed in the present embodiment is a detection function in a query analyzer.
The query analyzer is a graphical query tool in the database and is used for editing a transaction-SQL sentence in a text mode and receiving analysis execution results. Using this tool, a user can interactively design and test Transact-SQL statements, batches, and scripts. The query analyzer may be opened directly from the SQL Server enterprise manager or from Microsoft SQL Server.
The detection functions in the query analyzer include, but are not limited to, a Paraseonly function, a Noexec function, and the like, when the Paraseonly function is used, the query analyzer compiles only the acquired statement, when the Noexec function is used, the query analyzer compiles and executes the acquired statement, and a feedback message is returned according to an execution result, in this embodiment, only the statement and the function need to be checked, that is, only the statement is required to be known to be correct, and the reason of the error does not need to be known when the error occurs, so in order to improve analysis efficiency, the embodiment adopts a preferred scheme, that is, the Paraseonly function is adopted to perform statement analysis.
S43: if the analysis result contains one or more error prompts, determining that the verification result of the database script is unqualified, and if the analysis result does not contain the error prompts, determining that the verification result of the database script is qualified.
Specifically, when the analysis result contains one or more error prompts, determining that the verification result of the database script is unqualified, at the moment, notifying a service personnel of the client of the verification result in a form of sending a prompt message, and when the analysis result does not contain the error prompts, determining that the verification result of the database script is qualified, and executing a subsequent pre-deployment method.
In this embodiment, each statement included in the database script is read and used as a statement to be detected, and a database script analysis tool is used to perform grammar checking and functional detection on the statement to be detected to obtain an analysis result, if the analysis result includes one or more error prompts, the verification result of the database script is determined to be unqualified, and if the analysis result does not include the error prompts, the verification result of the database script is determined to be qualified, so that the normalization of the database script is rapidly verified, and the efficiency of the normalization verification of the database script is improved.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present invention.
Fig. 7 shows a schematic block diagram of a data file deployment apparatus in one-to-one correspondence with the data file deployment method of the above embodiment. As shown in fig. 7, the data file deployment apparatus includes a file receiving module 10, a data extracting module 20, a script generating module 30, a script verifying module 40, a target data acquiring module 50, and a data deployment module 60. The functional modules are described in detail as follows:
The file receiving module 10 is configured to obtain a file to be processed included in a data file deployment request if the data file deployment request sent by the client is received;
the data extraction module 20 is configured to write a file to be processed into a preset development library, and extract data in the file to be processed in the preset development library as basic data;
the script generation module 30 is configured to generate a database script corresponding to the basic data according to a preset condition;
the script checking module 40 is configured to perform normalization check on the database script to obtain a checking result;
the target data acquisition module 50 is configured to perform pre-deployment processing on the base data if the verification result is qualified, so as to obtain target data;
the data deployment module 60 is configured to establish a connection between a preset development library and a preset production library, and deploy, through a preset data transmission script, the target data and the database script with the qualified verification result into the preset production library.
Further, the data extraction module 20 includes:
a data acquisition unit 21, configured to extract data in a file to be processed in a preset development library, as initial data;
a data dividing unit 22, configured to perform a data dividing operation on the initial data according to a preset data dividing method, to obtain at least one divided data;
The data verification unit 23 is configured to perform normalization verification on each piece of divided data according to a preset data standard, and select the piece of divided data that meets the preset data standard as the base data.
Further, the data acquisition unit 21 includes:
a data extraction subunit 211, configured to extract data in a file to be processed from a preset development library;
a type determining subunit 212, configured to perform data type determination on the extracted data, so as to obtain a data type of the extracted data;
the data determining subunit 213 is configured to perform data type conversion on the extracted data according to a preset data type conversion method if the data type of the extracted data does not belong to the preset data type, to obtain initial data that belongs to the preset data type, and to use the extracted data as the initial data if the data type of the extracted data belongs to the preset data type.
Further, the script generation module 30 includes:
the data traversing unit 31 is configured to traverse each piece of basic data according to the data identifier of the basic data, obtain each piece of field content contained in the basic data, and perform statistical analysis on the field content to obtain a field attribute corresponding to each piece of field content;
A basic sentence generating unit 32, configured to generate a basic sentence set including at least one basic sentence according to a preset database sentence generating condition, each field content, and a field attribute corresponding to each field content;
a relevance detection unit 33, configured to perform relevance detection on basic sentences in the basic sentence set, and take the basic sentences with relevance as a group of relevance sentences;
an associated sentence generating unit 34, configured to execute a database sentence splicing instruction for each group of associated sentences, obtain a target sentence set containing at least one target sentence, and delete the associated sentences from the base sentence set;
the sentence packaging unit 35 is configured to package the basic sentence set and the target sentence set into a database script.
Further, the script verification module 40 includes:
a sentence reading unit 41 for reading each sentence contained in the database script as a sentence to be detected;
a sentence analysis unit 42, configured to perform grammar checking and functionality detection on the sentence to be detected by using a database script analysis tool, so as to obtain an analysis result;
the result determining unit 43 is configured to determine that the verification result of the database script is not qualified if the analysis result includes one or more error prompts, and determine that the verification result of the database script is qualified if the analysis result does not include an error prompt.
For specific limitations on the data file deployment apparatus, reference may be made to the above limitation on the data file deployment method, and no further description is given here. The respective modules in the above-described data file deployment apparatus may be implemented in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
Fig. 8 is a schematic diagram of a computer device according to an embodiment of the present invention. The computer device may be a server, and its internal structure may be as shown in fig. 8. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing a development library, a production library and a preset data transmission script. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data file deployment method.
In one embodiment, a computer device is provided that includes a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the steps of the data file deployment method of the above embodiment, such as steps S10 to S60 shown in fig. 2. Alternatively, the processor, when executing the computer program, implements the functions of the modules/units of the data file deployment apparatus of the above embodiment, such as the functions of the modules 10 to 60 shown in fig. 7. In order to avoid repetition, a description thereof is omitted.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions.
In an embodiment, a computer readable storage medium is provided, on which a computer program is stored, which when executed by a processor implements the steps of the data file deployment method of the above embodiment, or which when executed by a processor implements the functions of the modules/units in the data file deployment device of the above embodiment. In order to avoid repetition, a description thereof is omitted.
It will be appreciated that the computer readable storage medium may comprise: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), an electrical carrier wave signal, a telecommunications signal, and the like.
The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention, and are intended to be included in the scope of the present invention.
Claims (7)
1. A data file deployment method, characterized in that the data file deployment method comprises:
if a data file deployment request sent by a client is received, acquiring a file to be processed contained in the data file deployment request;
Writing the file to be processed into a preset development library, and extracting data in the file to be processed from the preset development library as basic data;
generating a database script corresponding to the basic data according to preset conditions;
performing normalization verification on the database script to obtain a verification result;
if the verification result is qualified, pre-deploying the basic data to obtain target data;
establishing a connection between the preset development library and a preset production library, and deploying the target data and the database script with qualified verification results into the preset production library through a preset data transmission script;
the extracting the data in the file to be processed in the preset development library as basic data comprises the following steps:
extracting data in the file to be processed from the preset development library as initial data;
according to a preset data segmentation method, carrying out data segmentation operation on the initial data to obtain at least one piece of segmentation data;
performing normalization verification on each piece of divided data according to a preset data standard, and selecting the piece of divided data meeting the preset data standard as the basic data;
The extracting the data in the file to be processed in the preset development library, as initial data, includes:
extracting data in the file to be processed from the preset development library;
judging the data type of the extracted data to obtain the data type of the extracted data;
if the data type of the extracted data does not belong to the preset data type, performing data type conversion on the extracted data according to a preset data type conversion method to obtain the initial data belonging to the preset data type, and if the data type of the extracted data belongs to the preset data type, taking the extracted data as the initial data;
the basic data comprises a data identifier and field content, and the generating of the database script corresponding to the basic data according to preset conditions comprises the following steps:
traversing each piece of basic data according to the data identification of the basic data to obtain each piece of field content contained in the basic data, and carrying out statistical analysis on the field content to obtain a field attribute corresponding to each piece of field content;
Generating a basic sentence set containing at least one basic sentence according to preset database sentence generation conditions, each field content and a field attribute corresponding to each field content;
performing relevance detection on the basic sentences in the basic sentence set, and taking the basic sentences with relevance as a group of relevance sentences;
executing database statement splicing instructions for each group of associated statements to obtain a target statement set containing at least one target statement, and deleting the associated statements from the base statement set;
packaging the basic sentence set and the target sentence set into the database script;
the target data are data which meet the data storage standard in the production library, and the pre-deployment processing comprises the following steps:
acquiring a pre-stored data table template;
storing each basic data into a corresponding position in a database template according to the attribute of each basic data;
and obtaining target data according to the database template and the basic data stored in the database template.
2. The method for deploying a data file according to claim 1, wherein the performing a normalization check on the database script to obtain a check result comprises:
Reading each statement contained in the database script as a statement to be detected;
adopting a database script analysis tool to carry out grammar checking and functional detection on the statement to be detected to obtain an analysis result;
if the analysis result comprises one or more error prompts, determining that the verification result of the database script is unqualified, and if the analysis result does not comprise the error prompts, determining that the verification result of the database script is qualified.
3. A data file deployment device, the file deployment device comprising:
the file receiving module is used for acquiring a file to be processed contained in a data file deployment request sent by a client if the data file deployment request is received;
the data extraction module is used for writing the file to be processed into a preset development library, and extracting data in the file to be processed from the preset development library as basic data;
the script generation module is used for generating a database script corresponding to the basic data according to preset conditions;
the script verification module is used for carrying out normalization verification on the database script to obtain a verification result;
The target data acquisition module is used for performing pre-deployment processing on the basic data to obtain target data if the verification result is qualified;
the data deployment module is used for establishing the connection between the preset development library and the preset production library and deploying the target data and the database script with qualified verification results into the preset production library through a preset data transmission script;
the data file deployment device is configured to implement the method of claim 1.
4. The data file deployment apparatus of claim 3 wherein the data extraction module comprises:
the data acquisition unit is used for extracting data in the file to be processed from the preset development library and taking the data as initial data;
the data segmentation unit is used for carrying out data segmentation operation on the initial data according to a preset data segmentation method to obtain at least one piece of segmentation data;
and the data verification unit is used for carrying out normalization verification on each piece of divided data according to a preset data standard, and selecting the piece of divided data meeting the preset data standard as the basic data.
5. The data file deployment apparatus of claim 3 wherein the script generation module comprises:
The data traversing unit is used for traversing each piece of basic data according to the data identification of the basic data to obtain each piece of field content contained in the basic data, and carrying out statistical analysis on the field content to obtain a field attribute corresponding to each piece of field content;
the basic sentence generating unit is used for generating a basic sentence set containing at least one basic sentence according to preset database sentence generating conditions, each field content and the field attribute corresponding to each field content;
the relevance detection unit is used for carrying out relevance detection on the basic sentences in the basic sentence set, and taking the basic sentences with relevance as a group of relevance sentences;
the related statement generation unit is used for executing database statement splicing instructions for each group of related statements to obtain a target statement set containing at least one target statement, and deleting the related statements from the basic statement set;
and the statement packaging unit is used for packaging the basic statement set and the target statement set into the database script.
6. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the data file deployment method of any of claims 1 to 2 when the computer program is executed.
7. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the data file deployment method according to any one of claims 1 to 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811503111.9A CN109800258B (en) | 2018-12-10 | 2018-12-10 | Data file deployment method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811503111.9A CN109800258B (en) | 2018-12-10 | 2018-12-10 | Data file deployment method, device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109800258A CN109800258A (en) | 2019-05-24 |
| CN109800258B true CN109800258B (en) | 2024-01-26 |
Family
ID=66556519
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811503111.9A Active CN109800258B (en) | 2018-12-10 | 2018-12-10 | Data file deployment method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109800258B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111460152A (en) * | 2020-03-27 | 2020-07-28 | 深圳价值在线信息科技股份有限公司 | Extraction method, extraction device and extraction equipment for announcement text content |
| CN111737349B (en) * | 2020-06-18 | 2023-09-19 | 中国银行股份有限公司 | Data consistency verification method and device |
| CN111984670B (en) * | 2020-07-27 | 2024-03-19 | 北京天健源达科技股份有限公司 | Method for generating script of annotation information of display view |
| CN112035355A (en) * | 2020-08-28 | 2020-12-04 | 中国平安财产保险股份有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN112507353A (en) * | 2020-11-27 | 2021-03-16 | 中原银行股份有限公司 | Data extraction method and device |
| CN113190236B (en) * | 2021-05-27 | 2024-02-02 | 中国工商银行股份有限公司 | HQL script verification method and device |
| CN114979096A (en) * | 2022-05-19 | 2022-08-30 | 浪潮软件集团有限公司 | Domestic CPU and Os script task batch issuing platform and method |
| CN115904487B (en) * | 2022-11-29 | 2023-08-18 | 广发银行股份有限公司 | Analytical data caliber management method, system, equipment and storage medium |
| CN116776849B (en) * | 2023-06-06 | 2024-06-18 | 深圳市平方科技股份有限公司 | Template editing method and device for ship EDI, electronic equipment and storage medium |
| CN117194437B (en) * | 2023-11-02 | 2024-02-02 | 北京国电通网络技术有限公司 | File storage method, device, electronic equipment and computer readable medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7010546B1 (en) * | 2000-09-20 | 2006-03-07 | Parasoft Corporation | Method and system for testing data sources and database oriented software applications |
| CN103678485A (en) * | 2013-12-26 | 2014-03-26 | 中国运载火箭技术研究院 | Virtual experiment process node driving and movable packaging system |
| CN103885796A (en) * | 2014-03-03 | 2014-06-25 | 西安电子科技大学 | Software dynamic deploying method based on script tool |
| CN105511872A (en) * | 2015-11-30 | 2016-04-20 | 国云科技股份有限公司 | Application automatic deployment method based on cloud computing platform |
| CN106293820A (en) * | 2016-08-02 | 2017-01-04 | 山东大学 | Exploitation test O&M integral system, deployment, full dose and increment updating method |
| CN106991104A (en) * | 2016-01-21 | 2017-07-28 | 泰康保险集团股份有限公司 | Database script disposes device and database script dispositions method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7512942B2 (en) * | 2005-08-24 | 2009-03-31 | International Business Machines Corporation | Model-driven software deployment in an application server |
-
2018
- 2018-12-10 CN CN201811503111.9A patent/CN109800258B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7010546B1 (en) * | 2000-09-20 | 2006-03-07 | Parasoft Corporation | Method and system for testing data sources and database oriented software applications |
| CN103678485A (en) * | 2013-12-26 | 2014-03-26 | 中国运载火箭技术研究院 | Virtual experiment process node driving and movable packaging system |
| CN103885796A (en) * | 2014-03-03 | 2014-06-25 | 西安电子科技大学 | Software dynamic deploying method based on script tool |
| CN105511872A (en) * | 2015-11-30 | 2016-04-20 | 国云科技股份有限公司 | Application automatic deployment method based on cloud computing platform |
| CN106991104A (en) * | 2016-01-21 | 2017-07-28 | 泰康保险集团股份有限公司 | Database script disposes device and database script dispositions method |
| CN106293820A (en) * | 2016-08-02 | 2017-01-04 | 山东大学 | Exploitation test O&M integral system, deployment, full dose and increment updating method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109800258A (en) | 2019-05-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109800258B (en) | Data file deployment method, device, computer equipment and storage medium | |
| US10872034B2 (en) | Method, device and computer program product for executing test cases | |
| US10769228B2 (en) | Systems and methods for web analytics testing and web development | |
| CN112181804B (en) | Parameter verification method, device and storage medium | |
| US9529662B1 (en) | Dynamic rule-based automatic crash dump analyzer | |
| CN109829287A (en) | Api interface permission access method, equipment, storage medium and device | |
| CN108459850B (en) | Method, device and system for generating test script | |
| CN113271237A (en) | Industrial control protocol analysis method and device, storage medium and processor | |
| CN111435367B (en) | Knowledge graph construction method, system, equipment and storage medium | |
| CN112511384B (en) | Flow data processing method and device, computer equipment and storage medium | |
| CN111831574B (en) | Regression test planning method, regression test planning device, computer system and medium | |
| CN112445692B (en) | Case testing method and terminal | |
| CN112463596B (en) | Test case data processing method, device and equipment and processing equipment | |
| CN112363939A (en) | Method, system and equipment for quickly generating fuzzy test network protocol template | |
| CN115809466B (en) | Security requirement generation method and device based on STRIDE model, electronic equipment and medium | |
| CN112235124A (en) | Method and device for configuring pico-cell, storage medium and electronic device | |
| CN109560964B (en) | Equipment compliance checking method and device | |
| CN116225622A (en) | Docker-based PaaS application parameter template testing method | |
| CN116257404A (en) | Log analysis method and computing device | |
| CN111737090B (en) | Log simulation method and device, computer equipment and storage medium | |
| CN113778888A (en) | Code automatic test method, system, equipment and storage medium | |
| CN112559331A (en) | Test method and device | |
| CN117171800B (en) | Sensitive data identification method and device based on zero trust protection system | |
| CN111881128B (en) | Big data regression verification method and big data regression verification device | |
| CN114077447A (en) | Automatic and semi-automatic improvement of repair strategies for static analysis of violations |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |