CN112269728B

CN112269728B - System performance evaluation method, device, equipment and storage medium

Info

Publication number: CN112269728B
Application number: CN202011211405.1A
Authority: CN
Inventors: 王翠萍
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2020-11-03
Filing date: 2020-11-03
Publication date: 2023-08-04
Anticipated expiration: 2040-11-03
Also published as: CN112269728A

Abstract

The application discloses a system performance evaluation method, a device, equipment and a storage medium, and relates to the technical fields of system testing and information flow. The specific implementation scheme is as follows: determining a module operation time-consuming path of a system according to a test log obtained by the operation of a target firmware in the system; the method comprises the steps that the target firmware comprises current firmware and firmware to be updated, and the module running time-consuming path comprises a module current time-consuming path and a module time-consuming path to be updated; determining the node duty ratio of a module node hit of a system to a time-consuming path of the module operation; the node duty ratio comprises a current node duty ratio and a node duty ratio to be updated; and determining abnormal module nodes in the system after firmware upgrading according to the node duty ratio and the system performance test rule. The abnormal module node of the system performance deterioration after the firmware is upgraded can be more accurately estimated, and a new thought is provided for system performance estimation.

Description

System performance evaluation method, device, equipment and storage medium

Technical Field

The present application relates to the field of computer technology, and in particular, to the field of system testing and information flow technology. And more particularly, to a system performance evaluation method, apparatus, device, and storage medium.

Background

The speed index is one of core indexes of system performance evaluation, and in order to better ensure the stability of system performance after firmware updating, some large-scale distributed systems (particularly fan-out distributed systems) are connected to performance test evaluation on line to ensure the on-line quality after firmware updating of the system. Currently, common speed indicators include the total time spent by the system, 80-ary values, 99-ary values, and the like. Performance testing of a system generally analyzes system performance from a global system perspective, and for each module node in the system, workers involved in project development, testing and architecture are required to perform performance evaluation together according to codes, so that the cost is high, the accuracy is low, and improvement is needed.

Disclosure of Invention

The present disclosure provides a system performance evaluation method, apparatus, device, and storage medium.

According to a first aspect of the present disclosure, there is provided a system performance evaluation method, including:

determining a module operation time-consuming path of a system according to a test log obtained by the operation of a target firmware in the system; the method comprises the steps that the target firmware comprises current firmware and firmware to be updated, and the module running time-consuming path comprises a module current time-consuming path and a module time-consuming path to be updated;

Determining the node duty ratio of a module node hit of a system to a time-consuming path of the module operation; the node duty ratio comprises a current node duty ratio and a node duty ratio to be updated;

and determining abnormal module nodes in the system after firmware upgrading according to the node duty ratio and the system performance test rule.

According to a second aspect of the present disclosure, there is provided a system performance evaluation apparatus including:

the time-consuming path determining module is used for determining a time-consuming path of the module operation of the system according to a test log obtained by the operation of the target firmware in the system; the method comprises the steps that the target firmware comprises current firmware and firmware to be updated, and the module running time-consuming path comprises a module current time-consuming path and a module time-consuming path to be updated;

the node duty ratio determining module is used for determining the node duty ratio of a module node hit time-consuming path of the system; the node duty ratio comprises a current node duty ratio and a node duty ratio to be updated;

and the abnormal node determining module is used for determining abnormal module nodes in the system after the firmware is upgraded according to the node duty ratio and the system performance testing rule.

According to a third aspect of the present disclosure, there is provided an electronic device comprising:

At least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the system performance assessment method of any one of the embodiments of the present application.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions. The computer instructions are for causing a computer to perform the system performance evaluation method of any of the embodiments of the present application.

According to the method and the device for evaluating the system performance, the problems of high cost and low accuracy of each module node in the system according to the code manual evaluation are solved, the accuracy of determining abnormal module nodes in the system performance evaluation process is improved, and a new thought is provided for system performance evaluation.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The drawings are for better understanding of the present solution and do not constitute a limitation of the present application. Wherein:

Fig. 1 is a schematic structural diagram of an information flow system according to an embodiment of the present application;

FIG. 2A is a flow chart of a system performance evaluation method provided in accordance with an embodiment of the present application;

FIG. 2B is a schematic diagram of a system performance evaluation process provided in accordance with an embodiment of the present application;

fig. 2C is a schematic diagram of a module node topology of a system according to an embodiment of the present application.

FIG. 3 is a flow chart of another system performance evaluation method provided in accordance with an embodiment of the present application;

FIG. 4 is a flow chart of another system performance evaluation method provided in accordance with an embodiment of the present application;

FIG. 5A is a flow chart of another system performance evaluation method provided in accordance with an embodiment of the present application;

fig. 5B is a schematic diagram of another system performance evaluation process provided according to an embodiment of the present application.

FIG. 6 is a flow chart of another system performance evaluation method provided in accordance with an embodiment of the present application;

FIG. 7 is a schematic diagram of a system performance evaluation device according to an embodiment of the present application;

fig. 8 is a block diagram of an electronic device for implementing a system performance evaluation method of an embodiment of the present application.

Detailed Description

Exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Prior to describing embodiments of the present application, a description of the field Jing Jinhang of use of embodiments of the present application will be provided. When a worker develops a new version of the current firmware of the system (i.e., the firmware to be updated), performance evaluation needs to be performed on the updated firmware of the system through the current firmware of the system and the firmware offline to be updated. When performance evaluation is performed, system performance test needs to be performed first, namely, current firmware and firmware to be updated in a control system need to be run at least twice, and test logs in the running process of the firmware are obtained. The specific implementation process can be as follows: and simultaneously sending test requests to the current firmware and the firmware to be updated from an inlet module of the system, wherein each module node in the system can operate based on the current firmware and the firmware to be updated in the system, and in the firmware operation process, the performance information (such as the operation time consumption of the module node, the node connection relation, whether the module node operates normally or not) of each module node in the system is recorded to generate a test log. It should be noted that, in the performance test stage, multiple test requests may be sent to the current firmware and the firmware to be updated at the same time, and each time the firmware in the system runs, the performance information of the corresponding module node is obtained, and a test log is generated. Alternatively, a test log may be generated for each operation, or a test log may be generated for each of the current firmware and the firmware to be updated, or a total test log may be generated for each of the current firmware and the firmware to be updated, etc. For example, assuming that the information flow system shown in fig. 1 is tested, it may be that the current firmware and the firmware to be updated send multiple bilateral requests at the same time, and the current firmware in the system controls each module to interact to complete operations such as resource acquisition, processing, and sequencing based on each bilateral request. In the running process of each module in the system, performance information of each module (such as a fusion module, a resource aggregation module, a issuing history module, a queue 1 module and the like) in the system can be obtained to generate a test log. Based on the above, the embodiments of the present application execute the two later stages of performance evaluation, namely, performance analysis and node interception stages, according to the test log obtained by the target firmware operation in the system.

FIG. 2A is a flow chart of a system performance evaluation method provided in accordance with an embodiment of the present application; FIG. 2B is a schematic diagram of a system performance evaluation process provided in accordance with an embodiment of the present application; fig. 2C is a schematic diagram of a module node topology of a system according to an embodiment of the present application. The embodiment is suitable for evaluating the system performance after the firmware update according to the test log obtained by running the current firmware and the firmware to be updated in the system, and is particularly suitable for carrying out bilateral test on the system based on the current firmware and the firmware to be updated in an off-line manner and analyzing the module node deterioration condition after the system firmware update according to the obtained test log. The embodiment may be performed by a system performance evaluation device configured in an electronic device, which may be implemented in software and/or hardware. As shown in fig. 2A-2C, the method includes:

s201, determining a module operation time-consuming path of the system according to a test log obtained by the operation of the target firmware in the system.

The target firmware of the embodiment of the present application may be an execution program code written for a system (i.e., each module node in the system) based on a system running environment (such as a c++ running environment). Optionally, the target firmware in the embodiment of the present application includes: current firmware and firmware to be updated. The current firmware can be the firmware which is being used before the system is subjected to firmware upgrading; the firmware to be updated may be firmware that the system needs to update on the basis of the current firmware. The test log may be a log file used for recording performance information (such as time consuming running of the module node, node connection relationship, and whether to run normally) of each module node in the system during the running process of the target firmware. The module running time-consuming path of the system can be determined according to the node topological relation among the module nodes in the system and the running time consumption of the module nodes. And determining a module operation time-consuming path corresponding to the current operation every time the target firmware is operated. The module operation time-consuming path in the embodiment of the application comprises a current time-consuming path of the module and a time-consuming path to be updated of the module, wherein the current time-consuming path of the module is determined according to a test log obtained by current firmware operation; the time-consuming path of the module to be updated can be determined according to a test log obtained by running the firmware to be updated. The specific determination process of the two is consistent, but the test log according to the specific determination process is obtained by running two different firmware.

Optionally, as shown in fig. 2B, the system performance evaluation process of the embodiment of the present application includes three parts, i.e., performance test, performance analysis, and node interception. Alternatively, the operation of the performance test portion (i.e., the test log obtained by running the target firmware in the system) has been described, and the embodiment of the present application will mainly be described in the performance analysis and node interception portion in fig. 2B. Specifically, after a test log is obtained, a performance analysis stage is entered, and a time-consuming path of the operation of the module is determined according to the test log obtained by the operation of the target firmware. Since the test log includes the test log of the current firmware and the test log of the firmware to be updated, and the test log of each firmware is obtained by running the firmware at least once. Therefore, the current time-consuming path of the corresponding module is determined according to the test log obtained by the current firmware operation at least once in the system; and determining a time-consuming path to be updated of the module corresponding to each operation according to the test log obtained by at least one operation of the module to be updated in the system. For example, assuming that the test log is generated by running the current firmware and the firmware to be updated 10 times each, at this time, a current time-consuming path of at least 10 modules and a time-consuming path of at least 10 modules to be updated may be determined according to the test log. It should be noted that, the number of the determined time-consuming paths of running modules may be higher than the number of times of running firmware corresponding to the test log, because sometimes the firmware is run once, the system may have multiple time-consuming paths of running modules that are the same time-consuming and are the highest time-consuming.

Specifically, in the embodiment of the present application, according to a test log obtained by running any target firmware (i.e. the current firmware or the firmware to be updated) once, the process of determining the module running time-consuming path of the system may be: according to the test log, analyzing the running time of each module node of the system in the running process of the running firmware, calculating the total time of each module candidate running path of the system, and taking at least one module candidate running path with the highest total time consumption as the module running time-consuming path of the system in the running process of the running firmware. Alternatively, the node relation among the modules in the test log and the time consumption data of each module node may be transmitted to a topology sequencing interface, where the topology sequencing interface may calculate the time consumption path of the module operation corresponding to the test log. For example, fig. 2C shows a node topology relationship of the system, that is, the system includes 5 candidate running paths (i.e., paths 1 to 5), and further shows that when any one target firmware runs once, the corresponding running time of each module (i.e., module 1 to module 16) of the system is calculated, at this time, the total time of the 5 candidate running paths is analyzed, and the path 3 with the longest total time is taken as the current running time path of the module.

Optionally, in the embodiment of the present application, if each target firmware is run multiple times to generate a test log, or two target firmware is run multiple times to generate a total test log, the test log is larger at this time, and the time-consuming path of the module running is directly determined according to the test log, which consumes longer time. In this case, the embodiment of the present application may perform preprocessing on the test log including multiple running results, for example, performing hash processing and aggregation processing as shown in fig. 2B, so as to split the test log into a plurality of sub-logs, for example, may be that each firmware runs for a preset number of times (for example, once) corresponds to one sub-log. And simultaneously analyzing the sub-logs to determine a module operation time-consuming path of the system so as to improve the determination efficiency of the module operation time-consuming path.

S202, determining the node duty ratio of a time-consuming path of a module node hit module operation of the system.

The node duty ratio is a speed index specially set for system performance evaluation in the embodiment of the application, and the speed index can represent the long tail condition of the system. Specifically, in the multiple running processes of each target firmware in the system, the number of times that the module node of the system hits (i.e. locates) the time-consuming path of the module corresponding to the target firmware accounts for the ratio of the total running times of the target firmware (or the total number of time-consuming paths of the module of the target firmware). Because the module operation time-consuming path in the embodiment of the present application includes the current time-consuming path of the module and the time-consuming path to be updated of the module, correspondingly, the node duty cycle in the embodiment of the present application also includes the current node duty cycle and the node duty cycle to be updated.

Specifically, in the embodiment of the present application, when determining the node ratio of the running time-consuming path of the module node hit module of the system, the current node ratio of the current time-consuming path of the module node hit module (i.e. the number of times of the current time-consuming path of the hit module divided by the current firmware running time or the total number of the current time-consuming paths of the module) and the node ratio to be updated of the time-consuming path to be updated of the module node hit module (i.e. the number of times of the time-consuming path to be updated of the hit module divided by the firmware running time to be updated or the total number of the time-consuming paths to be updated of the module) may be determined for each node of the system.

For example, regarding the module node topology described in fig. 2C, assume that 100 test logs are run according to the current firmware, and the number of times each module candidate running path is determined as the current time-consuming path of the module is shown in table 1 below. And the current node ratio=the number of times of hit module current time-consuming paths/the current firmware running number, at this time, according to this table 1, it can be determined that the node ratio of module 1, module 15 and module 16 in the system is (20+10+50+10)/100=100%; the node ratio of the module 2 is (20+10)/100=30%; the node ratios of the modules 3, 7 and 12 are 50/100=50%, and the node ratios of other modules are calculated in a similar manner, and are not described in detail herein.

TABLE 1 relationship table of module candidate run paths and current time-consuming paths of hit modules thereof

It should be noted that, in the embodiment of the present application, the operations of performing index calculation by the performance analysis portion shown in fig. 2B according to the test log are completed through S201 and S202, and in the embodiment of the present application, when calculating the index, in addition to calculating the node duty ratio of the module node according to the above manner, the index of other dimensions, such as flat response (i.e. average response time), 80 split values, virtual node time consumption, 99 split values, and small flow node time consumption, may also be calculated. The embodiments of the present application are not limited thereto.

S203, determining abnormal module nodes in the system after firmware upgrade according to the node duty ratio and the system performance test rule.

The system performance test rule may be a pre-specified metric for determining whether the module node becomes an abnormal module node after upgrading from the current firmware to the firmware to be updated based on an index variation of the module node running under the current firmware and the firmware to be updated. The abnormal module node may be a node with serious system performance degradation after the system is upgraded from the current firmware to the firmware to be updated.

Optionally, a corresponding threshold range of acceptable index variation may be set for the node duty ratio index in the system performance test rule. And S202, after determining the current node duty ratio and the node duty ratio to be updated of each module node in the system, calculating the node duty ratio variation of each module node, comparing the node duty ratio variation with a corresponding index variation threshold value of the node duty ratio set in a system performance rule, and if the node duty ratio variation is not in the threshold variation range, indicating that the performance of the module node is seriously deteriorated, and taking the module node as an abnormal module node in the system after firmware upgrading. Optionally, in the embodiment of the present application, after determining an abnormal module node in the system after firmware upgrade, operations such as interception, redness and the like may be performed on the abnormal module node, so as to remind a staff to improve a program code corresponding to the abnormal module node in the firmware to be updated.

Optionally, in the performance analysis step, indexes of other dimensions of the module node, such as flat sound, 80 division into values, virtual node time consumption, 99 division of bit values, small flow node time consumption, and the like, may be calculated, so in order to improve accuracy of determining the abnormal module node, in the system performance test rule of the embodiment of the application, an index change threshold range corresponding to the indexes of other dimensions may be further included, and then, in combination with the change amount of the multidimensional index of the module node, abnormal module nodes in the system after firmware upgrading are accurately determined.

According to the technical scheme, a current time-consuming path of a module is determined according to a test log of current firmware in a system, and the current node ratio of the current time-consuming path of the module node hit module of the system is determined; determining a time-consuming path to be updated of a module according to a test log of firmware to be updated in the system, and determining a node duty ratio of the time-consuming path to be updated of a module node hit module of the system; and determining abnormal module nodes in the system after firmware upgrading according to the change quantity of the duty ratio of the two nodes and the system performance test rule. According to the embodiment of the application, the speed index capable of effectively intercepting the long tail deterioration of the system, namely the node duty ratio, is designed for the module nodes in the system, and based on the node duty ratio, the performance analysis of each node in the system can be automatically carried out in the performance evaluation process, the abnormal module nodes with serious long tail deterioration in the system can be accurately found, the accuracy of determining the abnormal module nodes in the system performance evaluation process is improved, and a new thought is provided for the system performance evaluation.

Fig. 3 is a flowchart of another system performance evaluation method according to an embodiment of the present application, where a specific description of determining abnormal module nodes in a system after firmware upgrade according to a node duty ratio and a system performance test rule is given based on the above embodiment, as shown in fig. 3, and the method includes:

s301, determining a module operation time-consuming path of the system according to a test log obtained by the operation of the target firmware in the system.

The target firmware comprises current firmware and firmware to be updated, and the module running time-consuming path comprises a module current time-consuming path and a module time-consuming path to be updated.

S302, determining the node duty ratio of a time-consuming path of a module node hit module operation of the system.

The node duty ratio comprises a current node duty ratio and a node duty ratio to be updated.

S303, grading the module nodes of the system according to the node duty ratio to obtain grade change information of the module nodes after the current firmware is upgraded to the firmware to be updated.

Optionally, in the embodiment of the present application, after determining the node duty ratio (that is, the current node duty ratio and the node duty ratio to be updated) of each module node in the system, performing primary grading on each module node according to the current node duty ratio of each module node to obtain the current grade information of each module node; and carrying out secondary level division on each module node according to the duty ratio of the node to be updated of each module node to obtain the level information to be updated of each module node. And then, the grade change quantity from the current grade information to the grade information to be updated is used as grade change information of each module node after each module node is updated from the current firmware to the firmware to be updated.

Specifically, in this embodiment of the present application, there are many ways of classifying the module nodes of the system according to each node duty ratio (i.e., the current node duty ratio or the node duty ratio to be updated), which is not limited in this embodiment of the present application, the corresponding node duty ratio ranges may be set for different levels, for example, the first level node duty ratio range is 71% -100%, the second level node duty ratio range is 31% -80%, and the third level node duty ratio range is 0% -30%. And judging which level the node duty ratio of each module node is in the corresponding node duty ratio range, and dividing the module node into which level. The module nodes can be classified by combining the node duty ratio with other data (such as node topological relation of the system, overall time-consuming condition of the system, etc.), and the specific implementation will be described in detail in the following embodiments.

S304, determining abnormal module nodes in the system after firmware upgrading according to the level change information and the system performance test rule.

Optionally, in the embodiment of the present application, there may be many ways to determine the abnormal module node existing in the system after the firmware is upgraded according to the level change information of the module node and the system performance test rule, which is not limited to this embodiment of the present application. The level change information may be used as a new performance evaluation index, and a corresponding interception rule is set in the system performance test rule for the performance evaluation index, for example, a module node with an increased node level is used as an abnormal module node, or a module node with an increased node level (such as a first level) is used as an abnormal module node, etc. And then according to the interception rule set for the level change information in the system performance test rule, determining whether each module node is an abnormal module node existing after the system firmware is upgraded by combining the level change information of each module node. In the system performance test rule, different interception rules are set for different grades, and according to the interception rules corresponding to different grades, grade change information of each module node is combined, for example, whether the module and the current grade change of the module node can cause the module node to become an abnormal module node or not is determined by using the interception rules corresponding to the grades before or after the module grade change.

According to the technical scheme, after determining a module operation time-consuming path (namely, a module current time-consuming path and a module time-consuming path to be updated) of a system according to test logs of target firmware (namely, current firmware and firmware to be updated) in the system, determining node duty ratios (namely, current node duty ratio and node duty ratio to be updated) of each module node hit module operation time-consuming path, grading according to the node duty ratios, obtaining grade change information of the module nodes, and determining abnormal module nodes in the system after firmware upgrading by combining with a system performance test rule. After the node proportion of the module node is determined, the module node is classified and then the abnormal module node is determined, so that the change condition of the module node before and after firmware updating is more accurately indicated, and the accuracy of determining the abnormal module node in the system performance evaluation process is further improved.

Fig. 4 is a flowchart of another system performance evaluation method according to an embodiment of the present application, where, based on the foregoing embodiment, a specific description is given of grading a module node of a system according to a node duty ratio to obtain grading change information of the module node after the module node is upgraded from the current firmware to the firmware to be updated, as shown in fig. 4, where the method includes:

S401, determining a module operation time-consuming path of the system according to a test log obtained by the operation of the target firmware in the system.

S402, determining the node duty ratio of a time-consuming path of a module node hit module operation of the system.

S403, determining a target path with highest node duty ratio weight in the system according to the node duty ratio, and grading the module nodes of the system according to the node duty ratio, the target path and the node topological relation of the system, so as to determine the grade information of the module nodes.

The target path in the embodiment of the present application is determined according to the node duty ratio of each module node in the system, and since the node duty ratio of the module node includes the current node duty ratio and the node duty ratio to be updated, the target path determined in this step also includes the current target path and the target path to be updated correspondingly. And the grade information of the module nodes after the node duty ratio and the target path are divided also comprises current grade information and grade information to be updated.

Optionally, in the embodiment of the present application, according to the node duty ratio, the process of determining the target path with the highest node duty ratio weight in the system includes: summing the current node duty ratio of each module node contained in each module candidate running path in the system aiming at each module candidate running path in the system to obtain the current node duty ratio weight of the module candidate running path; and summing the duty ratio of the nodes to be updated of each module node contained in the module candidate running path to obtain the duty ratio weight of the nodes to be updated of the module candidate running path. Further selecting a module candidate running path with highest current node weight as a current target path of the system; and selecting the module candidate running path with the highest node weight to be updated as a target path to be updated of the system.

Optionally, in this embodiment of the present application, when grading a module node of the system according to a node duty ratio, a target path, and a node topology relationship of the system, the module node of the system may be graded by combining one or more of a size of the node duty ratio of the module node, a position in the node topology relationship of the system, and whether the module node hits the target path. Optionally, the specific dividing mode includes:

(1) And taking the module node (namely, a class of long tail nodes) with the node ratio of which is larger than or equal to the first ratio threshold value in the serial module nodes of the system and the module node (namely, a class of long tail nodes) which is positioned on the target path in the concurrent module nodes of the system as the module nodes of the first level. The serial module node of the system is a node with an outbound degree and an inbound degree of 1 in the system, such as a module 15 in fig. 2C. If the node ratio of the serial node (module 15 in fig. 2C) in the system is greater than or equal to the first ratio threshold (e.g., 90%), it is explained that the time-consuming increase of such module node necessarily brings about the time-consuming increase of the whole system, so the class of module node is classified as the first class (i.e., high class); the concurrent modules in the system may be nodes in the system that are triggered to run in parallel, such as modules 2-14 in fig. 2C. If concurrent module nodes in the system are located on the target path, the time consumption rise of the module nodes inevitably affects the overall long tail time consumption of the system, so the grades of the module nodes are also classified into the first grade. For example, if the target path in fig. 2C is path 3, the ranks of modules 3, 7, and 12 are the first rank.

(2) Taking the module node (namely, the non-long tail node) with the node ratio of the module node of the non-first level of the system being larger than or equal to the second proportion threshold value as the module node of the second level; wherein the first proportional threshold is greater than the second proportional threshold. After determining the node of the first level, the embodiments of the present application determine other module nodes except the first level, and if the node ratio of the other module nodes is greater than or equal to the second ratio threshold (5%), the time consumption of such node increases without affecting the total time consumption of the system too much, so the level of such module nodes can be divided into the second level (i.e. the middle level).

(3) And taking the module node (namely the low time consumption node) with the node duty ratio smaller than the second proportion threshold value in the system as a third-level node (namely the low level), wherein the time consumption of the module node has no influence on the total time consumption of the system.

It should be noted that, in the embodiment of the present application, according to the current node duty ratio of the module nodes, the node topology relationship of the system, and the current target path, the module nodes in the system are classified according to the above rules, so as to obtain the current class information of each module node; and grading the module nodes in the system according to the node occupation ratio to be updated of the module nodes, the node topological relation of the system and the target path to be updated according to the rules, and obtaining the grade information to be updated of the module nodes.

S404, determining the grade change information of the module node after the current firmware is upgraded to the firmware to be updated according to the current grade information and the grade information to be updated.

Optionally, after determining the current level information and the level information to be updated of each module node in the system, the change condition between the current level information and the level information to be updated is used as the level change information of the module node, for example, if the current level information of a certain module node is the second level and the level information to be updated is the first level, after the module is upgraded from the current firmware to the firmware to be updated, the level change information is upgraded from the second level to the first level, and the level is increased by one level.

S405, determining abnormal module nodes in the system after firmware upgrade according to the level change information and the system performance test rule.

According to the technical scheme, after determining a time-consuming path (namely a current time-consuming path of a module and a time-consuming path of the module to be updated) of the module operation of the system according to a test log of target firmware (namely the current firmware and the firmware to be updated) in the system, determining a node duty ratio (namely the current node duty ratio and the node duty ratio to be updated) of the time-consuming path of the module operation of each module node hit module, classifying the module nodes of the system according to the node duty ratio, the target path determined based on the node duty ratio and the node topological relation of the system to obtain the level change information of the module nodes, and determining abnormal module nodes in the system after firmware upgrading by combining with a system performance test rule. When the module nodes of the system are divided, the method combines the size of the node ratio of the module nodes, the positions in the node topological relation of the system and the multidimensional data such as the hit target path, so that the grading result is more accurate, the influence degree of the graded module nodes on the total time consumption of the system is more accurately represented by the graded module nodes, and the guarantee is provided for the follow-up accurate evaluation of the abnormal module.

FIG. 5A is a flow chart of another system performance evaluation method provided in accordance with an embodiment of the present application; fig. 5B is a schematic diagram of another system performance evaluation process according to an embodiment of the present application, where specific descriptions of determining abnormal module nodes in a system after firmware upgrade according to level change information and a system performance test rule are given based on the above embodiment, and as shown in fig. 5A-5B, the method includes:

s501, determining a module operation time-consuming path of the system according to a test log obtained by the operation of the target firmware in the system.

S502, determining the node duty ratio of a time-consuming path of a module node hit module operation of the system.

Wherein the node duty ratio comprises a current node duty ratio and a node duty ratio to be updated

And S503, grading the module nodes of the system according to the node duty ratio to obtain grade change information of the module nodes after the current firmware is upgraded to the firmware to be updated.

S504, determining abnormal module nodes in the system after firmware upgrading according to the level change information and the system performance test rule.

Optionally, as shown in fig. 5B, in the node interception stage, the embodiment of the present application sets interception rules and exemption rules for the system performance test rules. Specifically, different rules may be set according to different conditions of the level information change, for example, interception rules are set for the case that the level is raised and the level is kept unchanged at a high level (i.e., the first level), exemption rules are set for the case that the level is kept unchanged at a medium level (i.e., the second level), and no rules are set for the case that the level is kept unchanged at a low level because the time consumption of the module node at the low level hardly affects the total time consumption of the system. After the index variation of the module node is obtained in the performance analysis stage, judging the threshold value corresponding to the interception rule and the exemption rule, and further determining the abnormal module node in the system after firmware upgrading based on the interception rule and the exemption rule.

Specifically, according to the level change information and the system performance test rule, the abnormal module node in the system after firmware upgrade is determined, which comprises at least one of the following conditions:

interception rule (1): and taking the level change information in the system as a module node of which the second level is increased to the first level (namely, the level is increased from the middle level to the high level), and taking the module node as an abnormal interception module node of the system in the running environment to be updated.

The interception rule (2) takes the level change information in the system as a module node which is continuously in a first level (namely continuously in a high level) and the candidate index variable exceeds the interception threshold range as an abnormal interception module node of the system in the running environment to be updated; among the candidate metrics in the embodiments of the present application include, but are not limited to: node duty cycle, flat response, 80 split value, virtual node time consumption, 99 split value, small flow node time consumption, etc.

And the exemption rule (3) changes the medium level change information in the system into a continuous second level (namely a continuous medium level), and the module nodes with candidate index variables within the range of the exemption threshold value are used as abnormal exemption module nodes of the system in the running environment to be updated.

Illustratively, table 2 shows an interception rule table of an embodiment of the present application; table 3 shows an exemption rule table of an embodiment of the present application.

Table 2 intercept rules table

Specifically, for the interception rule, the overall system angle analysis may be performed, for example, the system exit total time consumption flat response (or 80-bit value) index in table 2 is analyzed, whether the variation of the total time consumption flat response (or 80-bit value) when the system runs under the current firmware and the firmware to be updated is between [ -5.0,2.5] is judged, if not, the interception rule is satisfied, and the corresponding interception reason in table 2, namely, the total time consumption of the system is displayed. And then analyzing the angle of each module node in the system, specifically aiming at the interception rule (1), judging whether the grade change information of each module node of the system is upgraded from the second grade to the first grade (such as a newly added long tail node and a non-long tail node to the long tail node in the table 2), if so, taking the module nodes as abnormal interception module nodes, and displaying the corresponding interception reasons in the table 2, namely the newly added long tail node or the non-long tail to the long tail. When the module node needs to be described, both the two cases indicate that the module node has a long tail degradation trend, so that the module node needs to be intercepted. For the interception rule (2), judging that the level change information of each module node of the system is a continuous first level, namely, a class long-tail node or a class long-tail node before and after the change, if so, calculating candidate index variables (such as a class long-tail node flat response/80 minutes, a class long-tail node duty ratio and a class long-tail node flat response/80 minutes) of the class long-tail node in table 2 for the module nodes, if the candidate index variables of the class module nodes exceed a threshold range set in an interception mode in the interception rule (such as a class long-tail node flat response or a class 80 minute value change amount exceeds [ -5.0,2.5 ]) in table 2), taking the candidate index variables as abnormal interception modules, and displaying corresponding interception reasons in table 2.

TABLE 3 exemption rule form

Specifically, the exemption rule (3) is mainly a rule set for a module node (i.e., a non-long tail node) of the second level. The method may be that whether the level change information of each module node of the system is a continuous second level, that is, the node with non-long tail before and after the change is a non-long tail node, if yes, calculating a candidate index scalar (such as the time-consuming change amount of the node with non-long tail in table 3) for the node with non-long tail, if the candidate index change amount meets the threshold range set by the interception mode in the exemption rule (such as the threshold range [ -5.0,3] in the interception mode in table 3), the node with non-long tail is an abnormal exemption node, and displaying the corresponding exemption reason in table 3, that is, the influence of the non-long tail on the whole system is smaller. Optionally, the exemption rule in the embodiment of the present application may further analyze a node time consumption index under a small flow, if a node time consumption variation range under a small flow is within a threshold range [ -5.0,3] range under an interception mode, the small flow node is used as an abnormal exemption node, and a corresponding exemption reason in table 3 is displayed, that is, there are fewer small flow data samples and larger fluctuation.

According to the technical scheme, after determining a time-consuming path (namely a current time-consuming path of a module and a time-consuming path of the module to be updated) of the module operation of the system according to test logs of target firmware (namely the current firmware and the firmware to be updated) in the system, determining a node ratio (namely the current node ratio and the node ratio to be updated) of the time-consuming path of the module operation hit module of each module node, grading according to the node ratio, obtaining grade change information of the module nodes, and then judging abnormal module nodes by the module nodes subjected to the grade grading by combining preset interception rules and exemption rules. According to the embodiment of the application, aiming at the level change information of the module, two different interception rules and exemption rules are set, after firmware upgrading can be accurately analyzed from the angle of the multidimensional index, whether the module node of the system is deteriorated to be an abnormal module node or not is judged, and the accuracy is higher.

FIG. 6 is a flowchart of another system performance evaluation method according to an embodiment of the present application, where, based on the foregoing embodiment, a specific description is given of determining a module running time-consuming path of a system according to a test log obtained by running target firmware in the system, as shown in FIG. 6, and the method includes:

s601, extracting node topological relation of the system and time-consuming operation of the module nodes from a test log obtained by operation of target firmware in the system.

The test log in the embodiment of the present application may be a json structure, which at least includes: the name of each module node in the system, the name of the parent node module (or child node module) of the module node, and the running of the module node is time-consuming; optionally, a small flow experiment number for each request hit, etc. may also be included. Specifically, the test log may include two parameters, namely, a time-consuming parameter (cost) and a topology map parameter (graph), where a key of the cost parameter is a current node, and a value is a time-consuming, and a name and a time-consuming of each module node are recorded therein. Taking fig. 2C as an example, the cost parameters in the test log corresponding to fig. 2C are as follows: cost: { 'Module 1':5, 'Module 2':22, 'Module 3':28, … …, 'Module 16':1}. The graph parameter stores father-son relationship, the key is father node, the value is child node, and the relationship between each module node and father node or child node is recorded; the graph parameters in the test log corresponding to fig. 2C are as follows: graph: { ' Module 1' [ ' Module 2, module 3, module 4' ], module 2' [ ' Module 5, module 6' ], … …, ' Module 15' [ ' Module 16' ] }.

Optionally, the test log obtained by running the target firmware in the system contains time-consuming parameters and topology map parameters. Therefore, the embodiment of the application can extract the node topological relation of the system and the operation time consumption of the module nodes according to the test log. Specifically, the connection relationship between each module and its parent node in the system may be extracted from the topological graph parameters (graph) in the test log, and then, according to the extracted connection relationship, the node topological relationship corresponding to the system is constructed when the firmware runs each time (the constructed node topological relationship is shown in fig. 2C). And extracting the time consuming parameters (cost) in the test log from the time consuming parameters (cost) of each operation of the firmware in the system, wherein the operation of each module node in the system is time consuming. Alternatively, the running time of the module node may be correspondingly recorded in the node topology relationship, as shown in fig. 2C.

S602, determining at least two module candidate running paths of the system according to the node topological relation.

Optionally, after obtaining the node topological relation of the system, all executable paths of the system can be analyzed according to the node topological relation to be used as module candidate running paths of the system. Taking the node topological relation shown in fig. 2C as an example, the system has 5 module candidate running paths, namely, a path 1 corresponding to a module 1-a module 2-a module 5-a module 10-a module 15-a module 16; module 1-module 2-module 6-module 11-module 15-module 16; module 1-module 3-module 7-module 12-module 15-module 16; the paths 4 corresponding to the modules 1-4-8-13-15-16; module 1-module 4-module 9-module 14-module 15-module 16 corresponds to path 5.

S603, determining a module operation time-consuming path of the system from at least two module candidate operation paths according to the operation time-consuming of the module node.

Optionally, after determining a plurality of module candidate operation paths of the system, the embodiments of the present application determine total time consumption of all module nodes included in each module candidate operation path by combining operation time consumption of the module nodes extracted from the test log in S601, and use a module candidate operation path with the longest total time consumption in the plurality of module candidate operation paths as a module operation time consumption path of the system. For example, take the 5 candidate running paths of the module shown in fig. 2C as an example, where the path with the longest time consumption of the module node is path 3, so path 3 is taken as the running time-consuming path of the module of the system.

S604, determining the node duty ratio of the time-consuming path of the module node hit module operation of the system.

The node duty ratio comprises a current node duty ratio and a node duty ratio to be updated;

s605, determining abnormal module nodes in the system after firmware upgrading according to the node duty ratio and the system performance test rule.

According to the technical scheme, node topological relations and module node operation time consumption are extracted from test logs obtained by target firmware operation in a system, a plurality of module candidate operation paths of the system are determined according to the node topological relations, then the operation time consumption of each module node contained in the module candidate operation paths is combined, the module candidate operation path with the longest total operation time consumption is selected as a module operation time consumption path, and then the node ratio of a module node hit module to-be-updated time consumption path of the system is determined; and determining abnormal module nodes in the system after firmware upgrading according to the two-node duty ratio and the system performance test rule. According to the scheme of the embodiment of the application, for a system with a complex structure, the time-consuming path of the module operation can be determined by means of the node topological relation, the accuracy and the high efficiency of determining the time-consuming path of the module operation are improved, in addition, the network topological relation of the nodes of the network can be automatically extracted through the test log, and for the system with the complex module connection relation, the node does not need to be analyzed manually in advance, the cost is low, and the accuracy is high.

Fig. 7 is a schematic structural diagram of a system performance evaluation device according to an embodiment of the present application. The embodiment is suitable for evaluating the system performance after the firmware update according to the test log obtained by running the current firmware and the firmware to be updated in the system, and is particularly suitable for carrying out bilateral test on the system based on the current firmware and the firmware to be updated in an off-line manner and analyzing the module node deterioration condition after the system firmware update according to the obtained test log. The device can realize the system performance evaluation method of any embodiment of the application. The apparatus 700 specifically includes the following:

a time-consuming path determining module 701, configured to determine a module running time-consuming path of a system according to a test log obtained by running a target firmware in the system; the method comprises the steps that the target firmware comprises current firmware and firmware to be updated, and the module running time-consuming path comprises a module current time-consuming path and a module time-consuming path to be updated;

a node duty ratio determining module 702, configured to determine a node duty ratio of a module node of the system to hit a time-consuming path for running the module; the node duty ratio comprises a current node duty ratio and a node duty ratio to be updated;

And the abnormal node determining module 703 is configured to determine an abnormal module node in the system after the firmware upgrade according to the node duty ratio and the system performance test rule.

Further, the abnormal node determining module 703 includes:

the grading unit is used for grading the module nodes of the system according to the node duty ratio to obtain grade change information of the module nodes after the current firmware is upgraded to the firmware to be updated;

and the abnormal node determining unit is used for determining abnormal module nodes in the system after the firmware is upgraded according to the grade change information and the system performance testing rule.

Further, the grading unit includes:

a target path determining subunit, configured to determine, according to the node duty ratio, a target path with a highest node duty ratio weight in the system;

the grading subunit is used for grading the module nodes of the system according to the node duty ratio, the target path and the node topological relation of the system, and determining the grade information of the module nodes; the target path comprises a current target path and a target path to be updated; the grade information of the module node comprises current grade information and grade information to be updated;

and the change information determining subunit is used for determining the grade change information of the module node after the current firmware is upgraded to the firmware to be updated according to the current grade information and the grade information to be updated.

Further, the grading subunit is specifically configured to:

taking a module node with the node ratio larger than or equal to a first ratio threshold value in serial module nodes of the system and a module node positioned on the target path in concurrent module nodes of the system as a module node of a first level;

taking the module nodes with the node duty ratio larger than or equal to the second proportion threshold value in the module nodes of the system which are not in the first level as the module nodes of the second level; wherein the first proportional threshold is greater than the second proportional threshold.

Further, the abnormal node determining unit is specifically configured to perform at least one of the following:

the method comprises the steps that the grade change information in a system is taken as a module node of which the second grade rises by a first grade and is taken as an abnormal interception module node of the system in an operation environment to be updated;

the method comprises the steps that the medium level change information in a system is a module node which is continuously in a first level and has candidate index variables exceeding an interception threshold range, and the module node is used as an abnormal interception module node of the system in an operation environment to be updated;

and taking the medium level change information in the system as a module node with a continuous second level and candidate index variables within an exemption threshold range as an abnormal exemption module node of the system in the running environment to be updated.

Further, the time-consuming path determining module 701 includes:

the information extraction unit is used for extracting the node topological relation of the system and the time consumption of the operation of the module nodes from a test log obtained by the operation of the target firmware in the system;

the candidate path determining unit is used for determining at least two module candidate running paths of the system according to the node topological relation;

and the time-consuming path determining unit is used for determining a module operation time-consuming path of the system from the at least two module candidate operation paths according to the operation time consumption of the module node.

According to embodiments of the present application, an electronic device and a readable storage medium are also provided.

As shown in fig. 8, a block diagram of an electronic device according to a system performance evaluation method according to an embodiment of the present application is shown. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the application described and/or claimed herein.

As shown in fig. 8, the electronic device includes: one or more processors 801, memory 802, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the electronic device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In other embodiments, multiple processors and/or multiple buses may be used, if desired, along with multiple memories and multiple memories. Also, multiple electronic devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). One processor 801 is illustrated in fig. 8.

Memory 802 is a non-transitory computer-readable storage medium provided herein. Wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the system performance evaluation methods provided herein. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to perform the system performance evaluation method provided by the present application.

The memory 802, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions/modules (e.g., the time-consuming path determination module 701, the node duty determination module 702, and the abnormal node determination module 703 shown in fig. 7) corresponding to the system performance evaluation method in the embodiments of the present application. The processor 801 executes various functional applications of the server and data processing, i.e., implements the system performance evaluation method in the above-described method embodiments, by running non-transitory software programs, instructions, and modules stored in the memory 802.

Memory 802 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created according to the use of the electronic device of the system performance evaluation method, and the like. In addition, memory 802 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 802 may optionally include memory located remotely from processor 801, which may be connected to the electronics of the system performance evaluation method via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The electronic device of the system performance evaluation method may further include: an input device 803 and an output device 804. The processor 801, memory 802, input devices 803, and output devices 804 may be connected by a bus or other means, for example in fig. 8.

The input device 803 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic device of the system performance evaluation method, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointer stick, one or more mouse buttons, a track ball, a joystick, etc. The output device 804 may include a display apparatus, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device may be a touch screen.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASIC (application specific integrated circuit), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

These computing programs (also referred to as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.

According to the technical scheme of the embodiment of the application, the current time-consuming path of the module is determined according to the test log of the current firmware in the system, and the current node ratio of the current time-consuming path of the module node hit module of the system is determined; determining a time-consuming path to be updated of a module according to a test log of firmware to be updated in the system, and determining a node duty ratio of the time-consuming path to be updated of a module node hit module of the system; and determining abnormal module nodes in the system after firmware upgrading according to the change quantity of the duty ratio of the two nodes and the system performance test rule. According to the embodiment of the application, the speed index capable of effectively intercepting the long tail deterioration of the system, namely the node duty ratio, is designed for the module nodes in the system, and based on the node duty ratio, the performance analysis of each node in the system can be automatically carried out in the performance evaluation process, the abnormal module nodes with serious long tail deterioration in the system can be accurately found, the accuracy of determining the abnormal module nodes in the system performance evaluation process is improved, and a new thought is provided for the system performance evaluation.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially, or in a different order, provided that the desired results of the technical solutions disclosed in the present application can be achieved, and are not limited herein.

The above embodiments do not limit the scope of the application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application are intended to be included within the scope of the present application.

Claims

1. A system performance evaluation method, comprising:

determining a module operation time-consuming path of a system according to a test log obtained by the operation of a target firmware in the system; the target firmware comprises current firmware and a new version of the current firmware, and the module running time-consuming path comprises a module current time-consuming path and a module time-consuming path to be updated;

determining the node duty ratio of a module node hit of a system to a time-consuming path of the module operation; the node duty ratio comprises a current node duty ratio and a node duty ratio to be updated; the node duty ratio is used for grading the module nodes of the system and determining a target path with highest node duty ratio weight in the system;

Taking the module nodes with the node duty ratio larger than or equal to the second proportion threshold value in the module nodes of the system which are not in the first level as the module nodes of the second level; wherein the first proportional threshold is greater than the second proportional threshold;

according to the node duty ratio and the system performance test rule, determining abnormal module nodes in the system after firmware upgrade, wherein the abnormal module nodes comprise at least one of the following components:

2. The method of claim 1, wherein determining abnormal module nodes in the system after firmware upgrade according to the node duty cycle and system performance test rules comprises:

Grading the module nodes of the system according to the node duty ratio to obtain grade change information of the module nodes after the current firmware is upgraded to the new version of the current firmware;

and determining abnormal module nodes in the system after firmware upgrading according to the grade change information and the system performance test rule.

3. The method of claim 2, wherein grading the module nodes of the system according to the node duty cycle to obtain the grade change information of the module nodes after the module nodes are upgraded from the current firmware to the new version of the current firmware comprises:

determining a target path with highest node duty ratio weight in the system according to the node duty ratio, and grading module nodes of the system according to the node duty ratio, the target path and the node topological relation of the system to determine grade information of the module nodes; the target path comprises a current target path and a target path to be updated; the grade information of the module node comprises current grade information and grade information to be updated;

and determining the grade change information of the module node after the current firmware is upgraded to the new version of the current firmware according to the current grade information and the grade information to be updated.

4. The method of claim 1, wherein determining a module run time-consuming path of the system from a test log obtained from a target firmware run in the system comprises:

extracting node topological relation of the system and time-consuming operation of the module nodes from a test log obtained by operation of target firmware in the system;

determining at least two module candidate running paths of a system according to the node topological relation;

and determining a module operation time-consuming path of the system from the at least two module candidate operation paths according to the operation time-consuming of the module node.

5. A system performance evaluation device, comprising:

the time-consuming path determining module is used for determining a time-consuming path of the module operation of the system according to a test log obtained by the operation of the target firmware in the system; the target firmware comprises current firmware and a new version of the current firmware, and the module running time-consuming path comprises a module current time-consuming path and a module time-consuming path to be updated;

the node duty ratio determining module is used for determining the node duty ratio of a module node hit time-consuming path of the system; the node duty ratio comprises a current node duty ratio and a node duty ratio to be updated; the node duty ratio is used for grading the module nodes of the system and determining a target path with highest node duty ratio weight in the system;

An abnormal node determining module, configured to:

6. The apparatus of claim 5, wherein the abnormal node determination module comprises:

the grading unit is used for grading the module nodes of the system according to the node duty ratio to obtain grade change information of the module nodes after the current firmware is upgraded to the new version of the current firmware;

7. The apparatus of claim 6, wherein the ranking unit comprises:

And the change information determining subunit is used for determining the grade change information of the module node after the current firmware is upgraded to the new version of the current firmware according to the current grade information and the grade information to be updated.

8. The apparatus of claim 5, wherein the time-consuming path determination module comprises:

9. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the system performance assessment method of any one of claims 1-4.

10. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the system performance assessment method of any one of claims 1-4.