CN103761173A - Log based computer system fault diagnosis method and device - Google Patents

Abstract

The invention discloses a log management and fault diagnosis method and device under multiple hosts. The method comprises fault log collection, fault log analysis and fault log correlation analysis. Fault log collection comprises collecting fault logs of all hardware and software in the cluster and storing the logs in a log server uniformly. Fault log analysis comprises filtering the fault logs, extracting log template information and classifying the logs according to log types. Fault log correlation analysis comprises performing fault reason analysis by using log analysis results and combining time windows, clustering the related faults caused by the same fault into a category and trying to find the root of the type of the faults. The method and the device are capable of analyzing the system operation condition effectively, assisting administrators to process system faults and improving the fault type determination accuracy.

Description

A kind of computer system method for diagnosing faults and device based on daily record

Technical field

The invention belongs to the fault analysis field under computer system security, more specifically, relate to a kind of computer system method for diagnosing faults and device based on daily record.

Background technology

Along with the introducing of cloud computing structure, computer system becomes and becomes increasingly complex, and running software is thereon also more and more abundanter, and more frequent alternately, coupling is high, makes the reason of the system failure and kind also be difficult to analyze.In order to solve some crucial application harsh requirements to system processing power and fault-tolerant ability, therefore must monitor and record its daily behavior.So that after there is mistake, can find out the reason that fault occurs in time to diagnosing malfunction and location, thereby the performance of the mistake of repair system, raising system guarantees that similar fault can not occur again.

In a complete infosystem, log system is a very important ingredient.Daily record is the real picture of computer system running orbit, and it is widely used in system debug, monitoring and safety detection.Log management and analysis are the infrastructure of system management and intrusion detection, are evaluating system operation conditionss, the necessary means of supervising network security strategy validity.By system journal, not only can detect and analytic system fault, can also monitoring system situation.Optimization system performance, adjustment System behavior.

When daily record is analyzed, conventionally there are 3 main stages: (1) daily record is filtered, filter the daily record irrelevant with fault; (2) daily record fault analysis, from finding the type of fault in the middle of daily record, understands the wrong process occurring in detail; (3) log correlation analysis, the daily record in same fault source is assigned to same group, and analysis of failure is at the mechanism of transmission of different nodes and inter-module.

The existing diagnostic method based on daily record exists the problem of several respects:

(1) need a large amount of manual interventions.Because the fault diagnosis of the classification of daily record is all the center that is judged as with people, rely on merely the automatic processing of computing machine to be difficult to the result that obtains wanting.Therefore traditional fault diagnosis system tends to rely on expert system, need to be that it is by basic data collection by a large amount of artificial treatment, computing machine just can produce more satisfactory result through learning for a long time after correction, and this has wasted a large amount of system managers' time;

(2) response speed is slow.Current analytic system is the determinating mode based on a set of complexity normally, is difficult in real time the fault in system be responded;

(3) degree of accuracy is low.The accuracy of classifying for daily record is often lower, and lacks the algorithm of revising erroneous judgement.

Summary of the invention

For the defect of prior art, the object of the invention is to rapidly the daily record producing in computer system be classified, and only need less manual intervention, just can realize high-precision classification.In addition, can also revise at any time the result of classification, the new classificating knowledge of convenient study.

For achieving the above object, the present invention proposes a kind of computer system method for diagnosing faults based on daily record, comprise the following steps:

(S1) fault log analysis: the daily record in computer system is carried out to real-time analysis, utilize the failure modes result of the artificial study of fail close keyword matrix quantization, according to different fault types, every fault log is confirmed to fault type;

(S2) fault log is associated: utilizing the result of fault log analysis binding time window to carry out failure reason analysis, is a class the dependent failure log aggregation being caused by same fault, finds the root of this class fault.

The invention allows for a kind of computer system trouble-shooter based on daily record, comprise with lower module:

Fault log analysis module, for the daily record of computer system is carried out to real-time analysis, utilizes the failure modes result of the artificial study of fail close keyword matrix quantization, according to different fault types, every fault log is confirmed to fault type;

Fault log relating module, for utilizing result the binding time window of fault log analysis to carry out failure reason analysis, is a class the dependent failure log aggregation being caused by same fault, finds the root of this class fault.

Compared with prior art, system of the present invention has following beneficial effect:

(1) adopted fail close keyword matrix to store the result of machine learning, can easily to daily record, carry out failure modes rapidly, also can revise at any time judgment rule and add new fault type.Compare and improved processing speed with traditional approach, saved the time relearning;

(2) proposed new daily record sorting technique, can to daily record, according to different fault types, classify accurately;

(3) adopt improved time-based daily record correlation analysis, utilize the fault type of daily record to instruct correlation analysis, improved the accuracy of classification, reduced rate of false alarm and rate of failing to report;

(4) by fail close keyword matrix, preserve the time window of different faults type, can be real-time modify, has improved the learning ability to new fault type.

Accompanying drawing explanation

Fig. 1 is the structural representation of the system fault diagnosis device based on fault log for multi-host environment of the embodiment of the present invention;

Fig. 2 is the process flow diagram of the daily record Fault Classification of the embodiment of the present invention;

Fig. 3 is the process flow diagram of the time-based fault correlation analytical approach of the embodiment of the present invention.

Embodiment

In order to make object of the present invention, technical scheme and advantage clearer, below in conjunction with drawings and Examples, the present invention is further elaborated.Should be appreciated that specific embodiment described herein, only in order to explain the present invention, is not intended to limit the present invention.

As shown in Figure 1, the invention provides a kind of system fault diagnosis device that is applicable to large-scale cloud platform.This device can be collected the daily record producing in whole computer system, daily record is added up and is learnt, and mark the fault type of each daily record.When having fault to occur, keeper can click corresponding daily record, and system can find with this daily record and have associated daily record, according to the order of fault propagation, forms fault tree, helps keeper to find the basic reason of fault diagnosis fault.This device mainly comprises three assemblies:

1) fault log analysis module, carries out real-time analysis to the daily record in computer system, utilizes the failure modes result of the artificial study of fail close keyword matrix quantization, according to different fault types, every fault log is confirmed to fault type;

2) fault log relating module, utilizes the result of log analysis binding time window to carry out failure reason analysis, is a class, and attempts to find the root of this class fault the dependent failure log aggregation being caused by same fault.

Correspondingly, a kind of computer system method for diagnosing faults based on daily record of the embodiment of the present invention, comprises the following steps:

(S1) fault log analysis: the daily record in computer system is carried out to real-time analysis, utilize the failure modes result of the artificial study of fail close keyword matrix quantization, according to different fault types, every fault log is confirmed to fault type;

(S2) fault log is associated: utilizing the result of fault log analysis binding time window to carry out failure reason analysis, is a class the dependent failure log aggregation being caused by same fault, finds the root of this class fault.

As shown in Figure 2, the fault log analysis of the embodiment of the present invention comprises the following steps:

(1) daily record pre-service

Daily record pre-service has comprised two parts, comprises that the daily record to repeating is filtered and insignificant word in every daily record is filtered.Specifically comprise the following steps:

(1-1) daily record repeating producing within certain period in same process is filtered.

By finding the statistics of daily record, the time that this redundancy daily record produces all concentrates in together, and the process of generation should be all identical.So carrying out pretreated time, filtering the similar daily record that same process produces within certain period.The concrete time period is selected different threshold values according to different computer systems.Daily record and last daily record that certain equipment produces ought be detected just the same, and the time producing in the threshold values of stipulating, just remove this daily record.If occurred that in time threshold values different daily record or same log have exceeded time range, so just retained this daily record.

When concrete application, need to select a suitable threshold values to the time period that can merge, time span is too large else if, and the similar log information that different faults may be produced, to filtering out, causes and fails to report.In the present embodiment, because the filtration is here just as pre-service, for guaranteeing not fail to report, in judgement, can only the duplicate daily record occurring continuously be filtered out, it is looser that the time threshold values of this spline filter just can design.

(1-2) for nonsensical word, use English to stop vocabulary and filter, only retain the word that really has real justice.

For this nonsensical word of the function word in English, can use English to stop vocabulary and filter, this table has comprised the most of function word in English.Consider in addition the singularity of daily record, further, can also filter most adjective and adverbial word, only retain the word that really has real justice.

2) extract daily record invariant, also by filtering out the variable in daily record text, extract the central structural information of daily record.

After daily record is carried out to pre-service, next need to extract the Template Information of daily record, Template Information is further to extract in daily record and the result of fault related content, Templated daily record will be used for carrying out the study of regulation, after templating, can reduce solution space, the fail close keyword matrix update finally obtaining is simplified, in addition, the daily record that template produces in the time of also can being used for matching system operation, carries out Fast Classification.

Variable in daily record refers to the word that can change in the middle of daily record text, comprises numeral, IP address, memory address, catalogue, filename, program name, port etc.

Particularly, the process of extraction daily record invariant is divided into two steps.

(2-1) adopt regular expression to remove numeral, catalogue, IP address, the memory address in daily record text.These are parts of easily removing, and wherein, deleted element adopts the symbol with different meanings to replace.Succession because word on grammer exists, simultaneously in order to distinguish different positions, by symbolic substitution, more can reflect the raw information of daily record, reduces distortion in leaching process.

(2-2) adopt the method based on word frequency statistics further log information to be screened, leave the Template Information of daily record.

In the present embodiment, the described method based on word frequency statistics can adopt improved TF-IDF to calculate the importance that each word accounts for whole daily record, and detailed process is as follows:

The word frequency here (term frequency, TF) refers to the frequency that some given words occur in all daily records of this equipment output.This numeral is the normalization to word number (term count), to prevent the long file of its deflection.(same word may have higher word number than short essay part in long article part, and no matter whether important this word is.) for the word t in a certain particular device _i, its importance can be expressed as:

${\mathrm{tf}}_{i,j}=\frac{n_{i,j}}{{\mathrm{\Σ}}_k{n}_{k,j}}$

N in above formula _{i, j}that this word is at equipment d _jin occurrence number, denominator is at equipment d _jin the occurrence number sum of all words.

Reverse file frequency (inverse document frequency, IDF) is the tolerance of a word general importance.A certain particular words t _iiDF, can be by this equipment d _jthe total daily record number producing is divided by the number of the daily record that comprises this word, then the business who obtains is taken the logarithm and obtained:

${\mathrm{idf}}_i=\log \frac{\left|D\right|}{\left|\right\{j:t_i\∈d_j\left\}\right|}$

Wherein:

| D|: the sum that this equipment produces

| { j:t _i∈ d _j|: comprise word t _idaily record number (be n _{i, j}≠ 0 number of files) if it is zero that this word not in daily record, will cause dividend, therefore generally use 1+|{j:t _i∈ d _j|, then:

tfidf _i，j＝tf _i，j×idf _i

By calculating TF-IDF, can obtain the importance that each word accounts for whole daily record, by predefined threshold values, daily record be screened afterwards.

(3) daily record Template Information filters

Before carrying out artificial classification, consider that template number is still huger, the present embodiment has taked the mode of automatic cluster to sort out daily record, further dwindles the space that needs artificial judgment.

In the automatic cluster stage, the content daily record similar with form can be put in the middle of a classification, in manual sort, only need to judge and to carry out trickle adjustment just passable each class like this.

Automatic classification has adopted DBSCAN algorithm to carry out automatic cluster to daily record template here, and the benefit of this algorithm maximum is to set in advance the quantity of clustering cluster, can be as required when classifying cluster dividing class automatically.

When chosen distance formula, consider the singularity of daily record text, adopted editing distance to investigate two distances between daily record.In concrete calculating, editing distance is revised: in editing distance, added coefficient, reduced the impact that length is adjusted the distance and calculated.The editing distance formula (LLD) of definition daily record, the distance between daily record A and daily record B is:

$\mathrm{LLD}(A,B)=\frac{2\×\mathrm{LD}(A,B)}{\mathrm{length}\left(A\right)+\mathrm{length}\left(B\right)}$

Wherein LD (A, B) refers to original editing distance, and length () represents the length of daily record.

(4) obtain fail close keyword matrix

(4-1) daily record in sample is carried out to manual sort.In this step, first keeper is revised as the mark with practical significance the automatic mark of system, then the daily record template in each classification is adjusted, revised the result of automatic cluster, guarantee that the daily record in each type is relevant to the label of the type.

(4-2) manual sort's result is learnt, set up fail close keyword matrix.

Fail close keyword matrix (matrix A) is that the word in template appears at the two-dimensional matrix that the probability of every kind of fault type forms, as shown below:

$A=\left(\begin{array}{cccc}{a}_{\mathrm{1,1}}& a_{\mathrm{1,2}}& ...& a_{1,n}\\ a_{\mathrm{2,1}}& a_{\mathrm{2,2}}& ...& a_{2,n}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ a_{m,1}& a_{m,2}& ...& a_{m,n}\end{array}\right)$

Fail close keyword matrix A is the matrix of a m * n, and m represents all various words numbers that appear in sample daily record template, and n represents the quantity of daily record fault type, a _{i, j}represent that the probability that i word belongs to the failure modes of j kind is a _{i, j}.(note: the probability is here a relative probability coefficent is not real probability.）

Probability a is below described in detail in detail _{i, w}solution procedure:

A _{i, w}be used for judging whether word belongs to certain type, its value representation be the frequency that each word occurs in specific fault type, can think that so probability that i word occurs in fault type w is the ratio of total number of word in number and the type w of word i in type w, release a fundamental formular to be:

$P(i,w)=\frac{\mathrm{count}(i,w)}{{\mathrm{\Σ}}_{j=1}^m\mathrm{count}(j,w)}$

Wherein P (i, w) represents i the probability that word occurs in fault type w, and count (i, w) represents the occurrence number of word i in fault type w.

Afterwards, above formula is revised, is added a scale-up factor:

$K(i,w)=-\log \left(\frac{\mathrm{sum}\left(i\right)-\mathrm{count}(i,w)}{\mathrm{sum}\left(i\right)}\right)+1$

Wherein sum (i) represents the number of times sum that word i occurs in all fault types of template base, that is:

$\mathrm{sum}\left(i\right)=\underset{t=1}{\overset{n}{\mathrm{\Σ}}}\mathrm{count}(i,t)$

Thus, can draw probability coefficent a _{i, w}the computing formula significance level that is word and the frequency of occurrences long-pending, that is:

a _i，w＝P(i，w)×K(i，w)

Importing two formulas above can obtain:

$a_{i,w}=\frac{\mathrm{count}(i,w)}{{\mathrm{\Σ}}_{j=1}^m\mathrm{count}(j,w)}[-\log (1-\frac{\mathrm{count}(i,w)}{\mathrm{sum}\left(i\right)})+1]$

On this basis, the detailed process of utilizing fail close keyword matrix to carry out fault type study is described below:

After obtaining manual sort's result, according to formula, according to row, carry out compute matrix A.To solve w, classify example as.

A) in program meeting statistical mask storehouse, belong to total number of the word of type w, be made as T (w), that is:

$T\left(w\right)=\underset{j=1}{\overset{m}{\mathrm{\Σ}}}\mathrm{count}(j,w)$

In the present embodiment, for convenient, fail close keyword matrix modified and upgraded, additionally adding one and be used for preserving T (w), reducing the number of times of double counting.

B) for each the word i in type w, all can calculate sum (i), same T (w) is similar, in order to reduce calculation times, also can increase additional space and be used for storing sum (i).Therefore, here fail close keyword matrix is expanded, increased a line and row are used for preserving statistical information, final matrix A is:

$A=\left(\begin{array}{ccccc}{a}_{\mathrm{1,1}}& a_{\mathrm{1,2}}& ...& a_{1,n}& \mathrm{sum}\left(1\right)\\ a_{\mathrm{2,1}}& a_{\mathrm{2,2}}& ...& a_{2,n}& \mathrm{sum}\left(2\right)\\ .& .& .& .& .\\ .& .& .& .& .\\ .& .& .& .& .\\ a_{m,1}& a_{m,2}& ...& a_{m,n}& \mathrm{sum}\left(m\right)\\ T\left(1\right)& T\left(2\right)& ...& T\left(n\right)& \end{array}\right)$

C) when calculating concrete a _{i, w}time, according to formula, need to calculate count (i, w).Consider that count (i, w) can change along with the renewal of fail close keyword matrix, therefore, when actual storage, in the middle of matrix A, can directly preserve count (i, w), when needs are used, can calculate a _{i, w}, owing to being used for calculating a _{i, w}variable be all kept in the middle of matrix A, therefore like this calculating can't be added extra expense to system.

In addition, after obtaining fail close keyword matrix, if classification is modified when system is moved, will adjust so fail close keyword matrix, adjust the three kinds of following situations that are divided into:

A), during the classification under need to revising certain template, system can scan all words in this template, and accordingly matrix is modified.

Suppose to have word i in template, in this template, occur n time, this template belonged to classification w originally, will change now classification u into.At this moment need to change as follows:

B) when needs add new template, system can directly be revised fail close keyword matrix, and concrete method is similar with the method for setting up matrix with artificial study.

C) when needs add new template to new classification, system can be added the new classification (being made as u classification) of a new row storage, then for each the word i in template, upgrade sum (i), then calculate T (u), in calculation template, the count (i, w) of each word is worth again.

5) fault log type judgement

There is fail close keyword matrix, just can to the daily record in the middle of system, carry out fault judgement easily.Concrete process is as follows:

Suppose to judge the fault type of daily record L, for the corresponding row in each the word lookup fail close keyword matrix A in daily record L, calculate the probable value that daily record L belongs to different faults type, if there is a most probable value, the fault type of daily record L is judged as to the corresponding fault type of most probable value.Provide the false code of concrete computation process below:

Further, in code, three last row are used for judging whether L has the possibility that belongs to other types.When there is other probable value and the difference between most probable value when being less than predetermined threshold, show cannot judge exactly the fault type under L by current fail close keyword matrix.Therefore must tell keeper, allow management select in most probable several situations.System log (SYSLOG) keeper's judgement, and utilize template and the judged result of L, according to the correction fail close keyword matrix described in step 4, occurs that similar fault log just can judge fault type exactly afterwards.

In addition, in the present embodiment, utilize fault correlation analysis that all fault logs that caused by same fault are all gathered together, form a fault propagation tree, and sequentially sort according to the time order and function occurring, for the basic reason of keeper's failure judgement is offered help.

As shown in Figure 3, the improved time-based fault correlation analytical approach of the embodiment of the present invention, comprises the following steps:

(1) by traditional time-based correlation analysis, daily record is carried out to polymerization.This method be the fault based on homology all can occur in the close time section in a this idea realize.Concrete method is to set up a time window, when fault occurs, according to each daily record of time order and function sequential scanning, within if daily record drops at the same time window, so just think that these daily records all belong to the fault log being produced by the same source of trouble, are classified as a class (referred to herein as " tuple ").

(2) daily record in each tuple is carried out to manual analysis, determine the time window size of different faults type.

(3) utilize fail close keyword matrix, in matrix, add a line, be used for storing every kind of time window size that fault type is different.

(4) select to carry out the daily record of fault diagnosis, confirm after the fault type of this daily record, by inquiry fail close keyword matrix, find the corresponding time window of this fault, then utilize time window to diagnose daily record.

Those skilled in the art will readily understand; the foregoing is only preferred embodiment of the present invention; not in order to limit the present invention, all any modifications of doing within the spirit and principles in the present invention, be equal to and replace and improvement etc., within all should being included in protection scope of the present invention.

Claims (10)

1. the computer system method for diagnosing faults based on daily record, comprises the following steps:

(S1) fault log analysis: the daily record in computer system is carried out to real-time analysis, utilize the failure modes result of the artificial study of fail close keyword matrix quantization, according to different fault types, every fault log is confirmed to fault type;

(S2) fault log is associated: utilizing the result of fault log analysis binding time window to carry out failure reason analysis, is a class the dependent failure log aggregation being caused by same fault, finds the root of this class fault.

2. method for diagnosing faults according to claim 1, wherein, step (S1) comprises the following steps:

(1) daily record pre-service, comprises that the daily record to repeating is filtered and insignificant word in every daily record is filtered;

(2) extract daily record invariant, also by filtering out the variable in daily record text, extract the central structural information of daily record;

(3) daily record Template Information filters: take automatic cluster mode that the content daily record similar with form is put in a classification;

(4) obtain fail close keyword matrix, wherein fail close keyword matrix is that word in template appears at the two-dimensional matrix that the probability of every kind of fault type forms;

(5) utilize fail close keyword matrix to carry out fault judgement to daily record.

3. method for diagnosing faults according to claim 2, wherein, step (1) comprises following sub-step:

(1-1) daily record repeating producing within certain period in same process is filtered;

(1-2) for the function word in English, use English to stop vocabulary and filter, only retain the word that really has real justice.

4. method for diagnosing faults according to claim 2, wherein, described variable refers to the word that can change in the middle of daily record text, comprises numeral, IP address, memory address, catalogue, filename, program name, port.

5. method for diagnosing faults according to claim 4, wherein, step (2) comprises following sub-step:

(2-1) adopt regular expression to remove numeral, catalogue, IP address, the memory address in daily record text;

(2-2) adopt the method based on word frequency statistics further log information to be screened, leave the Template Information of daily record.

6. method for diagnosing faults according to claim 5, in step (2-2), the described method based on word frequency statistics adopts improved TF-IDF to calculate each word t _iaccount for the importance tfidf of whole daily record _{i, j}=tf _{i, j}* idf _i, wherein,

Tf _{i, j}for word t _iat a certain particular device d _jin importance:

n in formula _{i, j}that this word is at equipment d _jin occurrence number, denominator is at equipment d _jin the occurrence number sum of all words;

Idf _ifor this word t _ireverse file frequency, by equipment d _jthe total daily record number producing is divided by the number of the daily record that comprises this word, then the business who obtains is taken the logarithm and obtained.

7. method for diagnosing faults according to claim 2, wherein, step (4) comprises following sub-step:

(4-1) daily record in sample is carried out to manual sort, the daily record template in each classification is adjusted, revise the result of automatic cluster;

(4-2) manual sort's result is learnt, is set up fail close keyword matrix A:

$A=\left(\begin{array}{cccc}{a}_{\mathrm{1,1}}& a_{\mathrm{1,2}}& ...& a_{1,n}\\ a_{\mathrm{2,1}}& a_{\mathrm{2,2}}& ...& a_{2,n}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ a_{m,1}& a_{m,2}& ...& a_{m,n}\end{array}\right)$

Wherein, m represents all various words numbers that appear in sample daily record template, and n represents the quantity of daily record fault type, a _{i, j}represent that i word belongs to the probability of j kind failure modes.

8. method for diagnosing faults according to claim 2, wherein, step (5) is specially: the fault type of supposing to judge daily record L, for the corresponding row in each the word lookup fail close keyword matrix A in daily record L, calculate the probable value that daily record L belongs to different faults type, if there is a most probable value, the fault type of daily record L is judged as to the corresponding fault type of most probable value.

9. fault according to claim 1 is examined method, and wherein, step (S2) comprises the following steps:

(1) by time-based correlation analysis traditionally, daily record is carried out to polymerization;

(2) daily record in each tuple is carried out to manual analysis, determine the time window size of different faults type;

(3) utilize fail close keyword matrix, in matrix, add a line, be used for storing every kind of time window size that fault type is different;

(4) select to carry out the daily record of fault diagnosis, confirm after the fault type of this daily record, by inquiry fail close keyword matrix, find the corresponding time window of this fault, then utilize time window to diagnose daily record.

10. the computer system trouble-shooter based on daily record, comprises with lower module:

Fault log analysis module, for the daily record of computer system is carried out to real-time analysis, utilizes the failure modes result of the artificial study of fail close keyword matrix quantization, according to different fault types, every fault log is confirmed to fault type;

Fault log relating module, for utilizing result the binding time window of fault log analysis to carry out failure reason analysis, is a class the dependent failure log aggregation being caused by same fault, finds the root of this class fault.

Images