CN107783990A - A kind of data compression method and terminal - Google Patents

Abstract

The embodiment of the present invention discloses a data compression method and a terminal, the method comprising: the terminal calculates the first characteristic value of the data block to be compressed through the first calculation strategy; the terminal judges whether the first reference value exists in the first search library ; If the first reference value exists, the terminal uses a similar compression technique to use the data block corresponding to the first reference value as a reference data block to compress the data block to be compressed; if the first reference value does not exist value, then the terminal calculates the second eigenvalue of the data block to be compressed through the second calculation strategy; the terminal judges whether there is a second reference value in the second search library, and the second reference value is the same as the An eigenvalue with the same second eigenvalue; if the second reference value exists, the terminal compresses the data block to be compressed using a data block corresponding to the second reference value as a reference data block through a similar compression technique. According to the present invention, the compression ratio can be improved.

Description

A data compression method and terminal

technical field

The invention relates to the field of computer technology, in particular to a data compression method and a terminal.

Background technique

Data storage is an integral part of a computer system, and disks, tapes, flash memory, non-volatile memory, cloud storage, etc. can all be used to store data. Huge amounts of data are generated every day all over the world. If these data are stored unprocessed, they will take up a large storage space and the cost will be high. In order to effectively save these data, data reduction techniques are usually used to reduce these data Commonly used data reduction techniques include data deduplication, general lossless compression, and similar compression.

Taking the backup scenario as an example, there is usually a large amount of identical data between two consecutive backup files. The deduplication technology can be used to effectively reduce the actual number of backup files saved, that is, the latter backup file only saves the same data as the previous backup file" different data", which has the effect of saving storage costs and reducing the amount of data transmitted by the network. However, a large number of operations in database applications are to modify queries, and most of the "different data" that needs to be downloaded are generated by modification. There is a "similarity" between the "different data" generated by these modifications and the data backed up before the modification. Delta compression technology can be used to further compress the similar data. The compression principle of Delta compression technology is as follows:

First select similar data for the data to be compressed, then use Delta compression technology to compress the data to be compressed with reference to the similar data, the higher the similarity, the better the compression effect; the defect of the prior art is that the data to be compressed consists of many If the reference data block is selected for these data blocks through a higher similarity standard, some data blocks may not be able to find the reference data block, resulting in the data block not being able to find the reference data block. Based on the Delta compression technology compression, if the reference data block is selected for these data through a lower similarity standard, then the reference data block screened for some data blocks has a lower similarity angle with the data block, resulting in the compression effect of the data block not good.

Contents of the invention

The embodiment of the invention discloses a data compression method and a terminal, which can improve the compression rate.

In the first aspect, the embodiment of the present invention provides a data compression method, the method includes: the terminal calculates the first characteristic value of the data block to be compressed through the first calculation strategy; the terminal judges whether there is the first A reference value, the first reference value is the same feature value as the first feature value, the first search library contains N feature values and each feature value is based on the first calculation strategy for the feature The data block corresponding to the value is calculated, and the N feature values in the first search library correspond to N data blocks one by one, and N is greater than or equal to 1; if the first reference value exists, the terminal uses similar The compression technology uses the data block corresponding to the first reference value as a reference data block to compress the data block to be compressed; if the first reference value does not exist, the terminal calculates the data block to be compressed by using a second calculation strategy The second eigenvalue of the data block, when the similarity of the two data blocks is higher than the first similarity threshold, the eigenvalues of the two data blocks calculated by the first calculation strategy are the same; the two data blocks When the similarity of is higher than the second similarity threshold, the feature values of the two data blocks calculated by the second calculation strategy are the same, and the first similarity threshold is higher than the second similarity threshold; the terminal Judging whether there is a second reference value in the second search library, the second reference value is the same eigenvalue as the second eigenvalue, the second search library contains N eigenvalues and each eigenvalue is based on The second calculation strategy calculates the data block corresponding to the feature value, and the N feature values in the second search library correspond to the N data blocks one by one; if there is the second reference value , then the terminal compresses the to-be-compressed data block by using a similar compression technique using the data block corresponding to the second reference value as a reference data block.

By performing the above steps, the terminal compresses the data in units of data blocks, and first judges whether there is a reference data block with a high similarity to the data block to be compressed when compressing, and if so, refers to the reference data block with a high similarity Compress the data block to be compressed, if it does not exist, judge whether there is a reference data block with a lower similarity with the data block to be compressed, and if it exists, compress the data block with reference to the reference data block with a lower similarity That is to say, the embodiment of the present invention selects the reference data block from high to low through multi-level similarity standards, which generally improves the compression rate of data compression and saves storage space.

With reference to the first aspect, in a first possible implementation manner of the first aspect, before the terminal calculates the preset first eigenvalue of the data block to be compressed through the first calculation strategy, the method further includes: the The terminal divides M data units from the data block to be compressed, each of the M data units corresponds to its own initial reference value, and M is greater than or equal to 1; the terminal calculates the preset The first characteristic value of the data block to be compressed includes: substituting initial reference values of at least two data units in the M data units into preset P filter functions to calculate the first characteristic value of the data block to be compressed value, P is greater than or equal to 2; the terminal calculates the second characteristic value of the data block to be compressed through the second calculation strategy, including: substituting the initial reference values of at least two data units in the M data units into the preset The second eigenvalue of the data block to be compressed is calculated in the Q filter functions provided, and the P filter functions include the Q filter functions.

With reference to the first aspect, or the first possible implementation of the first aspect, in the second possible implementation of the first aspect, after compressing the data block to be compressed, the method further includes: the The terminal adds the first feature value to the first search library, and adds the second feature value to the second search library, and the first feature value in the first search library The corresponding data block is the data block to be compressed, and the data block corresponding to the second feature value in the second lookup library is the data block to be compressed.

With reference to the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, the method further includes: the terminal will exist in the first search library for more than a predetermined time It is assumed that the feature values of the time threshold are deleted from the first search library, and the feature values existing in the second search library for a time exceeding the preset time threshold are deleted from the second search library.

In combination with the first aspect, or the first possible implementation of the first aspect, or the second possible implementation of the first aspect, or the third possible implementation of the first aspect, in the first aspect of the first In four possible implementation manners, before the terminal calculates the first eigenvalue of the data block to be compressed through the first calculation strategy, the method further includes: during the statistical history compression process of the terminal, the second calculation strategy is used to calculate the multiple A plurality of eigenvalues calculated by the data block, and determining the number of identical eigenvalues in the second lookup library and the plurality of eigenvalues; the plurality of eigenvalues were not calculated by the first calculation strategy during the historical compression process The feature value of the data block; the terminal calculates the first bid winning rate according to the number of feature values in the plurality of feature values and the number of the same feature value; when the first bid winning rate is higher than the preset first winning bid rate When the threshold is reached, if there is a data block to be compressed, execute the step of calculating the first characteristic value of the data block to be compressed by the terminal through a first calculation strategy.

In combination with the first aspect, or the first possible implementation of the first aspect, or the second possible implementation of the first aspect, or the third possible implementation of the first aspect, in the first aspect of the first In five possible implementation manners, before the terminal calculates the second eigenvalue of the data block to be compressed by using the second calculation strategy, the method further includes: during the statistical history compression process of the terminal, use the first calculation strategy to A plurality of eigenvalues obtained by compressing a plurality of data blocks, and determining the number of identical eigenvalues in the first search library and the plurality of eigenvalues; the historical compression process did not calculate the second calculation strategy feature values of a plurality of data blocks; the terminal calculates a second bid winning rate according to the number of feature values in the plurality of feature values and the number of the same feature values; when the second winning rate is lower than the preset first When the bid winning threshold is selected, the step of calculating the second characteristic value of the data block to be compressed by the terminal through a second calculation strategy is performed if the first reference value does not exist.

In the second aspect, an embodiment of the present invention provides a terminal, which includes: a first calculation unit, configured to calculate a first characteristic value of a data block to be compressed through a first calculation strategy; a first judging unit, configured to judge the first Find whether there is a first reference value in the library, the first reference value is the same feature value as the first feature value, the first search library contains N feature values and each feature value is based on the first feature value A calculation strategy is calculated for the data block corresponding to the feature value, and the N feature values in the first search library correspond to N data blocks one by one, and N is greater than or equal to 1; the first compression unit is used for When the first judging unit judges that the first reference value exists, compress the data block to be compressed by using a similar compression technique using the data block corresponding to the first reference value as a reference data block; the second computing unit, When the first judging unit judges that the first reference value does not exist, the second calculation strategy is used to calculate the second characteristic value of the data block to be compressed, and the similarity between the two data blocks is higher than the first When the similarity threshold is reached, the feature values of the two data blocks calculated by the first calculation strategy are the same; when the similarity of the two data blocks is higher than the second similarity threshold, the feature values calculated by the second calculation strategy The feature values of the two data blocks obtained are the same, and the first similarity threshold is higher than the second similarity threshold; the second judging unit is used to judge whether there is a second reference value in the second search library, and the The second reference value is the same eigenvalue as the second eigenvalue, the second search library contains N eigenvalues, and each eigenvalue is a data block corresponding to the eigenvalue based on the second calculation strategy It is calculated that the N feature values in the second search library correspond to the N data blocks one by one; the second compression unit is used to determine the existence of the second reference value in the second judgment unit , the data block to be compressed is compressed by using a similar compression technique with the data block corresponding to the second reference value as a reference data block.

By performing the above operations, the terminal compresses the data in units of data blocks. When compressing, it first judges whether there is a reference data block with a high similarity to the data block to be compressed, and if it exists, it refers to the reference data block with a high similarity. Compress the data block to be compressed, if it does not exist, judge whether there is a reference data block with a lower similarity with the data block to be compressed, and if it exists, compress the data block with reference to the reference data block with a lower similarity That is to say, the embodiment of the present invention selects the reference data block from high to low through multi-level similarity standards, which generally improves the compression rate of data compression and saves storage space.

With reference to the second aspect, in the first possible implementation manner of the second aspect, the terminal further includes: a dividing unit, configured to calculate, at the first calculation unit, the number of preset data blocks to be compressed by using a first calculation strategy Before the first eigenvalue, M data units are divided from the data block to be compressed, and each data unit in the M data units corresponds to its own initial reference value, and M is greater than or equal to 1; the first calculation unit specifically It is used to substitute the initial reference values of at least two data units in the M data units into preset P filter functions to calculate the first characteristic value of the data block to be compressed, where P is greater than or equal to 2; the second The second calculation unit is specifically configured to substitute the initial reference values of at least two data units in the M data units into preset Q filter functions to calculate the second eigenvalue of the data block to be compressed, the P The filter functions include the Q filter functions.

With reference to the second aspect, or the first possible implementation manner of the second aspect, in the second possible implementation manner of the second aspect, the terminal further includes: an adding unit, configured to add the first feature value added to the first search library, and the second feature value is added to the second search library, the data block corresponding to the first feature value in the first search library is the to-be A data block is compressed, and the data block corresponding to the second characteristic value in the second lookup library is the data block to be compressed.

With reference to the second possible implementation manner of the second aspect, in a third possible implementation manner of the second aspect, the terminal further includes: a deletion unit configured to delete the time stored in the first search library The feature values exceeding the preset time threshold are deleted from the first lookup library, and the feature values existing in the second lookup library for a time exceeding the preset time threshold are deleted from the second lookup library .

In combination with the second aspect, or the first possible implementation of the second aspect, or the second possible implementation of the second aspect, or the third possible implementation of the second aspect, in the first possible implementation of the second aspect In four possible implementation manners, the method further includes: a first statistical unit, configured to make statistics on the historical compression process before the first calculation unit calculates the first characteristic value of the data block to be compressed through the first calculation strategy A plurality of eigenvalues calculated for a plurality of data blocks through the second calculation strategy, and determining the number of identical eigenvalues in the second search library and the plurality of eigenvalues; A calculation strategy to calculate the feature values of the plurality of data blocks; a third calculation unit, configured to calculate a first bid-winning rate according to the number of feature values in the plurality of feature values and the number of the same feature values; when the When the first successful bid rate is higher than the preset first successful bid threshold, if there is a data block to be compressed, the first calculation unit is triggered to calculate the first characteristic value of the data block to be compressed by using a first calculation strategy.

In combination with the second aspect, or the first possible implementation of the second aspect, or the second possible implementation of the second aspect, or the third possible implementation of the second aspect, in the first possible implementation of the second aspect In five possible implementation manners, the terminal further includes: a second statistical unit, configured for the terminal to count Compressing multiple eigenvalues obtained by compressing multiple data blocks through the first calculation strategy in the historical compression process, and determining the number of identical eigenvalues in the first lookup library and the multiple eigenvalues; Calculate the eigenvalues of the plurality of data blocks through the second calculation strategy; a fourth calculation unit, configured to calculate a second bid-winning rate according to the number of eigenvalues in the plurality of eigenvalues and the number of the same eigenvalues ; when the second bid winning rate is lower than the preset second bid winning threshold, triggering the second calculation unit to calculate the value of the data block to be compressed through a second calculation strategy when the first reference value does not exist the second eigenvalue.

In a third aspect, an embodiment of the present invention provides a terminal, the terminal includes a processor and a memory: the memory is used to store data and programs; the processor invokes the program in the memory to perform the following operations: by The first calculation strategy calculates the first eigenvalue of the data block to be compressed; judges whether there is a first reference value in the first search library, the first reference value is the same eigenvalue as the first eigenvalue, and the first reference value is the same as the first eigenvalue, and the first reference value is the same as the first eigenvalue. A lookup library includes N eigenvalues, and each eigenvalue is calculated based on the first calculation strategy for the data block corresponding to the eigenvalues, and the N eigenvalues in the first lookup library are in one-to-one correspondence N data blocks, where N is greater than or equal to 1; if there is the first reference value, the data block corresponding to the first reference value is used as a reference data block to compress the data block to be compressed by a similar compression technique; if not If the first reference value exists, the second characteristic value of the data block to be compressed is calculated by the second calculation strategy, and when the similarity of the two data blocks is higher than the first similarity threshold, the second characteristic value is calculated by the first calculation strategy The eigenvalues of the two data blocks are the same; when the similarity of the two data blocks is higher than the second similarity threshold, the eigenvalues of the two data blocks calculated by the second calculation strategy are the same , the first similarity threshold is higher than the second similarity threshold; judging whether there is a second reference value in the second search library, the second reference value is the same feature value as the second feature value, the The second search library contains N eigenvalues, and each eigenvalue is calculated based on the second calculation strategy for the data block corresponding to the eigenvalue, and the N eigenvalues in the second search library are one by one Corresponding to the N data blocks; if the second reference value exists, compress the data block to be compressed by using the data block corresponding to the second reference value as a reference data block through a similar compression technique.

By performing the above operations, the terminal compresses the data in units of data blocks. When compressing, it first judges whether there is a reference data block with a high similarity to the data block to be compressed, and if it exists, it refers to the reference data block with a high similarity. Compress the data block to be compressed, if it does not exist, judge whether there is a reference data block with a lower similarity with the data block to be compressed, and if it exists, compress the data block with reference to the reference data block with a lower similarity That is to say, the embodiment of the present invention selects the reference data block from high to low through multi-level similarity standards, which generally improves the compression rate of data compression and saves storage space.

With reference to the third aspect, in a first possible implementation manner of the third aspect, before the processor calculates the preset first eigenvalue of the data block to be compressed through the first calculation strategy, it is further configured to: The data block is divided into M data units, each of the M data units corresponds to its own initial reference value, and M is greater than or equal to 1; the processor calculates the preset data to be compressed through the first calculation strategy The first eigenvalue of the block is specifically: substituting initial reference values of at least two data units among the M data units into preset P filter functions to calculate the first eigenvalue of the data block to be compressed, P is greater than or equal to 2; the processor calculates the second characteristic value of the data block to be compressed through a second calculation strategy, specifically: substituting the initial reference values of at least two data units in the M data units into the preset The second eigenvalue of the data block to be compressed is calculated in the Q filter functions provided, and the P filter functions include the Q filter functions.

With reference to the third aspect, or the first possible implementation of the third aspect, in the second possible implementation of the third aspect, after compressing the data block to be compressed, the processor is further configured to: adding the first feature value to the first lookup library, and adding the second feature value to the second lookup library, in the first lookup library the first feature value corresponds to The data block is the data block to be compressed, and the data block corresponding to the second characteristic value in the second lookup library is the data block to be compressed.

With reference to the second possible implementation manner of the third aspect, in a third possible implementation manner of the third aspect, the processor is further configured to: set the time that exists in the first search library to exceed a preset time The characteristic values of the time threshold are deleted from the first search library, and the characteristic values existing in the second search library for a time exceeding the preset time threshold are deleted from the second search library.

In combination with the third aspect, or the first possible implementation of the third aspect, or the second possible implementation of the third aspect, or the third possible implementation of the third aspect, in the first possible implementation of the third aspect In four possible implementation manners, before the processor calculates the first eigenvalue of the data block to be compressed through the first calculation strategy, it is also used to: calculate the data obtained from multiple data blocks through the second calculation strategy during the statistical history compression process. multiple eigenvalues, and determine the number of identical eigenvalues in the second lookup library and the multiple eigenvalues; the features of the multiple data blocks have not been calculated by the first calculation strategy in the historical compression process value; according to the number of eigenvalues in the plurality of eigenvalues and the number of the same eigenvalues, calculate the first winning rate; when the first winning rate is higher than the preset first winning threshold, if there is a If the data block is a data block, the operation of calculating the first characteristic value of the data block to be compressed by using the first calculation strategy is performed.

In combination with the third aspect, or the first possible implementation of the third aspect, or the second possible implementation of the third aspect, or the third possible implementation of the third aspect, in the first possible implementation of the third aspect In five possible implementation manners, before the processor calculates the second eigenvalue of the data block to be compressed through the second calculation strategy, it is also used to: use the first calculation strategy to perform statistics on multiple data blocks during historical compression. Compressing the multiple eigenvalues obtained, and determining the number of identical eigenvalues in the first lookup library and the multiple eigenvalues; the multiple data blocks were not calculated by the second calculation strategy during the historical compression process eigenvalue; according to the number of eigenvalues in the plurality of eigenvalues and the number of the same eigenvalues, calculate the second winning rate; when the second winning rate is lower than the preset second winning threshold, execute the The operation of calculating the second characteristic value of the data block to be compressed by using the second calculation strategy if the first reference value does not exist is described.

By implementing the embodiment of the present invention, the terminal compresses the data in units of data blocks, and first judges whether there is a reference data block with a high similarity to the data block to be compressed, and if so, refers to the reference data block with a high similarity. The data block is compressed to the data block to be compressed. If it does not exist, it is judged whether there is a reference data block with a lower similarity with the data block to be compressed. If it exists, the reference data block with a lower similarity is referred to the data to be compressed Block compression; that is to say, the embodiment of the present invention selects reference data blocks from high to low through multi-level similarity standards, which generally improves the compression rate during data compression and saves storage space.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are For some embodiments of the present invention, those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is a schematic flow chart of a data compression method provided by an embodiment of the present invention;

Fig. 2 is a schematic flow chart of another data compression method provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of a scenario for generating feature values provided by an embodiment of the present invention;

FIG. 4 is a schematic diagram of another scenario for generating feature values provided by an embodiment of the present invention;

Fig. 5 is a schematic diagram of a scenario of selecting reference data blocks according to priority provided by an embodiment of the present invention;

FIG. 6 is a schematic flowchart of another data compression method provided by an embodiment of the present invention;

FIG. 7 is a schematic flowchart of another data compression method provided by an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of a terminal provided by an embodiment of the present invention;

FIG. 9 is a schematic structural diagram of another terminal provided by an embodiment of the present invention.

Detailed ways

The following drawings clearly and completely describe the technical solutions in the embodiments of the present invention. The general idea of the present invention is to divide the "similar" standard in similar compression (being Delta compression) technology into at least two grades (or be called "priority"), when there is data to be compressed, first pass through the higher grade The "similar" criterion is used to find data similar to the data to be compressed. If not found, the "similar" criterion of a lower level is used to find data similar to the data to be compressed, and so on until the data similar to the data to be compressed is found. Compress data with similar data, and compress the data to be compressed with reference to the similar data through Delta compression technology. In the embodiment of the present invention, the compression rate is equal to the size of the data before compression divided by the size of the data after compression. Therefore, the larger the compression rate, the smaller the compressed data and the better the compression effect.

Please refer to FIG. 1 . FIG. 1 is a schematic flowchart of a data compression method provided by an embodiment of the present invention. The method includes but is not limited to the following steps.

Step S101: the terminal calculates a first characteristic value of a data block to be compressed by using a first calculation strategy.

Specifically, the terminal may be a mobile phone, a tablet computer, a notebook computer, a handheld computer, a mobile internet device (English: mobile internet device, MID for short), a wearable device (such as a smart watch (such as iWatch, etc.), a smart bracelet, pedometer, etc.) or other terminal devices involving data compression. When compressing data, data block can be used as a unit to compress data, and the commonly used size of data block is 4K, 8K, etc., and the data block to be compressed refers to the data block to be compressed currently.

In the embodiment of the present invention, multiple calculation strategies can be preset and each calculation strategy can calculate a data block to obtain a feature value, which is used to reflect the characteristics of the calculated data block, two data blocks The closer the eigenvalues of the two data blocks calculated by the calculation strategy are, the closer they are.

There are priorities among the multiple computing strategies. Among any two computing strategies with adjacent priorities, the computing strategy with higher priority is called the first computing strategy, and the computing strategy with lower priority can be called the second computing strategy. Computing strategy, there is the following relationship between the first computing strategy and the second computing strategy: when the similarity of two data blocks is higher than the first similarity threshold, the two data blocks calculated by the first computing strategy The eigenvalues are the same; when the similarity of the two data blocks is higher than the second similarity threshold, the eigenvalues of the two data blocks calculated by the second calculation strategy are the same, and the first similarity threshold is higher than the second similarity threshold . That is to say, the standard for measuring the similarity of two data blocks based on a computing strategy with a higher priority is higher than the standard for measuring the similarity of two data blocks based on a computing strategy with a lower priority.

In the embodiment of the present invention, the terminal calculates the eigenvalue of the data block to be compressed based on the first calculation strategy, and the calculated eigenvalue may be referred to as the first eigenvalue. The specific manner of calculating the data block to be compressed by using the first calculation strategy is not limited here, and an optional implementation manner is exemplified below with reference to FIG. 2 .

Step 1: The terminal divides M data units from the data block to be compressed, and each data unit in the M data units corresponds to its own initial reference value, and M is greater than or equal to 1; for example, 4 bytes ( byte) length as the unit, the data block to be compressed is divided into M parts, and each part is called a data unit, and a hash is calculated for each data unit in the M data units through the preset hash function Hash value, the calculated hash value is the initial reference value, the initial reference value of the i-th data unit can be expressed as h( _xi ), i is greater than or equal to 1 and less than or equal to M.

Step 2: The terminal substitutes the initial reference values of at least two data units among the M data units into preset P filter functions to calculate the first characteristic value of the data block to be compressed, where P is greater than or equal to 2. The specific implementation can be as follows:

1. Configure the P group of parameters. The jth group of parameters in the P group of parameters can be expressed as: {S _j , p _j , q _j }, j is greater than or equal to 1 and less than or equal to P, S _j is the sampling ratio, (p _j , q _j ) is a linear parameter group;

2. Sampling from {h(x ₁ ), h(x ₂ ), ... h(x _i )..., h(x _M )} according to the sampling ratio S _j to obtain the sampling sequence {h(x _1j ), h( x _2j ),...h(x _kj )}, k is greater than or equal to 1 and less than or equal to M;

3. With the filter function f(kj)=h(x _kj )×p _j +q _j constructed based on the parameter set (p _j , q _j ), for the sampling sequence {h(x _1j ), h(x _2j ), ..., each element in h(x _kj )} is calculated separately, and the calculation sequence {f(1j), f(2j), ..., f(kj)} is obtained.

4. Obtain the maximum value in the calculation sequence {f(1j), f(2j), ... f(kj)}, which can be expressed as f(j) _max ;

5. The first eigenvalue H _P ={f(1) _max , f(2) _max , . . . , f(P) _max }.

In order to better understand the meaning expressed by the above formula, specific numerical values are provided as examples below. Assume that P is equal to 3, M is equal to 5, and K=3, that is, three sets of parameters {S ₁ , p ₁ , q ₁ }, {S1, p ₂ , q ₂ } and {S ₃ , p ₃ , q ₃ }, the sampling sequence obtained based on S ₁ , S1 and S ₃ sampling is {h(x ₁ ), h(x ₂ ), h (x ₃ )}.

Assume that the calculation results based on the above parameters are as follows:

f(11)=h(x ₁ )×p ₁ +q ₁ =6; f(21)=h(x ₂ )×p ₁ +q ₁ =9; f(31)=h(x ₂ )×p ₁ +q ₁ =1;

f(12)=h(x ₁ )×p ₂ +q ₂ =1; f(22)=h(x ₂ )×p ₂ +q ₂ =4; f(32)=h(x ₂ )×p ₂ +q ₂ =5;

f(13)=h(x ₁ )×p ₃ +q ₃ =3; f(23)=h(x ₂ )×p ₃ +q ₃ =2; f(33)=h(x ₂ )×p ₃ +q ₃ =7;

Then, calculate the maximum value f(1) _max in the sequence {f(11), f(21), ...f(31)} is 9, calculate the sequence {f(12), f(22), ...f(32) )} in the maximum value f(2) _max is 5, calculate the maximum value f(3) _max in the sequence {f(13), f(23), ... f(33)} is 7, so the first eigenvalue H _P ={9,5,7}. In an optional solution, the input of each of the P filtering functions is {h(x ₁ ), h(x ₂ ),...h( _xi )..., h(x _M )}, Figure 3 is a schematic diagram of the corresponding scene; in yet another alternative solution, the input of one filter function among the P filter functions is {h(x ₁ ), h(x ₂ ),...h(x _i )... , h(x _M )}, the output of the filter function is used as the input of another filter function, and the output of the other filter function is used as the input of another filter function, and so on. Figure 4 is a schematic diagram of the corresponding scene.

It should be noted that, for the manner of calculating the eigenvalues of other data blocks based on the first calculation strategy, reference may be made to the manner of calculating the first eigenvalue of the data block to be compressed based on the first calculation strategy.

Step S102: The terminal judges whether there is a first reference value in the first search library, the first reference value is the same feature value as the first feature value, the first search library contains N feature values and each feature value The data block corresponding to the feature value is calculated based on the first calculation strategy, and the N feature values in the first search library correspond to N data blocks one by one, and N is greater than or equal to 1.

Specifically, each computing strategy corresponds to a search library. For convenience, the search library corresponding to the first computing strategy can be called the first search library, and the search library corresponding to the second computing strategy can be called the second search library. And so on. The following takes the first search library as an example to describe the characteristics of the search library. The first search library contains the correspondence between eigenvalues and data blocks. Each eigenvalue in the first search library is based on the first calculation strategy. The data block corresponding to the feature value is calculated. Assuming that the search database contains N feature values, the N feature values correspond to the N data blocks one by one. The N data blocks may be data blocks that have been compressed before the compressed data block. The second lookup library also contains N eigenvalues corresponding to the N data blocks one-to-one, except that the N eigenvalues in the second lookup library are calculated for the N data blocks through the second calculation strategy owned.

The terminal needs to judge whether the first search database contains the same feature value as the first feature value, and if so, the same feature value can be called the first reference value for the convenience of subsequent description.

Step S103: If the first reference value exists, the terminal compresses the data block to be compressed by using the data block corresponding to the first reference value as a reference data block through a similar compression technique.

Specifically, if the first reference value exists in the first lookup library, it indicates that there is a data block similar to the data block to be compressed, because only two similar data blocks can be calculated based on the eigenvalues of the first calculation strategy. same.

Step S104: If the first reference value does not exist, the terminal calculates a second characteristic value of the data block to be compressed through a second calculation strategy.

Specifically, the way of using the second calculation strategy to calculate the eigenvalue of the data block to be compressed is similar to the principle of using the first calculation strategy to calculate the eigenvalue of the data block to be compressed. In an optional solution, when the terminal calculates the preset first characteristic value of the data block to be compressed through the first calculation strategy, specifically: the initial reference value of at least two data units among the M data units When substituting into the preset P filter functions to calculate the first eigenvalue of the data block to be compressed, the terminal calculates the second eigenvalue of the data block to be compressed through the second calculation strategy, specifically: the M data The initial reference values of at least two data units in the unit are substituted into preset Q filter functions to calculate the second characteristic value of the data block to be compressed, and the P filter functions include the Q filter functions. Referring to the above example where P is equal to 3, M is equal to 5, and K=3, assuming that Q=2, then the second eigenvalue may be H _Q ={9,5}.

Step S105: The terminal judges whether there is a second reference value in the second search library, the second reference value is the same feature value as the second feature value, the second search library contains N feature values and each feature value The data block corresponding to the feature value is calculated based on the second calculation strategy, and the N feature values in the second search library correspond to N data blocks one by one;

Specifically, the terminal needs to judge whether the second search library contains the same feature value as the second feature value, and if so, the same feature value can be called the second reference value for the convenience of subsequent description.

Step S106: If the second reference value exists, the terminal compresses the data block to be compressed by using the data block corresponding to the second reference value as a reference data block through a similar compression technique.

Specifically, if there is a second reference value in the second lookup library, it indicates that there is a data block similar to the data block to be compressed, because only two similar data blocks can be calculated based on the eigenvalues obtained by the second calculation strategy. same.

It can be seen from the above method that if there is a first reference value, it indicates that there is a data block with a higher similarity, so the data block with a higher similarity is referred to for compression; if there is no first reference value but there is a second reference value, it indicates that there is A data block with a lower similarity is therefore compressed with reference to the data block with a lower similarity, improving the overall compression ratio. FIG. 5 is a schematic flowchart of a data compression process provided by an embodiment of the present invention, in which multiple computing strategies are shown, and the priorities of the computing strategies decrease from top to bottom.

In an optional solution, after the data block to be compressed is compressed, the method further includes: the terminal adding the first feature value to the first search library, and adding the second feature value to the In the second search library, the data block corresponding to the first characteristic value in the first search library is the data block to be compressed, and the data block corresponding to the second characteristic value in the second search library is the data to be compressed piece. That is to say, the compressed data block can be used as a reference data block for subsequent compression.

In yet another optional solution, the method further includes: the terminal deleting the feature values existing in the first search library for a time exceeding a preset time threshold from the first search library, and deleting the feature values existing in the first search library The feature values whose time exceeds the preset time threshold in the second search library are deleted from the second search library. The preset time threshold can be pre-set according to actual needs, that is to say, the feature values that have existed for a long time can be deleted from the search library, because the data blocks corresponding to the feature values that have existed for a long time probably do not have reference Value, so delete it to free up more storage space. Optionally, when deleting, you can also first determine whether the data block corresponding to the feature value has never been used as a reference data block. If the time of the feature value If the data block is longer and the corresponding data block is not used as a reference data block, then the characteristic value is deleted from the lookup library. Fig. 6 shows a detailed flowchart.

In yet another optional solution, before the terminal calculates the first eigenvalue of the data block to be compressed through the first calculation strategy, the method further includes: the terminal statistics history compression process through the second calculation strategy on multiple data A plurality of eigenvalues calculated by the block, and determining the number of identical eigenvalues in the second search library and the plurality of eigenvalues, the eigenvalues of the plurality of data blocks were not calculated by the first calculation strategy in the historical compression process ; The terminal calculates a first bid winning rate according to the number of eigenvalues in the plurality of eigenvalues and the number of the same eigenvalues; when the first winning rate is higher than the preset first winning threshold, if there is a data block to be compressed , then execute the step of calculating the first eigenvalue of the data block to be compressed by the terminal through the first calculation strategy.

Specifically, the first computing strategy in this optional solution is the computing strategy with the highest priority among multiple computing strategies, and this optional solution describes how to add the first computing strategy to the multiple computing strategies. When the first computing strategy is not added to the multiple computing strategies, the second computing strategy is the computing strategy with the highest priority, therefore, the terminal will use the second computing strategy to calculate the characteristic value of the data block instead of passing The first calculation strategy calculates the feature value of the data block. Assuming that the number of multiple eigenvalues calculated by the second calculation strategy for multiple data blocks in the statistical history compression process is X, and determining the number Y of the same eigenvalues in the second search library and the multiple eigenvalues, Then, the first winning rate can be Y divided by X, and when the first winning rate is higher than the preset first winning threshold, it indicates that there are many data blocks corresponding to feature values in the second search library that can be used as reference data block, which reflects that the second computing strategy may have too low a standard for measuring the similarity of two data blocks, so it is necessary to add a computing strategy with a higher priority, that is, the first computing strategy, to the multiple computing strategies.

In yet another optional solution, before the terminal calculates the second eigenvalue of the data block to be compressed through the second calculation strategy, the method further includes: during the compression history of the terminal statistics, the first calculation strategy is used to calculate the A plurality of eigenvalues obtained by data block compression, and determining the number of identical eigenvalues in the first lookup library and the plurality of eigenvalues; the characteristics of the plurality of data blocks were not calculated by the second calculation strategy during the historical compression process value; the terminal calculates a second bid winning rate according to the number of feature values in the plurality of feature values and the number of the same feature value; when the second bid winning rate is lower than the preset second winning threshold, execute the if not present For the first reference value, the terminal calculates a second characteristic value of the data block to be compressed through a second calculation strategy.

Specifically, the second computing strategy in this optional solution is the computing strategy with the lowest priority among multiple computing strategies, and this optional solution describes how to add the second computing strategy to the multiple computing strategies. When the second computing strategy is not added to the multiple computing strategies, the first computing strategy is the computing strategy with the lowest priority, therefore, the terminal will use the first computing strategy to calculate the eigenvalue of the data block without passing The second calculation strategy calculates the feature value of the data block. Assuming that the number of multiple eigenvalues calculated by the first calculation strategy for multiple data blocks in the statistical history compression process is S, and determining the number T of the same eigenvalues in the first search library and the multiple eigenvalues, Then, the second winning rate can be T divided by S, and when the second winning rate is lower than the preset second winning threshold, it indicates that there are fewer data blocks corresponding to feature values in the first search library that can be used as Referring to data blocks, this reflects that the first computing strategy may have too high a standard for measuring the similarity of two data blocks, so it is necessary to add a computing strategy with a lower priority, that is, the second computing strategy, to the plurality of computing strategies.

Based on the above two optional schemes of adding the first calculation strategy and the second calculation strategy, other schemes can also be derived, for example, referring to the calculation of the second bid winning rate based on the first calculation strategy and the calculation of the first bid winning rate based on the second calculation strategy According to the principle of the rate, calculate the bid winning rate of each of the above multiple computing strategies. When the winning rate of the computing strategy with the highest priority is higher than the preset upper threshold _TH , it indicates that the computing strategy measures The similarity standard of two data blocks is relatively low, so a higher-level computing strategy is added to the multiple computing strategies for subsequent use (the newly added computing strategy has a higher computing level than any of the original computing strategies ); when the bid-winning rate of the calculation strategy with the highest priority is not higher than the preset upper threshold _TH , it is further judged whether the bid-winning rates of all the calculation strategies are less than the preset lower threshold T _L , if all If the bidding success rates of the calculation strategies are all less than the lower threshold _TL , it indicates that these calculation strategies measure the similarity of two data blocks with relatively high standards. Therefore, a calculation strategy with a lower priority is added to the multiple calculation strategies (the newly added calculation strategy is lower than the calculation level of any one of the existing calculation strategies), if not all calculation strategies have a bid winning rate less than the lower threshold T _L , then keep the multiple calculation strategies unchanged; Figure 7 shows the corresponding process schematic diagram.

In the method described in Figure 1, the terminal compresses the data in units of data blocks, and first judges whether there is a reference data block with a high similarity to the data block to be compressed when compressing, and if it exists, refer to the high similarity Compress the data block to be compressed, if it does not exist, judge whether there is a reference data block with a lower similarity with the data block to be compressed, and if it exists, refer to the reference data block with a lower similarity to the data block to be compressed Compressed data block compression; that is to say, the embodiment of the present invention selects reference data blocks from high to low through multi-level similarity standards, which generally improves the compression rate during data compression and saves storage space.

The method of the embodiment of the present invention has been described in detail above. In order to facilitate better implementation of the above solution of the embodiment of the present invention, correspondingly, the following provides the device of the embodiment of the present invention.

Please refer to FIG. 8. FIG. 8 is a schematic structural diagram of a terminal 80 provided by an embodiment of the present invention. The terminal 80 may include a first calculation unit 801, a first judgment unit 802, a first compression unit 803, and a second calculation unit 804. . A second judging unit 805 and a second compressing unit 806, wherein the detailed description of each unit is as follows.

The first calculation unit 801 is configured to calculate a first eigenvalue of the data block to be compressed through a first calculation strategy;

The first judging unit 802 is used to judge whether there is a first reference value in the first search library, the first reference value is the same feature value as the first feature value, and the first search library contains N feature values And each eigenvalue is calculated based on the first calculation strategy for the data block corresponding to the eigenvalue, and the N eigenvalues in the first search library correspond to N data blocks one by one, and N is greater than or equal to 1;

The first compression unit 803 is configured to, when the first judging unit 802 judges that the first reference value exists, use a similar compression technique to use the data block corresponding to the first reference value as a reference data block to compress the to-be-compressed data block compression;

The second calculation unit 804 is configured to calculate the second characteristic value of the data block to be compressed through the second calculation strategy when the first judgment unit 802 judges that the first reference value does not exist, and the two data blocks When the similarity is higher than the first similarity threshold, the eigenvalues of the two data blocks calculated by the first calculation strategy are the same; when the similarity of the two data blocks is higher than the second similarity threshold, by the The feature values of the two data blocks calculated by the second calculation strategy are the same, and the first similarity threshold is higher than the second similarity threshold;

The second judging unit 805 is used to judge whether there is a second reference value in the second search library, the second reference value is the same feature value as the second feature value, and the second search library contains N feature values And each eigenvalue is calculated based on the second calculation strategy for the data block corresponding to the eigenvalue, and the N eigenvalues in the second search library correspond to the N data blocks one by one;

The second compression unit 806 is configured to, when the second judging unit 805 judges that the second reference value exists, use a similar compression technique to use the data block corresponding to the second reference value as a reference data block to compress the to-be-compressed Data block compression.

By running the above units, the terminal 80 compresses the data in units of data blocks, and first judges whether there is a reference data block with a higher similarity with the data block to be compressed when compressing, and if it exists, refer to the reference data with a higher similarity block compresses the data block to be compressed, if it does not exist, it is judged whether there is a reference data block with a lower similarity with the data block to be compressed, and if it exists, refer to the reference data block with a lower similarity to the data block to be compressed Compression; that is to say, the embodiment of the present invention selects reference data blocks from high to low through multi-level similarity standards, which generally improves the compression rate during data compression and saves storage space.

In an optional solution, the terminal further includes:

A division unit, configured to divide M data units from the data block to be compressed before the first calculation unit 801 calculates the preset first eigenvalue of the data block to be compressed through the first calculation strategy, and the M data units Each data unit in the data unit corresponds to its own initial reference value, and M is greater than or equal to 1;

The first calculation unit 801 is specifically configured to substitute the initial reference values of at least two data units in the M data units into preset P filter functions to calculate the first characteristic value of the data block to be compressed, P is greater than or equal to 2;

The second calculation unit 804 is specifically configured to substitute the initial reference values of at least two data units in the M data units into preset Q filter functions to calculate the second characteristic value of the data block to be compressed, The P filter functions include the Q filter functions.

In yet another optional solution, the terminal further includes:

An adding unit, configured to add the first feature value to the first lookup library, and add the second feature value to the second lookup library, in the first lookup library, The data block corresponding to the first characteristic value is the data block to be compressed, and the data block corresponding to the second characteristic value in the second search database is the data block to be compressed.

In yet another optional solution, the terminal further includes:

A deletion unit, configured to delete the feature values that exist in the first search library for a time exceeding a preset time threshold from the first search library, and delete the feature values that exist in the second search library for a time exceeding the preset time threshold. The characteristic value of the preset time threshold is deleted from the second search library.

In yet another optional solution, the terminal further includes:

The first statistical unit is configured to, before the first calculation unit calculates the first eigenvalue of the data block to be compressed by the first calculation strategy, count the number of data blocks calculated by the second calculation strategy for multiple data blocks during the historical compression process eigenvalues, and determine the number of identical eigenvalues in the second search library and the plurality of eigenvalues; the eigenvalues of the plurality of data blocks are not calculated by the first calculation strategy in the historical compression process;

A third calculation unit, configured to calculate a first bid winning rate according to the number of feature values in the plurality of feature values and the number of the same feature values; when the first bid winning rate is higher than a preset first winning bid threshold , if there is a data block to be compressed, triggering the first calculation unit to calculate the first characteristic value of the data block to be compressed by using a first calculation strategy.

In yet another optional solution, the terminal further includes:

The second statistical unit is configured to use the first calculation strategy to analyze multiple data blocks during the statistical history compression process of the terminal before the first calculation unit calculates the second eigenvalue of the data block to be compressed through the second calculation strategy. Compressing the multiple eigenvalues obtained, and determining the number of identical eigenvalues in the first lookup library and the multiple eigenvalues; the multiple data blocks were not calculated by the second calculation strategy during the historical compression process eigenvalues;

A fourth calculation unit, configured to calculate a second bid-winning rate according to the number of feature values in the plurality of feature values and the number of the same feature values; when the second bid-winning rate is lower than a preset second winning threshold and triggering the second calculation unit to calculate the second characteristic value of the data block to be compressed by using a second calculation strategy when the first reference value does not exist.

It should be noted that, in the embodiment of the present invention, the specific implementation of each unit may also refer to the corresponding description of the method embodiment shown in FIG. 1 .

In the terminal 80 described in FIG. 8 , the terminal 80 compresses data in units of data blocks. When compressing, it first judges whether there is a reference data block with a higher similarity with the data block to be compressed. If it exists, refer to the higher The reference data block of similarity is compressed to the data block to be compressed, and if it does not exist, it is judged whether there is a reference data block with a lower similarity with the data block to be compressed, and if it exists, the pair of reference data blocks with lower similarity is referred to The data block to be compressed is compressed; that is to say, the embodiment of the present invention selects the reference data block from high to low through multi-level similarity standards, which improves the compression rate of data compression as a whole and saves storage space.

Please refer to FIG. 9. FIG. 9 is a terminal 90 provided by an embodiment of the present invention. The terminal 90 includes a processor 901 and a memory 902, and the processor 901 and the memory 902 are connected to each other through a bus.

Memory 902 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), or portable read-only memory (CD-ROM). Memory 902 is used for related instructions and data.

The processor 901 may be one or more central processing units (English: Central Processing Unit, CPU for short). When the processor 901 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

The processor 901 in the terminal 90 is used to read the program code stored in the memory 902, and perform the following operations:

Calculate the first eigenvalue of the data block to be compressed through the first calculation strategy; determine whether there is a first reference value in the first search library, the first reference value is the same eigenvalue as the first eigenvalue, and the first reference value is the same as the first eigenvalue. The first search library contains N eigenvalues, and each eigenvalue is calculated based on the first calculation strategy for the data block corresponding to the eigenvalue, and the N eigenvalues in the first search library are one by one Corresponding to N data blocks, N is greater than or equal to 1; if there is the first reference value, the data block corresponding to the first reference value is used as a reference data block to compress the data block to be compressed through a similar compression technique; if If the first reference value does not exist, the second characteristic value of the data block to be compressed is calculated through the second calculation strategy, and when the similarity between the two data blocks is higher than the first similarity threshold, the first calculation strategy is used to calculate the second characteristic value of the data block to be compressed. The calculated eigenvalues of the two data blocks are the same; when the similarity of the two data blocks is higher than a second similarity threshold, the eigenvalues of the two data blocks calculated through the second calculation strategy Same, the first similarity threshold is higher than the second similarity threshold; judging whether there is a second reference value in the second search library, the second reference value is the same feature value as the second feature value, so The second search library includes N eigenvalues, and each eigenvalue is calculated based on the second calculation strategy for the data block corresponding to the eigenvalue, and the N eigenvalues in the second search library are one One corresponds to the N data blocks; if there is the second reference value, compress the data block to be compressed by using the data block corresponding to the second reference value as a reference data block through a similar compression technique.

By performing the above operations, the terminal 90 compresses the data in units of data blocks. When compressing, it first judges whether there is a reference data block with a high similarity with the data block to be compressed, and if it exists, refer to the reference data with a high similarity. block compresses the data block to be compressed, if it does not exist, it is judged whether there is a reference data block with a lower similarity with the data block to be compressed, and if it exists, refer to the reference data block with a lower similarity to the data block to be compressed Compression; that is to say, the embodiment of the present invention selects reference data blocks from high to low through multi-level similarity standards, which generally improves the compression rate during data compression and saves storage space.

In an optional solution, before the processor 901 calculates the preset first eigenvalue of the data block to be compressed through the first calculation strategy, it is further configured to: divide M data units from the data block to be compressed , each of the M data units corresponds to its own initial reference value, and M is greater than or equal to 1; the processor 901 calculates the first characteristic value of the preset data block to be compressed through a first calculation strategy, specifically It is: substituting the initial reference values of at least two data units in the M data units into preset P filter functions to calculate the first characteristic value of the data block to be compressed, where P is greater than or equal to 2; the processing The unit 901 calculates the second characteristic value of the data block to be compressed through the second calculation strategy, specifically: substituting the initial reference values of at least two data units in the M data units into the preset Q filter functions Calculating a second eigenvalue of the data block to be compressed, where the P filter functions include the Q filter functions.

In yet another optional solution, after the data block to be compressed is compressed, the processor 901 is further configured to: add the first feature value to the first lookup library, and add the The second feature value is added to the second lookup library, the data block corresponding to the first feature value in the first lookup library is the data block to be compressed, and the data block in the second lookup library is The data block corresponding to the second characteristic value is the data block to be compressed.

In yet another optional solution, the processor 901 is further configured to: delete a feature value that exists in the first search library for a time exceeding a preset time threshold from the first search library, and Deleting feature values that have existed in the second search library for a time exceeding the preset time threshold from the second search library.

In yet another optional solution, before the processor 901 calculates the first eigenvalue of the data block to be compressed through the first calculation strategy, it is also used to: use the second calculation strategy to perform statistics on multiple data during historical compression. block the calculated multiple eigenvalues, and determine the number of identical eigenvalues in the second lookup library and the multiple eigenvalues; the multiple data are not calculated by the first calculation strategy during the historical compression process The feature value of the block; calculate the first bid winning rate according to the number of feature values in the plurality of feature values and the number of the same feature value; when the first bid winning rate is higher than the preset first winning threshold, if If there is a data block to be compressed, the operation of calculating the first characteristic value of the data block to be compressed by using the first calculation strategy is performed.

In yet another optional solution, before the processor 901 calculates the second eigenvalue of the data block to be compressed through the second calculation strategy, it is further configured to: use the first calculation strategy to compare multiple A plurality of eigenvalues obtained by compressing data blocks, and determining the number of identical eigenvalues in the first lookup library and the plurality of eigenvalues; the plurality of eigenvalues were not calculated by the second calculation strategy in the historical compression process feature value of a data block; calculate a second bid winning rate according to the number of feature values in the plurality of feature values and the number of the same feature value; when the second bid winning rate is lower than the preset second winning threshold , performing the operation of calculating a second characteristic value of the data block to be compressed by using a second calculation strategy if the first reference value does not exist.

It should be noted that, in the embodiment of the present invention, the specific implementation of each unit may also refer to the corresponding description of the method embodiment shown in FIG. 2 .

In the terminal described in FIG. 9 , the terminal 90 compresses the data in units of data blocks. When compressing, it first judges whether there is a reference data block with a high similarity with the data block to be compressed. degree of reference data block to compress the data block to be compressed, if it does not exist, judge whether there is a reference data block with a lower similarity to the data block to be compressed, and if it exists, refer to the reference data block with a lower similarity to the data block to be compressed Compression of data blocks to be compressed; that is to say, the embodiment of the present invention selects reference data blocks from high to low through multi-level similarity standards, which generally improves the compression rate of data compression and saves storage space.

To sum up, by implementing the embodiment of the present invention, the terminal compresses data in units of data blocks, and first judges whether there is a reference data block with a high similarity with the data block to be compressed when compressing, and if so, refers to the reference data block. A reference data block with a high similarity is compressed to the data block to be compressed, and if it does not exist, it is judged whether there is a reference data block with a lower similarity with the data block to be compressed, and if it exists, the reference data block with a lower similarity is referred to The data block to be compressed is compressed; that is to say, the embodiment of the present invention selects the reference data block from high to low through multi-level similarity standards, which improves the compression rate of data compression as a whole and saves storage space.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. During execution, it may include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium includes various media capable of storing program codes such as ROM, RAM, magnetic disk or optical disk.

The above embodiments only disclose the preferred embodiments of the present invention, and cannot limit the scope of rights of the present invention with this. Those of ordinary skill in the art can understand the whole or part of the process of realizing the above embodiments, and make according to the claims of the present invention Equivalent changes still belong to the scope covered by the invention.

Claims (12)

1. A data compression method, characterized in that, comprising: The terminal calculates the first eigenvalue of the data block to be compressed by using the first calculation strategy; The terminal judges whether there is a first reference value in the first search library, the first reference value is the same feature value as the first feature value, the first search library contains N feature values and each feature The value is calculated based on the first calculation strategy for the data block corresponding to the feature value, and the N feature values in the first search library correspond to N data blocks one by one, and N is greater than or equal to 1; If the first reference value exists, the terminal compresses the data block to be compressed using a similar compression technique using the data block corresponding to the first reference value as a reference data block; If the first reference value does not exist, the terminal calculates the second characteristic value of the data block to be compressed through the second calculation strategy, and when the similarity between the two data blocks is higher than the first similarity threshold, the The feature values of the two data blocks calculated by the first calculation strategy are the same; when the similarity of the two data blocks is higher than the second similarity threshold, the two data blocks calculated by the second calculation strategy The feature values of the blocks are the same, and the first similarity threshold is higher than the second similarity threshold; The terminal judges whether there is a second reference value in the second search library, the second reference value is the same feature value as the second feature value, the second search library contains N feature values and each feature The value is calculated based on the second calculation strategy for the data block corresponding to the feature value, and the N feature values in the second search library correspond to the N data blocks one by one; If the second reference value exists, the terminal compresses the data block to be compressed by using a data block corresponding to the second reference value as a reference data block through a similar compression technique. 2. The method according to claim 1, wherein, before the terminal calculates the preset first eigenvalue of the data block to be compressed through the first calculation strategy, the method further comprises: The terminal divides M data units from the data block to be compressed, each of the M data units corresponds to its own initial reference value, and M is greater than or equal to 1; The terminal calculates the preset first eigenvalue of the data block to be compressed through the first calculation strategy, including: Substituting the initial reference values of at least two data units among the M data units into preset P filter functions to calculate the first characteristic value of the data block to be compressed, where P is greater than or equal to 2; The terminal calculates the second eigenvalue of the to-be-compressed data block through a second calculation strategy, including: Substituting the initial reference values of at least two data units in the M data units into preset Q filter functions to calculate the second characteristic value of the data block to be compressed, the P filter functions include the Q a filter function. 3. The method according to claim 1 or 2, characterized in that, after compressing the data block to be compressed, the method further comprises: The terminal adds the first feature value to the first search library, and adds the second feature value to the second search library, and in the first search library, the first The data block corresponding to the feature value is the data block to be compressed, and the data block corresponding to the second feature value in the second lookup library is the data block to be compressed. 4. method according to claim 3, is characterized in that, described method also comprises: The terminal deletes the feature value that exists in the first search library for a time exceeding a preset time threshold from the first search library, and deletes the feature value that exists in the second search library for a time exceeding the preset time threshold. It is assumed that the feature value of the time threshold is deleted from the second search library. 5. The method according to any one of claims 1 to 4, wherein before the terminal calculates the first eigenvalue of the data block to be compressed through the first calculation strategy, the method further comprises: A plurality of eigenvalues calculated by the second calculation strategy for a plurality of data blocks in the terminal statistical history compression process, and determining the number of identical eigenvalues in the second search library and the plurality of eigenvalues; history The eigenvalues of the plurality of data blocks are not calculated by the first calculation strategy during the compression process; The terminal calculates a first bid-winning rate according to the number of eigenvalues in the plurality of eigenvalues and the number of the same eigenvalues; When the first successful bid rate is higher than the preset first successful bid threshold, if there is a data block to be compressed, the terminal calculates a first characteristic value of the data block to be compressed by using a first calculation strategy. 6. The method according to any one of claims 1 to 4, wherein before the terminal calculates the second characteristic value of the data block to be compressed through the second calculation strategy, the method further comprises: The terminal counts the multiple eigenvalues obtained by compressing multiple data blocks through the first calculation strategy in the history compression process, and determines the number of identical eigenvalues in the first search library and the multiple eigenvalues; history During the compression process, the feature values of the plurality of data blocks are not calculated by the second calculation strategy; The terminal calculates a second bid-winning rate according to the number of feature values in the plurality of feature values and the number of the same feature values; When the second successful bid rate is lower than the preset second successful bid threshold, the terminal calculates the second value of the to-be-compressed data block through a second calculation strategy if the first reference value does not exist. Steps for eigenvalues. 7. A terminal, characterized in that, comprising: A first calculation unit, configured to calculate a first eigenvalue of the data block to be compressed through a first calculation strategy; A first judging unit, configured to judge whether there is a first reference value in the first search library, the first reference value is the same feature value as the first feature value, and the first search library contains N feature values And each eigenvalue is calculated based on the first calculation strategy for the data block corresponding to the eigenvalue, and the N eigenvalues in the first search library correspond to N data blocks one by one, and N is greater than or equal to 1; The first compression unit is configured to, when the first judgment unit judges that the first reference value exists, use a similar compression technique to use the data block corresponding to the first reference value as a reference data block to compress the data to be compressed block compression; The second calculation unit is used to calculate the second characteristic value of the data block to be compressed through the second calculation strategy when the first judgment unit judges that the first reference value does not exist, and the similarity of the two data blocks When the degree of similarity is higher than the first similarity threshold, the feature values of the two data blocks calculated by the first calculation strategy are the same; when the similarity of the two data blocks is higher than the second similarity threshold, by the The feature values of the two data blocks calculated by the second calculation strategy are the same, and the first similarity threshold is higher than the second similarity threshold; The second judging unit is configured to judge whether there is a second reference value in the second search library, the second reference value is the same feature value as the second feature value, and the second search library contains N feature values And each eigenvalue is calculated based on the second calculation strategy for the data block corresponding to the eigenvalue, and the N eigenvalues in the second search library correspond to the N data blocks one by one; The second compression unit is configured to use the data block corresponding to the second reference value as a reference data block to compress the data to be compressed by similar compression technology when the second judgment unit determines that the second reference value exists. block compression. 8. The terminal according to claim 7, further comprising: A division unit, configured to divide M data units from the data block to be compressed before the first calculation unit calculates the preset first eigenvalue of the data block to be compressed through the first calculation strategy, and the M data units Each data unit in the unit corresponds to its own initial reference value, and M is greater than or equal to 1; The first calculation unit is specifically configured to substitute the initial reference values of at least two data units in the M data units into preset P filter functions to calculate the first characteristic value of the data block to be compressed, P greater than or equal to 2; The second calculation unit is specifically configured to substitute the initial reference values of at least two data units in the M data units into preset Q filter functions to calculate the second characteristic value of the data block to be compressed, so The P filter functions include the Q filter functions. 9. The terminal according to claim 7 or 8, wherein the terminal further comprises: An adding unit, configured to add the first feature value to the first lookup library, and add the second feature value to the second lookup library, in the first lookup library, The data block corresponding to the first characteristic value is the data block to be compressed, and the data block corresponding to the second characteristic value in the second search database is the data block to be compressed. 10. The terminal according to claim 9, further comprising: A deletion unit, configured to delete the feature values that exist in the first search library for a time exceeding a preset time threshold from the first search library, and delete the feature values that exist in the second search library for a time exceeding the preset time threshold. The characteristic value of the preset time threshold is deleted from the second search library. 11. The terminal according to any one of claims 7-10, wherein the terminal further comprises: The first statistical unit is configured to, before the first calculation unit calculates the first eigenvalue of the data block to be compressed by the first calculation strategy, count the number of data blocks calculated by the second calculation strategy for multiple data blocks during the historical compression process eigenvalues, and determine the number of identical eigenvalues in the second search library and the plurality of eigenvalues; the eigenvalues of the plurality of data blocks are not calculated by the first calculation strategy in the historical compression process; A third calculation unit, configured to calculate a first bid winning rate according to the number of feature values in the plurality of feature values and the number of the same feature values; when the first bid winning rate is higher than a preset first winning bid threshold , if there is a data block to be compressed, triggering the first calculation unit to calculate the first characteristic value of the data block to be compressed by using a first calculation strategy. 12. The terminal according to any one of claims 7-10, wherein the terminal further comprises: The second statistical unit is configured to use the first calculation strategy to analyze multiple data blocks during the statistical history compression process of the terminal before the first calculation unit calculates the second eigenvalue of the data block to be compressed through the second calculation strategy. Compressing the multiple eigenvalues obtained, and determining the number of identical eigenvalues in the first lookup library and the multiple eigenvalues; the multiple data blocks were not calculated by the second calculation strategy during the historical compression process eigenvalues; A fourth calculation unit, configured to calculate a second bid-winning rate according to the number of feature values in the plurality of feature values and the number of the same feature values; when the second bid-winning rate is lower than a preset second winning threshold and triggering the second calculation unit to calculate the second characteristic value of the data block to be compressed by using a second calculation strategy when the first reference value does not exist.