風結ぶ言葉たち

System Evaluation is a comprehensive assessment of system value. And value is usually understood as the recognition of the evaluation subject's satisfaction with the evaluation object based on its utility perspective. It is closely related to the evaluation subject and the environmental conditions in which the evaluation object is located.

Matrix Method#

Matrix of Association is a common comprehensive evaluation method. It uses a matrix to represent the relationship between the alternative solutions and the evaluation values of specific indicators.

• $ A_1 , ⋯ , A_m $ are $ m $ alternative solutions for a certain evaluation object;

• $ X_1 , ⋯ , X_n $ are $ n $ evaluation indicators or evaluation items for evaluating alternative solutions;

• $ W_1 , ⋯ , W_n $ are the weights of $ n $ evaluation indicators;

• $ V_{i1} , ⋯ , V_{mn} $ are the value evaluation quantities of the $ i $th alternative solution $ A_i $ for the $ j $th indicator $ X_j $ ($ 1 \leq i \leq m, 1 \leq j \leq n $).

The following will combine an example to accept two methods of determining weights and value evaluation quantities.

A company produces a popular product and plans the following three production schemes:

$ A_1 $: Design a new production line by ourselves;

$ A_2 $: Import a highly automated production line from abroad;

$ A_3 $: Refurbish a production line based on existing equipment.

After discussions by authoritative departments and experts, five evaluation indicators are determined: expected profit, product yield rate, market share, investment cost, and product appearance.

According to the predictions and estimates of professionals, the results of the five evaluation items after implementing these three schemes are shown below.

Pairwise Comparison Method#

The basic steps of this method are as follows:

1. Compare the evaluation indicators of each alternative solution pairwise, and give higher scores to relatively important indicators to obtain the weights $ W_j $ of each evaluation item;

2. According to the evaluation scale given by the evaluation subject, evaluate each alternative solution one by one under different evaluation indicators to obtain the corresponding evaluation values;

3. Calculate the weighted sum to obtain the comprehensive evaluation value.

For the example, we first need to use the pairwise comparison method to calculate the weights of each evaluation indicator. As shown in the figure below, the expected profit is more important than the product yield rate, so the former gets a score of 1 and the latter gets a score of 0. Finally, the weights are calculated based on the cumulative scores of each evaluation item.

Then, the evaluation scale is determined by the evaluation subject, as shown in the figure below, to unify the actual results of the schemes under different indicators for weighted summation.

Based on the previous two figures, the comprehensive evaluation of each alternative solution is shown in the figure below. It can be seen that $ V_2 \gt V_1 \gt V_3 $, so $ A_2 \gt A_1 \gt A_3 $.

Gurin Method#

When the importance of each evaluation item can be quantitatively estimated, the Gurin method can be used to determine the indicator weights and solution value evaluation quantities.

The basic steps of this method are as follows:

1. Determine the importance of the evaluation item $ R_j $;

2. Normalize $ R_j $ with $ K_j $ as the unit and obtain the weights $ W_j $;

3. Calculate the evaluation values of alternative solutions using the same method as the association matrix;

4. Calculate the weighted sum to obtain the comprehensive evaluation value.

For the example, we first need to determine the importance of the evaluation indicators $ R_j $.

Then, $ R_j $ is normalized with $ K_j $ as the unit, and the $ K $ value of the last evaluation indicator is set to 1. The $ K $ values of other items are calculated from bottom to top.

After completing the above steps, $ K_j $ is normalized by adding up the numbers in each column of $ K_j $ to obtain $ \sum K_j $. Divide $ K_j $ by $ \sum K_j $ as shown in the figure to obtain $ V_{ij} $.

In the calculation of the fourth step, since the smaller the investment cost, the better, its proportion is taken as the reciprocal, that is, $ R_{14} = 180/110 = 1.636 $, $ R_{24} = 50/180 = 0.279 $.

It can be seen that $ V_2 \gt V_1 \gt V_3 $, so $ A_2 \gt A_1 \gt A_3 $.

Simplified method:

Divide the value evaluation quantities of each alternative solution by the value in the last row;

Then divide the values in each row by the sum of the values in each column;

Normalize to quickly obtain the weighted sum $ V_i $ to obtain the result.

Analytic Hierarchy Process#

The Analytic Hierarchy Process (AHP) can be modeled in four steps:

1. Establish a hierarchical structure model to describe the relationship between elements in the evaluation system;

2. Create pairwise comparison matrices to compare each element in the same layer and construct the matrix;

3. Calculate the relative weights of the elements, analyze the importance of each element in the same layer to the upper-level criteria, calculate the relative weights of each element through the judgment matrix, and perform consistency checks;

4. Calculate the synthesis (overall) weights of each element to the overall goal of the system and rank each candidate solution.

In this example, the AHP is used to determine the weights of $ C_1 $ to $ C_6 $ relative to the overall goal $ A $.

A university needs to scientifically evaluate a series of research topics and select research topics. The indicator system for selecting research topics is shown in the figure below. How to make a scientific selection among many candidate evaluation topics? Please use the Analytic Hierarchy Process (AHP) to determine the weights of $ C_1 $ to $ C_6 $ relative to the overall goal $ A $.

First, calculate the weights $ W_i $ and normalize them to obtain the following results.

Then, perform the calculations, and the final results are shown in the figure below. By comparing the evaluation values, the priority order is determined as $ C_2 > C_1 > C_3 $.

Fuzzy Comprehensive Evaluation Method#

The Fuzzy Comprehensive Evaluation Method is based on fuzzy mathematics and uses the principle of fuzzy relation synthesis to quantify factors that are difficult to quantify and make judgments about the membership level of multiple factors in evaluating things.

The modeling process of the Fuzzy Comprehensive Evaluation Method can be roughly divided into three steps:

1. Determine the factor set $ F $ and the evaluation set $ E $. The factor set $ F $ is a set of evaluation items or indicators, generally denoted as $ F = {f_i}, 1 \leq i \leq n $. The evaluation set is a set of evaluation levels, generally denoted as $ E = {e_j}, 1 \leq j \leq m $.

2. Statistically determine the membership degree vector of single-factor evaluation and form the membership degree matrix $ R $. The membership degree is the most basic and important concept in fuzzy comprehensive judgment. The membership degree $ R_{ij} $ refers to the possibility size of multiple evaluation subjects making a $ e_j $ evaluation of a certain evaluation object in terms of $ f_i $.

3. Determine the weight vector $ W_F $, which is the weight or coefficient vector of evaluation items or indicators. In addition, there may be numerical results (standard satisfaction degree vector) $ W'_E $ or weights $ W_E $ obtained from the evaluation set.

In this example, the Fuzzy Comprehensive Evaluation Method is used to determine the priorities of three refrigerator models $ A_1 $, $ A_2 $, and $ A_3 $.

Before purchasing a refrigerator, a person wants to determine the priority order of models $ A_1 $, $ A_2 $, and $ A_3 $ by using the Fuzzy Comprehensive Evaluation Method. Five family members evaluate the models using the method. The evaluation items (factors) include price $ f_1 $, quality $ f_2 $, and appearance $ f_3 $. The corresponding weights are obtained from the judgment matrix in Figure 1. The evaluation scale is divided into three levels, for example, the price is divided into low (0.3), medium (0.2), and high (0.1). The evaluation results are shown in Figure 2. Please calculate the priorities of these three refrigerators and rank them.

First, calculate the weights $ W_i $ and normalize them to obtain the following results.

Then, perform the calculations, and the final results are shown in the figure below. By comparing the evaluation values, the priority order is determined as $ A_2 > A_1 > A_3 $.

This article is also updated to xLog by Mix Space
The original link is https://nishikori.tech/posts/tech/Systems-Evaluation-Method