Simple method to get criteria weights for decision making

“Calculating multi-attribute criteria weights with Julia”

Julia
R
Python
Data science
Programming
Decision intelligence
Scientific computing
Author
Affiliation

Tomás Aragón

California Department of Public Health

Published

March 31, 2023

1 Introduction

At the California Department of Public Health, we promote decision intelligence (DI)—improving how we make team decisions to find, select, and solve problems and to improve outcomes. A foundational component of DI is decision quality—the six requirements of good decision making. At a minimum, a decision quality checklist (DQ) (Table 1) improves the quality of decisions at any stage of problem solving. A good decision is only as strong as its weakest link. (To learn more about DQ watch 3.5min video from the Decision Education Foundation.)

Table 1: Key questions to answer for decision quality (DQ)
No. Requirement Key question to ask
1 Frame Are we clear on the problem we are solving?
2 Values Have we identified what we truly want?
3 Alternatives Do we have a good set of alternatives?
4 Information Have we gathered the relevant information?
5 Reasoning How will we evaluate alternatives to find the one that gets us the most of what what we truly want?
6 Commitment Are we committed to follow through on our choice.

Making higher quality decisions as a group can be challenging. Our brains prefer intuitive decision making. Deliberations are cognitively exhausting, especially in the face of competing objectives, uncertainty, and time constraints. That is why, unfortunately, decision-making is often deferred to an authority figure. It’s better to embrace team decision making but practice with simple tools and methods that promote group engagement, deliberation, and decision quality.

2 Making single and multi-attribute decisions

2.1 Single-attribute decision

Suppose our team needs to select from several alternatives and it will be based on a single attribute; for example, choosing one of five available colors to paint our office. That attribute—color—becomes our single criterion and we apply the ranking method illustrated below. In this example the single attribute (color) represents the alternatives (five colors to choose from).

2.2 Multi-attribute decision

However, more commonly, based on our frame (values, objectives, and key results), our team needs to select from among several alternatives, each of which has mutliple attributes. Selection examples include buying a smartphone, hiring an employee, or selecting a vendor. As individuals and as a group, we have preferences for these attributes based on what we know (information) and what consequences and impacts we care about (prospects). We need sound reasoning but it will not require a complex decision analysis. Engaging the team in fair and transparent deliberative decision making builds trust, consensus, and commitment.

Our main task is to (a) develop criteria based on these attributes, (b) weight the criteria based on importance to us (preference), and (c) rate the alternatives based on the criteria and evidence (data, community, experts, analysis). Ideally, the criteria should be weighted without any knowledge of the alternatives. This is to prevent evaluators from biasing the criteria weights in favor of their favorite alternative.

In this blog posting I show how to weight criteria using a simple ranking method. In a future blog post, I will show how to apply weighted criteria to rank and select alternatives. This first step, weighting criteria, is very powerful and practical. We will use a trivial example to nail down the concepts.

Now, suppose we wish to buy a car and our choices are a Honda Civic, and Subaru Impreza, or Toyota Corolla. We have data on the following attributes: safety (S), mileage (M), reliability (R), color (C), price (P), and resale value (V). Table 2 summarizes the DQ requirements for buying our car.

Table 2: Decision quality requirements for buying a car
No. Requirement Key question to ask Answer
1 Frame Are we clear on the problem we are solving? Need personal transportion.
2 Values Have we identified what we truly want? Car within my budget.
3 Alternatives Do we have a good set of alternatives? Civic, Corolla, or Impreza
4 Information Have we gathered the relevant information? Color, mileage, price, reliability, safety, resale value
5 Reasoning How will we evaluate alternatives to find the one that gets us the most of what what we truly want? Ranking algorithm using weight calculations
6 Commitment Are we committed to follow through on our choice. Yes, my spouse approves.

3 Calculating criteria weights — the easy way

Group deliberative decision-making is cognitively exhausting. So, anything you can do to make the process easier will keep team members engaged. Do not let “perfection become the enemy of the good”. The easiest way to generate criteria weights from a team of evaluators is to use a rank ordinal method [1].

Give evaluators small pieces of paper with one criterion printed on on each. If we have five criteria, they get five small pieces of paper. Have them rank them from top to bottom. Once they have ranked them, tape their ranking onto an 8.5in x 11in paper and hand to the facilitator. This ranking is entered into the computer for analysis (see below).

3.1 Ratio ordinal method in Julia

I will demonstrate this method using the Julia language. This method can also be implemented using R, Python, or Microsoft Excel.

For rating the cars we have six criteria (attributes) for which we need to calculate weights:

  1. Color (C)
  2. Mileage (M)
  3. Price (P)
  4. Reliability (R)
  5. Safety (S)
  6. Value, resale (V)

We have five evaluators that will rank the criteria based on their knowledge, experience, expertise, and wisdom. It is much better for them to rank the criteria independently and without thinking about specific cars, otherwise they may game (bias) the weighting.

Here are the steps:

  1. Select a ranking method to calculate weights for a specific number of criteria, in this case we have six criteria. We will write and use a Julia function that implements the SR method from [1].
  2. Have each evaluator independently rank the criteria.
  3. Use Julia to calculate the final criteria weights. We will use the split-apply-combine workflow that I introduced in a previous blog post and book review [2].

3.1.1 Step 1: The SR method for generating criteria weights

Here is the formula1 where \(N\) is the number of criteria, and \(w_i^{SR}\) is the weight for the \(i^{th}\) criterion [1].

\[ w_i^{SR} = \frac{1/i + \frac{N+1-i}{N}}{\sum_{j=1}^N (1/i + \frac{N+1-i}{N})} \]

using DataFrames
using StatsBase
function calculate_rank_weights(n::Int64)
    num = zeros(n)
    for i in 1:n
        num[i] = (1/i) + ( (n + 1 - i) / n )
    end
    wi = num / sum(num)
    return wi
end
wi = calculate_rank_weights(6)
6-element Vector{Float64}:
 0.3361344537815126
 0.22408963585434175
 0.1680672268907563
 0.12605042016806722
 0.0896358543417367
 0.05602240896358543

The weights sum to 1, as expected.

round(sum(wi))
1.0

3.2 Step 2: Evaluators rank the criteria

Five evaluators rank the criteria based on their expertise.

eval1 = ["Mileage", "Color", "Price", "Safety", "Reliability", "Value"]
eval2 = ["Mileage", "Color", "Safety", "Reliability", "Value", "Price"]
eval3 = ["Color", "Value", "Price", "Mileage", "Reliability", "Safety"]
eval4 = ["Mileage", "Value", "Color", "Safety", "Reliability", "Price"]
eval5 = ["Safety", "Price", "Color", "Reliability", "Mileage", "Value"]
6-element Vector{String}:
 "Safety"
 "Price"
 "Color"
 "Reliability"
 "Mileage"
 "Value"

Next, we organize the evaluator criteria rankings and the SR method criteria weights into a data frame with three columns:

  • evaluators (optional)
  • evaluator_rankings (required)
  • weights (required)
ne = 5; # number of evaluators
nc = 6; # number for criteria
evaluators = repeat(["eval" .* string.(1:ne)...], inner=repeat([nc]))
evaluator_rankings = vcat( eval1, eval2, eval3, eval4, eval5 )
weights = repeat( wi, ne )
df = DataFrame(
    hcat(evaluators, evaluator_rankings, weights), 
    ["evaluator", "criteria", "weight"]
    )
30×3 DataFrame
5 rows omitted
Row evaluator criteria weight
Any Any Any
1 eval1 Mileage 0.336134
2 eval1 Color 0.22409
3 eval1 Price 0.168067
4 eval1 Safety 0.12605
5 eval1 Reliability 0.0896359
6 eval1 Value 0.0560224
7 eval2 Mileage 0.336134
8 eval2 Color 0.22409
9 eval2 Safety 0.168067
10 eval2 Reliability 0.12605
11 eval2 Value 0.0896359
12 eval2 Price 0.0560224
13 eval3 Color 0.336134
⋮ ⋮ ⋮ ⋮
19 eval4 Mileage 0.336134
20 eval4 Value 0.22409
21 eval4 Color 0.168067
22 eval4 Safety 0.12605
23 eval4 Reliability 0.0896359
24 eval4 Price 0.0560224
25 eval5 Safety 0.336134
26 eval5 Price 0.22409
27 eval5 Color 0.168067
28 eval5 Reliability 0.12605
29 eval5 Mileage 0.0896359
30 eval5 Value 0.0560224

3.3 Step 3: Calculate mean criteria weights

Calculate mean criteria weights using split-apply-combine workflow. In other words, stratifying by criterion, what is the mean weight for each criterion (attribute)?

## split
gdf = groupby(df, :criteria)

GroupedDataFrame with 6 groups based on key: criteria

First Group (5 rows): criteria = "Mileage"
Row evaluator criteria weight
Any Any Any
1 eval1 Mileage 0.336134
2 eval2 Mileage 0.336134
3 eval3 Mileage 0.12605
4 eval4 Mileage 0.336134
5 eval5 Mileage 0.0896359

⋮

Last Group (5 rows): criteria = "Value"
Row evaluator criteria weight
Any Any Any
1 eval1 Value 0.0560224
2 eval2 Value 0.0896359
3 eval3 Value 0.22409
4 eval4 Value 0.22409
5 eval5 Value 0.0560224 
## combine and apply 
crit_weights = combine(gdf, :weight => mean)
sort!(crit_weights, :weight_mean, rev = true)
6×2 DataFrame
Row criteria weight_mean
Any Float64
1 Mileage 0.244818
2 Color 0.22409
3 Safety 0.162465
4 Price 0.134454
5 Value 0.129972
6 Reliability 0.104202

These are the final criteria weights and, as expected, they sum to 1.

round(sum(crit_weights.weight_mean))
1.0

In this example, mileage (M) has the highest weight (24.5%) and reliability (R) has the lowest weight (10.4%).

In a future blog, I will show how to score the alternatives (Honda Civic, Toyota Corolla, and Subaru Impreza) with these weighted criteria, and then making a final ranking of which car to select.

4 Appendix

4.1 Function to automate calculating mean weights

We created the calculate_rank_weights function to calculate the weights for \(n\) criteria using the SR method. Then we used Julia to calculate the mean weights for five evaluators.

Now we create a final function to automate calculating the mean weights. This function will handle an arbitrary number of vectors with rankings, or a matrix created from those vectors beforehand. Notice that this new function will call our previous calculate_rank_weights function.

function calculate_mean_weights(x...)
    x = hcat(x...)
    ncrit, neval = size(x, 1), size(x, 2)
    wts_i = calculate_rank_weights(ncrit)
    weights = repeat( wts_i, neval )
    evaluators = repeat( 
        ["eval" .* string.(1:neval)...], 
        inner = repeat([ncrit])
    )
    evalvator_rankings = reshape(x, ncrit * neval)
    df = DataFrame( 
        hcat(evaluators, evaluator_rankings, weights), 
        ["evaluator", "criteria", "weight"] 
    )
    gdf = groupby(df, :criteria)
    mean_wts = sort!(
        combine(gdf, :weight => mean), 
        :weight_mean, rev = true
    )   
    return (
        weights = mean_wts,
        data = df
    )
end
calculate_mean_weights (generic function with 1 method)

Here we test passing vectors as arguments to the calculate_mean_weights function.

r1 = calculate_mean_weights(eval1, eval2, eval3, eval4, eval5);

The results are saved in r1 and the semi-colon (;) suppresses the output. We use typeof function to evaluate the type of r1.

typeof(r1)
@NamedTuple{weights::DataFrame, data::DataFrame}

We see that r1 is a NamedTuple and it contains two data frames named weights and data. We can index each separately.

r1.weights
6×2 DataFrame
Row criteria weight_mean
Any Float64
1 Mileage 0.244818
2 Color 0.22409
3 Safety 0.162465
4 Price 0.134454
5 Value 0.129972
6 Reliability 0.104202 
r1.data
30×3 DataFrame
5 rows omitted
Row evaluator criteria weight
Any Any Any
1 eval1 Mileage 0.336134
2 eval1 Color 0.22409
3 eval1 Price 0.168067
4 eval1 Safety 0.12605
5 eval1 Reliability 0.0896359
6 eval1 Value 0.0560224
7 eval2 Mileage 0.336134
8 eval2 Color 0.22409
9 eval2 Safety 0.168067
10 eval2 Reliability 0.12605
11 eval2 Value 0.0896359
12 eval2 Price 0.0560224
13 eval3 Color 0.336134
⋮ ⋮ ⋮ ⋮
19 eval4 Mileage 0.336134
20 eval4 Value 0.22409
21 eval4 Color 0.168067
22 eval4 Safety 0.12605
23 eval4 Reliability 0.0896359
24 eval4 Price 0.0560224
25 eval5 Safety 0.336134
26 eval5 Price 0.22409
27 eval5 Color 0.168067
28 eval5 Reliability 0.12605
29 eval5 Mileage 0.0896359
30 eval5 Value 0.0560224

Next, we create the matrix of evaluator data and then pass it to the calculate_mean_weights function.

eval_rankings_tab = hcat(eval1, eval2, eval3, eval4, eval5)
r2 = calculate_mean_weights(eval_rankings_tab);

The results are saved in r and the semi-colon (;) suppresses the output. We use typeof function to evaluate the type of r2.

typeof(r2)
@NamedTuple{weights::DataFrame, data::DataFrame}

We see that r2 is a NamedTuple and it contains two data frames named weights and data. We can index each separately.

r2.weights
6×2 DataFrame
Row criteria weight_mean
Any Float64
1 Mileage 0.244818
2 Color 0.22409
3 Safety 0.162465
4 Price 0.134454
5 Value 0.129972
6 Reliability 0.104202 
r2.data
30×3 DataFrame
5 rows omitted
Row evaluator criteria weight
Any Any Any
1 eval1 Mileage 0.336134
2 eval1 Color 0.22409
3 eval1 Price 0.168067
4 eval1 Safety 0.12605
5 eval1 Reliability 0.0896359
6 eval1 Value 0.0560224
7 eval2 Mileage 0.336134
8 eval2 Color 0.22409
9 eval2 Safety 0.168067
10 eval2 Reliability 0.12605
11 eval2 Value 0.0896359
12 eval2 Price 0.0560224
13 eval3 Color 0.336134
⋮ ⋮ ⋮ ⋮
19 eval4 Mileage 0.336134
20 eval4 Value 0.22409
21 eval4 Color 0.168067
22 eval4 Safety 0.12605
23 eval4 Reliability 0.0896359
24 eval4 Price 0.0560224
25 eval5 Safety 0.336134
26 eval5 Price 0.22409
27 eval5 Color 0.168067
28 eval5 Reliability 0.12605
29 eval5 Mileage 0.0896359
30 eval5 Value 0.0560224

The calculate_mean_weights function worked for both vectors or a matrix. This was possible because of the splat operator (...) as in the two lines below.

function calculate_mean_weights(x...)
    x = hcat(x...)

4.2 Review of selected Julia code

Did you know you can use tuples to bind variable names to values?

x, y = 33, 99
(33, 99)
x
33
y
99

This is super handy when we want to swap values between variables in one step.

x, y = y, x
(99, 33)
x
99
y
33

Here is how to prefill arrays:

zeros(2, 3)
2×3 Matrix{Float64}:
 0.0  0.0  0.0
 0.0  0.0  0.0
ones(2, 3)
2×3 Matrix{Float64}:
 1.0  1.0  1.0
 1.0  1.0  1.0
fill(1, (2, 3))
2×3 Matrix{Int64}:
 1  1  1
 1  1  1

The reshape function reshapes an array.

A = Vector(1:16)
16-element Vector{Int64}:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
reshape(A, (4, 4))
4×4 Matrix{Int64}:
 1  5   9  13
 2  6  10  14
 3  7  11  15
 4  8  12  16
reshape(A, 2, :)
2×8 Matrix{Int64}:
 1  3  5  7   9  11  13  15
 2  4  6  8  10  12  14  16

Her is vertical concatenation (vcat): (hcat):

x = [1, 2, 4]
y = [4, 5, 6]
vcat(x, y)
6-element Vector{Int64}:
 1
 2
 4
 4
 5
 6

Here is horizontal concatenation (hcat):

hcat(x, y)
3×2 Matrix{Int64}:
 1  4
 2  5
 4  6

The ! operator signifies mutating (permanently changing) an object:

## combine and apply 
crit_weights = combine(gdf, :weight => mean)
sort!(crit_weights, :weight_mean, rev = true)
6×2 DataFrame
Row criteria weight_mean
Any Float64
1 Mileage 0.244818
2 Color 0.22409
3 Safety 0.162465
4 Price 0.134454
5 Value 0.129972
6 Reliability 0.104202

The repeat function repeat a pattern.

repeat([2, 5], 3)
6-element Vector{Int64}:
 2
 5
 2
 5
 2
 5
## equivalent to above
repeat([2, 5], outer=repeat([3]))
6-element Vector{Int64}:
 2
 5
 2
 5
 2
 5
repeat([2, 5], inner=repeat([3]))
6-element Vector{Int64}:
 2
 2
 2
 5
 5
 5

Generating array of strings. Note how the splat (...) operator works.

["eval" .* string.(1:3)]
1-element Vector{Vector{String}}:
 ["eval1", "eval2", "eval3"]
["eval" .* string.(1:3)...]
3-element Vector{String}:
 "eval1"
 "eval2"
 "eval3"
repeat(["eval" .* string.(1:3)], inner=repeat([2]))
2-element Vector{Vector{String}}:
 ["eval1", "eval2", "eval3"]
 ["eval1", "eval2", "eval3"]
repeat(["eval" .* string.(1:3)...], inner=repeat([2]))
6-element Vector{String}:
 "eval1"
 "eval1"
 "eval2"
 "eval2"
 "eval3"
 "eval3"

References

1.
Danielson M, Ekenberg L. Trade-offs for ordinal ranking methods in multi-criteria decisions. In: Bajwa D, Koeszegi ST, Vetschera R, editors. Group decision and negotiation Theory, empirical evidence, and application [Internet]. Cham: Springer International Publishing; 2017. p. 16–27. Available from: https://doi.org/10.1007/978-3-319-52624-9_2
2.
Kaminski B. Julia for data analysis. New York, NY: Manning Publications; 2023.

Footnotes

  1. The SR method was selected because it was the best performing.↩︎