Expected impact

In this entry, you’ll learn what it means to evaluate a question.

Evaluating questions and choosing the right question to think about is extremely important. Thinking about the wrong question produces nothing. It’s a complete waste of resources.

That’s why we need to think about what a “wrong” question is and what it means to evaluate questions “rightly.” Although it’s not an easy topic and there’s no correct answer, let’s work through it.

This entry assumes you understand the overall question-setting process. Read the entry below first.

Let’s begin.

toc

What It Means to Evaluate a Question

The question-setting process

Evaluating questions is the second step of question setting. Instead of making a question you’ve identified into the question as is, you consciously examine whether it’s worth thinking about.

What Exactly Do You Do? Expected Impact

So what exactly does evaluating a question involve?

To give the answer first, evaluating a question means estimating its expected impact. Take a look at the slide below.

Evaluating questions: the structure of expected impact

Since the slide is packed with content, a quick glance is enough for now. We’ll dig into it over the next few entries.

First, the definition: expected impact is “the anticipated benefit of making a question the question.” You multiply the impact a question would yield by the probability of solving it. It’s essentially the mathematical concept of “expected value,” but since you can’t actually calculate a number, I’ve given it a different name (my own coinage).

Estimating this value is the goal of the “evaluating questions” step.

Point

Evaluating a question means estimating its expected impact (the anticipated benefit of making it the question)

Tests and Expected Impact

You’ve probably already estimated expected impact, because you generally estimate expected impact on tests.

No one takes the strategy on a test of “work through the problems from the beginning and don’t move on until you’ve solved each one.” With that strategy, if there’s a problem you can’t solve early on, it’s game over. At the very least, your strategy should be: “Start in order, but if you can’t solve a problem after spending a fair amount of time, give up and move on. If there’s time left after attempting every problem, go back to the ones you couldn’t solve.”

This is exactly estimating expected impact. As you know, you get no points for problems you can’t solve on a test (= no impact). So you judge whether you can solve each problem (solvability) and change which problems you work on accordingly.

On university entrance exams in math, people sometimes estimate expected impact even more explicitly: “Spend the first 30 minutes thinking briefly about every problem and rank them by difficulty. Then solve them starting with the ones that seem solvable.” This strategy ensures you score points and avoid disaster1.

In this way, you’ve very likely already estimated expected impact. Don’t overthink it.

However, the tests you take in school and the questions you think about in the real world differ greatly in two ways:

  1. On tests, the impact (points) is given and quantified, but for real-world questions, you have to decide the impact yourself, and it can’t be quantified
  2. Tests never include questions that no one could answer, but in the real world, questions with zero solvability are common

So estimating expected impact for real-world questions isn’t as easy as on tests. That’s exactly why it’s worth evaluating questions consciously and carefully.

The Structure of Expected Impact

Here, let’s look at the structure of expected impact. Since explaining everything would never end, this entry covers only the overall structure. Look at the slide again.

Evaluating questions: the structure of expected impact

As shown, expected impact is determined by multiplying “impact” by “solvability.”

Keyword

Expected impact = impact × solvability

This simply applies the mathematical concept of “expected value,” so it should be straightforward. If you’ve forgotten, expected value is the average outcome of a single trial: the sum of every possible value multiplied by the probability that it occurs.

In this explanation, “every possible value” is the “impact,” and “the probability that it occurs” is the “solvability.”

Note that actual expected impact can’t be expressed in numbers. I’m only using mathematical expressions to make things easier to understand. Expressions like “value” and “calculate” will come up often from here on, but they’re only metaphors.

Although it may not click yet, let’s move on. You should understand as we look at the details.

What Is Impact?

Evaluating questions: the structure of expected impact

Let’s look at the elements in turn, starting with impact.

Impact is the benefit gained when a question is solved. Think of it as what’s commonly called “value.” Specifically, it’s the sum of all the following elements:

  • Impact on you (yourself)
    • Satisfying intellectual curiosity
    • Financial reward
    • Satisfying other desires
  • Impact on clients
  • Impact on society

The easiest to picture is satisfying intellectual curiosity. For example, it feels good to solve a hard math problem. “Impact” is the totality of such “good or welcome things that happen when a question is solved.”

Keyword

Impact: the benefit (as a whole) gained when a question is solved

What Is Solvability?

Evaluating questions: the structure of expected impact

Next, solvability is the probability of solving a question. It means just what it says.

Keyword

Solvability: the probability of solving a question

As the slide shows, solvability is determined by balancing the difficulty of the question against your resources.

Naturally, the more difficult a question, the lower its solvability. In general, however, questions don’t have a big impact unless they’re difficult. You can’t just pick easy questions. So you consider how difficult a question to take on based on the resources you have.

Both the difficulty of a question and your resources can be evaluated from more detailed angles. I’ll explain these in the following entries.

That covers the overall framework for evaluating questions. Next, let’s dig deeper into the impact of a question.

Entries related to critical thinking are collected on the page below. Feel free to use them as a reference.

Footnotes

  1. However, since it uses time inefficiently, it’s probably not the optimal strategy for a high score. ↩