Improving human predictions

Chapter 9 of Kahneman et al. (2021) discusses how predictions made by humans can be less accurate than predictions made using statistical models. Part of the chapter describes research by Goldberg (1970) and subsequent authors showing that models of human predictions can out-perform the humans on which those models are based.

For example, suppose I’m asked to make predictions in a range of contexts $i\in \{1,2,\dots ,n\}$. My goal is to use some contextual data $x_i\in R^k$ to predict the value of a context-specific outcome $y_i$. I generate predictions $${\overline{y}}_i=y_i+u_i,$$ where the $u_i$ are context-specific errors. The accuracy of my predictions can be measured via their mean squared error (MSE) $$\frac{1}{n}\sum _{i=1}^n({\overline{y}}_i-y_i{)}^2=\frac{1}{n}\sum _{i=1}^n{u}_i^2,$$ where a lower MSE implies higher accuracy. Another way to generate predictions could be to posit a linear model $$y_i=\theta x_i+{ϵ}_i,$$ where $\theta$ is a row vector of coefficients and the ${ϵ}_i$ are random errors. But I don’t know the true outcomes $y_i$—hence needing to predict them—and so I can’t just use ordinary least squares (OLS) to estimate $\theta$. Instead, Goldberg (1970) suggests replacing this linear model with $${\overline{y}}_i=\beta x_i+{\epsilon }_i,$$ where $\beta$ is a (possibly different) vector of coefficients and the ${\epsilon }_i$ are (possibly different) random errors. This second model describes the linearized relationship between my (possibly incorrect) predictions ${\overline{y}}_i$ and the data $x_i$ on which those predictions are based. Since I know my predictions ${\overline{y}}_i$, I can use OLS to obtain an estimate $\hat{\beta }$ of $\beta$ and produce a set of “modeled predictions” $${\hat{y}}_i=\hat{\beta }{x}_i.$$ The difference between the ${\overline{y}}_i$ and ${\hat{y}}_i$ is that the latter ignore the non-linearities in my method for generating predictions. Intuitively, the ${\hat{y}}_i$ represent what I would predict using a simple, linear formula; my predictions ${\overline{y}}_i$ may be generated using a formula that is much more complex, or may not be generated using a formula at all.

So, how do my raw predictions ${\overline{y}}_i$ and their modeled counterparts ${\hat{y}}_i$ compare? The chart below plots the ${\overline{y}}_i$ and ${\hat{y}}_i$ against the true values $y_i$ when

1. the $x_i$ and $u_i$ are iid standard normal, and
2. $y_i=(x_i+z_i)/2$ with $z_i$ iid standard normal.

The modeled predictions are far more accurate: they have an MSE of 0.22, whereas my raw predictions have an MSE of 0.76. In this case, the true relationship between the $y_i$ and $x_i$ is linear, and so a linear model of my predictions is well-placed to out-perform those predictions.

However, modeling predictions does not always improve their accuracy. For example, suppose the contextual data $x_i$ are scalars, and the $x_i$, $y_i$, and $u_i$ have zero means. Then the MSE of the modeled predictions turns out to be $$\frac{1}{n}\sum _{i=1}^n({\hat{y}}_i-y_i{)}^2={\sigma }_y^2+{\rho }_{ux}^2{\sigma }_u^2-{\rho }_{xy}^2{\sigma }_y^2,$$ where ${\sigma }_y^2$ and ${\sigma }_u^2$ are the variances of the $y_i$ and $u_i$, where ${\rho }_{ux}$ is the correlation of the $u_i$ and $x_i$, and where ${\rho }_{xy}$ is the correlation of the $x_i$ and $y_i$. Consequently, replacing my raw predictions ${\overline{y}}_i$ with their modeled counterparts ${\hat{y}}_i$ leads to an accuracy improvement if and only if $${\sigma }_y^2(1-{\rho }_{xy}^2)<{\sigma }_u^2(1-{\rho }_{ux}^2).$$ This condition holds in the example plotted above: both ${\sigma }_u^2$ and ${\sigma }_y^2$ equal unity, but ${\rho }_{xy}=0.69$ is much larger in absolute value than ${\rho }_{ux}=-0.09$. In general, the condition is most likely to hold when

1. ${\sigma }_u^2$ is larger than ${\sigma }_y^2$ (i.e., my raw predictions are relatively noisy);
2. $\left|{\rho }_{xy}\right|$ is large (i.e., the relationship between the $y_i$ and $x_i$ is approximately linear and deterministic); and
3. $\left|{\rho }_{ux}\right|$ is small (i.e., the errors $u_i$ in my raw predictions are relatively uncorrelated with the $x_i$).

Intuitively, if the outcomes $y_i$ are a linear function of the $x_i$ (i.e., if $\left|{\rho }_{xy}\right|=1$) then linearizing my predictions improves their accuracy by removing non-linear errors. On the other hand, if my prediction errors $u_i$ are a linear function of the $x_i$ (i.e., if $\left|{\rho }_{ux}\right|=1$) then linearizing my predictions cannot improve their accuracy because there are no non-linear errors to remove.