Learning in continuous time

This post describes a continuous-time model of Bayesian learning about a binary state. It complements the discrete-time models discussed in previous posts (see, e.g., here or here). I present the model, discuss its learning dynamics, and derive these dynamics analytically.

The model has been used to study decision times (Fudenberg et al., 2018), experimentation (Bolton and Harris, 1999; Moscarini and Smith, 2001), information acquisition (Morris and Strack, 2019), and persuasion (Liao, 2021). It also underlies the drift-diffusion model of reaction times used by psychologists—see Ratcliff (1978) for an early example, and Hébert and Woodford (2023) or Smith (2000) for related discussions.

Model

Suppose I want to learn about a state \(\mu\) that may be high (equal to \(H\)) or low (equal to \(L<H\)). I observe a continuous sample path \((X_t)_{t\ge0}\) with random, instantaneous increments $$\DeclareMathOperator{\E}{E} \newcommand{\der}{\mathrm{d}} \newcommand{\R}{\mathbb{R}} \der X_t=\mu\der t+\sigma \der W_t,$$ where \(\sigma>0\) amplifies the noise generated by the standard Wiener process \((W_t)_{t\ge0}\). These increments provide noisy signals of the state \(\mu\). I use these signals, my prior belief \(p_0=\Pr(\mu=H)\), and Bayes’ rule to form a posterior belief $$p_t\equiv \Pr\left(\mu=H\mid (X_s)_{s<t}\right)$$ about \(\mu\) given the sample path observed up to time \(t\). As shown below, this posterior belief has increments $$\der p_t=p_t(1-p_t)\frac{(H-L)}{\sigma}\der Z_t,$$ where \((Z_t)_{t\ge0}\) is a Wiener process with respect to my information at time \(t\). Its increments $$\der Z_t=\frac{1}{\sigma}\left(\der X_t-\hat\mu_t\der t\right)$$ exceed zero precisely when the corresponding increments \(\der X_t\) in the sample path exceed my posterior estimates $$\begin{align} \hat\mu_t &\equiv \E\left[\mu\mid (X_s)_{s<t}\right] \\ &= p_tH+(1-p_t)L. \end{align}$$

Learning dynamics

My belief increments \(\der p_t\) get smaller as \(p_t\) approaches zero or one. The ratio \((H-L)/\sigma\) controls how quickly this happens. Intuitively, if \((H-L)\) is large then the high and low states are easy to tell apart from the trends in \((X_t)_{t\ge0}\) they imply. But if \(\sigma\) is large then these trends are blurred by the random fluctuations \(\sigma\der W_t\).

I illustrate these dynamics in the chart below. It shows the sample paths \((X_t)_{t\ge0}\) and corresponding beliefs \((p_t)_{t\ge0}\) when \((H,L,\mu,p_0)=(1,0,H,0.5)\) and \(\sigma\in\{1,2\}\). I use the same realization of the underlying Wiener process \((W_t)_{t\ge0}\) for each value of \(\sigma\). Increasing this value slows my convergence to the correct belief \(p_t=1\) because it makes the signals \(\der X_t\) less informative about \(\mu=H\).

Deriving the belief increments

The increments \(\der W_t\) of the Wiener process \((W_t)_{t\ge0}\) are iid normally distributed with mean zero and variance \(\der t\): $$\der W_t\sim N(0,\der t).$$ Thus, given \(\mu\), the increments \(\der X_t\) of the sample path \((X_t)_{t\ge0}\) are iid normal with mean \(\mu\der t\) and variance \(\sigma^2\der t\): $$\der X_t\mid\mu\sim N(\mu\der t,\sigma^2\der t).$$ So these increments have conditional PDF $$\begin{align} f_\mu(\der X_t) &= \frac{1}{\sigma\sqrt{2\pi\der t}}\exp\left(-\frac{(\der X_t-\mu\der t)^2}{2\sigma^2\der t}\right) \\ &= \frac{1}{\sigma\sqrt{2\pi\der t}}\exp\left(-\frac{(\der X_t)^2}{2\sigma^2\der t}\right)\exp\left(\frac{\mu\der X_t}{\sigma^2}-\frac{\mu^2\der t}{2\sigma^2}\right). \end{align}$$ But the rules of Itô calculus imply \((\der X_t)^2=\sigma^2\der t\) and $$\begin{align} \exp\left(\frac{\der X_t\mu}{\sigma^2}-\frac{\mu^2\der t}{2\sigma^2}\right) &= \sum_{k\ge0}\frac{1}{k!}\left(\frac{\mu\der X_t}{\sigma^2}-\frac{\mu^2\der t}{2\sigma^2}\right)^k \\ &= 1+\frac{\mu\der X_t}{\sigma^2} \end{align}$$ because these rules treat terms of order \((\der t)^2\) or smaller as equal to zero. Thus $$f_\mu(\der X_t)=\frac{1}{\sigma^3\sqrt{2\pi\der t}}\exp\left(-\frac{1}{2}\right)\left(\mu\der X_t+\sigma^2\right)$$ for each \(\mu\in\{H,L\}\). Applying Bayes’ rule then gives $$\begin{align} p_{t+\der t} &= \frac{p_tf_H(\der X_t)}{p_tf_H(\der X_t)+(1-p_t)f_L(\der X_t)} \\ &= \frac{p_t\left(H\der X_t+\sigma^2\right)}{\hat\mu_t\der X_t+\sigma^2}, \end{align}$$ where \(\hat\mu_t=\E[\mu\mid (X_s)_{s<t}]\) is my posterior estimate of \(\mu\). So the belief process \((p_t)_{t\ge0}\) has increments $$\begin{align} \der p_t &\equiv p_{t+\der t}-p_t \\ &= \frac{p_t(1-p_t)(H-L)\der X_t}{\hat\mu_t\der X_t+\sigma^2}. \end{align}$$ Finally, taking a Maclaurin series expansion and applying the rules of Itô calculus gives $$\begin{align} \frac{\der X_t}{\hat\mu_t\der X_t+\sigma^2} &= \der X_t\sum_{k\ge0}\frac{(-1)^kk!}{(\sigma^2)^{k+1}}(\der X_t)^k \\ &= \der X_t\left(\frac{1}{\sigma^2}-\frac{1}{\sigma^4}\der X_t\right) \\ &= \frac{1}{\sigma^2}\left(\der X_t-\hat\mu_t\der t\right), \end{align}$$ from which we obtain the expressions for \(\der p_t\) and \(\der Z_t\) provided above.