12 Cross-Fitting and Double Machine Learning
12.1 Why Flexible Nuisance Estimation Is Both Attractive and Dangerous
In Chapters 10 and 11 we developed estimation and inference using estimating equations, influence functions, doubly robust scores, and semiparametric efficiency. A central feature of these methods is that the causal parameter of interest depends on nuisance functions such as the outcome regressions \[\mu_t(x) = \E(Y \mid T=t,\, X=x), \qquad t \in \{0,1\},\] and the propensity score \[\pi(x) = P(T=1 \mid X=x).\] To reduce notation, this chapter drops the stars used for the true nuisance functions in Chapter 11: here \(\mu_t\) and \(\pi\) denote the truth, while hats denote estimated functions.
In many real applications, these nuisance functions may be too complex to model accurately using low-dimensional parametric forms. This makes flexible methods attractive, including penalized regression, splines and sieves, random forests, boosting, neural networks, and ensemble learning.
These methods can substantially improve predictive accuracy, but they also create new statistical difficulties: they may converge more slowly than parametric estimators; they may overfit the observed sample; their asymptotic behavior may be difficult to characterize; and naive plug-in inference can fail even when prediction quality is good.
Better nuisance prediction therefore does not automatically imply valid inference for the target causal parameter. The goal of this chapter is to explain how orthogonal scores, sample splitting, and cross-fitting make it possible to combine flexible nuisance estimation with valid large-sample inference. The solution developed here combines an orthogonal score, which removes first-order sensitivity to nuisance estimation error, with cross-fitting, which separates nuisance estimation from score evaluation to control the remaining empirical process remainder.
The entire chapter is organized by a single decomposition. Writing \(\varphi_{\eta}(O) = \varphi_{\mathrm{eff}}(O;\,\tau_0,\eta)\) for the ATE score at nuisance \(\eta\) (the score \(\varphi_{\mathrm{eff}}\) is introduced formally in Section 12.4), \(\mathbb{P}_n\) for the empirical average, and \(P\) for the population expectation (notation reviewed in Section 12.5.1), the error of the estimator splits into three terms: \[\hat\tau - \tau_0 = \underbrace{(\mathbb{P}_n - P)\,\varphi_{\eta_0}}_{\text{ordinary CLT}} \;+\; \underbrace{P\,(\varphi_{\hat\eta} - \varphi_{\eta_0})}_{\substack{\text{orthogonality} \\ \text{+ product rate}}} \;+\; \underbrace{(\mathbb{P}_n - P)(\varphi_{\hat\eta} - \varphi_{\eta_0})}_{\substack{\text{Donsker control} \\ \text{or cross-fitting}}}. \tag{12.1}\] The first term is a sample average of a fixed function and obeys the central limit theorem. The second is the population drift, tamed by Neyman orthogonality together with the product-rate condition. The third is the empirical-process remainder, tamed either by a Donsker condition or — the route taken by DML — by cross-fitting. The figure below maps each term to the sections that control it; the rest of the chapter fills in this map.
12.2 Why Naive Plug-In Estimation Can Fail
Suppose the parameter of interest is written as a functional \(\psi = \Psi(\eta)\), where \(\eta\) denotes a nuisance object such as \((\mu_0, \mu_1, \pi)\). A natural estimator is the plug-in estimator \(\hat\psi_{\mathrm{plug}} = \Psi(\hat\eta)\), obtained by replacing the nuisance functions with estimators.
12.2.1 The First-Order Taylor Expansion
To see precisely how nuisance error propagates, assume that \(\Psi\) is twice Fréchet differentiable in a neighborhood of \(\eta_0\), with locally bounded second derivative in the norm \(\|\cdot\|\), and apply a first-order Taylor expansion of \(\Psi\) around the true nuisance \(\eta_0\): \[\hat\psi_{\mathrm{plug}} - \psi = \underbrace{D_\eta\Psi(\eta_0)[\hat\eta - \eta_0]}_{\text{linear term}} + \underbrace{R(\hat\eta,\,\eta_0)}_{\text{second-order remainder}}, \tag{12.2}\] where \(D_\eta\Psi(\eta_0)[h]\) denotes the derivative of \(\Psi\) at \(\eta_0\) in direction \(h\); under the smoothness assumption above, the remainder satisfies \(|R(\hat\eta, \eta_0)| \lesssim \|\hat\eta - \eta_0\|^2\). (Mere existence of a Gateaux derivative would not deliver a quadratic remainder.) The expansion is schematic: a causal plug-in estimator typically depends on the empirical distribution of \(X\) as well as on estimated conditional nuisance functions, but the schematic form suffices to display the roles of the two terms.
For root-\(n\) inference we need \(\sqrt{n}(\hat\psi_{\mathrm{plug}} - \psi)\) to have a centered limiting distribution. The second-order remainder is typically manageable: if \(\|\hat\eta - \eta_0\| = o_p(n^{-1/4})\) then \(R = o_p(n^{-1/2})\), which is negligible at the root-\(n\) scale. The obstacle is the linear term, which is proportional to the nuisance estimation error \(\hat\eta - \eta_0\).
12.2.2 Finite-Dimensional Nuisance: Orthogonality Is Sufficient
Suppose first that \(\eta \in \mathbb{R}^d\) for a fixed finite \(d\), and that \(\hat\eta\) is a standard parametric estimator satisfying \(\hat\eta - \eta_0 = O_p(n^{-1/2})\). Then the linear term in Equation 12.2 is also \(O_p(n^{-1/2})\), the same order as the leading term in the asymptotic expansion of \(\hat\psi_{\mathrm{plug}}-\psi\), so nuisance estimation contributes at first order to the limiting distribution of \(\sqrt{n}(\hat\psi_{\mathrm{plug}}-\psi)\). This does not by itself invalidate inference: in a finite-dimensional regular model, the contribution can be incorporated through the joint influence function of the pair \((\hat\psi_{\mathrm{plug}}, \hat\eta)\), the delta method, or a stacked sandwich variance (Chapter 10). The nuisance-estimation term is part of the estimator’s influence function, not an extraneous contamination.
If the functional \(\Psi\) satisfies \(D_\eta\Psi(\eta_0) = 0\) — the Neyman orthogonality condition introduced formally in Section 12.3 — then the linear term vanishes identically. The expansion Equation 12.2 reduces to the second-order remainder alone, which under parametric rates is \(O_p(n^{-1})\) and hence asymptotically negligible. In the finite-dimensional setting, Neyman orthogonality is therefore sufficient rather than necessary for valid inference: it makes the nuisance-estimation contribution asymptotically negligible, so no joint variance correction is required.
One distinction is worth flagging. For an estimator that also contains an oracle empirical-average term — such as the AIPW estimator studied from Section 12.4 onward — orthogonality removes the first-order nuisance contribution while leaving the oracle sampling term unchanged, so the estimator behaves to first order as though the nuisance were known. For a pure direct plug-in \(\Psi(\hat\eta)\) with no such term, a zero first derivative can instead produce a degenerate root-\(n\) limit or a nonstandard second-order limit; that case requires a separate analysis.
12.2.3 Infinite-Dimensional Nuisance: Orthogonality Is Not Enough
When \(\eta\) belongs to a function space — as is the case for the nuisance triple \((\mu_0, \mu_1, \pi)\) — orthogonality is no longer sufficient on its own. Orthogonality removes the first derivative of the target with respect to nuisance perturbations, but it does not by itself control the stochastic fluctuation of the score when the nuisance estimate is a random function chosen from a large class. A second source of first-order error arises from evaluating the estimating equation on the same sample used to estimate \(\hat\eta\). Write the plug-in estimating equation as \(\mathbb{P}_n\{\phi(\,\cdot\,;\psi, \hat\eta)\} = 0\), and decompose the deviation from the population equation as \[\begin{aligned} &\mathbb{P}_n\{\phi(\,\cdot\,;\psi, \hat\eta)\} - \mathbb{P}_n\{\phi(\,\cdot\,;\psi, \eta_0)\} \\ &\quad= \underbrace{P\bigl\{\phi(\,\cdot\,;\psi,\hat\eta) - \phi(\,\cdot\,;\psi,\eta_0)\bigr\}}_{\text{population drift}} + \underbrace{(\mathbb{P}_n - P)\bigl\{\phi(\,\cdot\,;\psi,\hat\eta) - \phi(\,\cdot\,;\psi,\eta_0)\bigr\}}_{\text{empirical process term}}. \end{aligned} \tag{12.3}\] Neyman orthogonality controls the population drift: because the Gateaux derivative of \(P\phi\) with respect to \(\eta\) vanishes at the truth, the population drift is second order in \(\|\hat\eta - \eta_0\|\) under the smoothness conditions of Section 12.2.1.
Neyman orthogonality, however, says nothing about the empirical process term, whose behavior depends on the complexity of the function class \(\{\phi(\,\cdot\,;\psi,\eta) : \eta \in \mathcal{H}\}\) traced out as \(\hat\eta\) varies. When the class is small — as for parametric estimators — classical empirical-process arguments control this term at order \(o_p(n^{-1/2})\). For flexible machine-learning estimators the class is typically too rich for such arguments, and the empirical process term can remain non-negligible even as \(\hat\eta\) converges consistently. Section 12.5 makes this term precise for the ATE score and shows why sample splitting or cross-fitting (Section 12.6–Section 12.7) are needed alongside orthogonality for valid root-\(n\) inference.
12.3 Orthogonal Scores
The key device for reducing sensitivity to nuisance estimation error is an orthogonal score. Let \(\E\{\phi(O;\,\psi,\eta)\} = 0\) be an estimating equation for the target parameter \(\psi\), where \(\eta\) denotes nuisance parameters.
Definition 12.1 (Neyman Orthogonality) The score function \(\phi(O;\,\psi,\eta)\) is called orthogonal (or Neyman orthogonal) at the truth \((\psi_0, \eta_0)\) if the Gateaux derivative of the estimating equation with respect to the nuisance, evaluated at the truth, vanishes: \[\left.\frac{\partial}{\partial r} \E\bigl\{\phi(O;\,\psi_0,\,\eta_0 + r h)\bigr\}\right|_{r=0} = 0\] for all perturbation directions \(h\) in a suitable class.
This condition means that small local perturbations of the nuisance parameter have no first-order effect on the estimating equation at the truth. Under suitable second-order smoothness — or, for the ATE score, by the exact drift calculation in the proof of Theorem 12.1 — the remaining population drift is second order in the nuisance error; the zero derivative alone gives local first-order insensitivity, not automatically a quadratic bound. Orthogonality does not mean nuisance estimation no longer matters: it controls only the population-level drift, and a separate argument is needed to rule out the additional bias that can arise when the same sample is used for both nuisance estimation and score evaluation. That argument is the subject of Section 12.5–Section 12.7.
Orthogonality is the principal reason that semiparametric estimators can remain valid under flexible nuisance estimation. In particular, the doubly robust score introduced in Chapter 11 is orthogonal, which is exactly why the product-rate condition Equation 12.13 below (identical to the condition stated in Chapter 11) is sufficient for asymptotic negligibility of nuisance estimation error.
12.4 The Orthogonal Score for the Average Treatment Effect
We continue to use the average treatment effect \(\tau = \E\{Y(1) - Y(0)\}\) as the running example. Under consistency, conditional exchangeability, and positivity, the efficient influence function derived in Chapter 11 is \[\varphi_{\mathrm{eff}}(O;\,\tau,\eta) = \frac{T}{\pi(X)}\{Y - \mu_1(X)\} - \frac{1-T}{1-\pi(X)}\{Y - \mu_0(X)\} + \mu_1(X) - \mu_0(X) - \tau, \tag{12.4}\] where \(\eta = (\mu_0, \mu_1, \pi)\). Chapter 11 identified this as the right score for the ATE; the role of Chapter 12 is to explain how to estimate its nuisance components safely when flexible learners are used. DML does not introduce a new causal estimand or a new ATE score; it is the cross-fitted implementation of the AIPW score, together with rate and regularity conditions that permit flexible nuisance learning. (The cross-fitted estimator is, of course, a different finite-sample object from the same-sample AIPW estimator; what is unchanged is the estimand and the orthogonal score.) We now prove that this score is Neyman orthogonal. The proof verifies Definition 12.1 separately for perturbations in each of the three nuisance components.
Lemma 12.1 (Neyman Orthogonality of the ATE Score) The score \(\varphi_{\mathrm{eff}}(O;\,\tau,\eta)\) in Equation 12.4 is Neyman orthogonal at every truth \((\tau_0, \eta_0) = (\tau_0, \mu_0, \mu_1, \pi)\) whose nuisance components are the observed-data functions defined below, provided strong overlap holds (\(c \le \pi(X) \le 1-c\) for some \(c>0\)) and \(\E|Y| < \infty\). That is, for every bounded perturbation direction \(h(x)\), \(g(x)\), or \(\ell(x)\), \[\begin{aligned} \left.\frac{\partial}{\partial r} \E\!\left\{\varphi_{\mathrm{eff}}(O;\,\tau_0,\,\mu_0,\,\mu_1,\,\pi+rh)\right\}\right|_{r=0} &= 0, \\ \left.\frac{\partial}{\partial r} \E\!\left\{\varphi_{\mathrm{eff}}(O;\,\tau_0,\,\mu_0,\,\mu_1+rg,\,\pi)\right\}\right|_{r=0} &= 0, \\ \left.\frac{\partial}{\partial r} \E\!\left\{\varphi_{\mathrm{eff}}(O;\,\tau_0,\,\mu_0+r\ell,\,\mu_1,\,\pi)\right\}\right|_{r=0} &= 0. \end{aligned}\]
Proof. Throughout, the perturbation paths are assumed to remain inside the nuisance parameter space — in particular, \(\pi + rh\) stays in \((0,1)\) for all sufficiently small \(r\), which holds for bounded \(h\) under the strong-overlap hypothesis — and differentiation and expectation are interchanged under standard domination arguments, which use the bounds on \(1/\pi\) and \(1/(1-\pi)\) supplied by strong overlap together with \(\E|Y| < \infty\); the law of iterated expectations is applied by conditioning on \(X\). By definition of the observed-data nuisance functions, \[\E(Y \mid T=1, X) = \mu_1(X), \qquad \E(Y \mid T=0, X) = \mu_0(X), \qquad \E(T \mid X) = \pi(X).\] These equalities hold by definition and require no causal assumptions. The identification assumptions — consistency, conditional exchangeability, and positivity — serve a different purpose: they identify \(\tau_0 = \E\{Y(1)-Y(0)\} = \E\{\mu_1(X)-\mu_0(X)\}\) as a causal quantity. They play no role in the algebraic verification of Neyman orthogonality once the observed-data score has been specified; the calculation below is an observed-data probability calculation.
The calculations below rest on two complementary families of conditional-mean identities. The defining identity \(\E(T\mid X) = \pi(X)\) implies the two balancing identities \[\E\!\left(\frac{T}{\pi(X)}\;\middle|\; X\right) = 1, \qquad \E\!\left(\frac{1-T}{1-\pi(X)}\;\middle|\; X\right) = 1, \tag{12.5}\] which eliminate the perturbations in \(\mu_1\) and \(\mu_0\). The defining identities \(\E(Y \mid T=t, X) = \mu_t(X)\) imply the two residual identities \(\E\{T(Y-\mu_1(X))\mid X\} = 0\) and \(\E\{(1-T)(Y-\mu_0(X))\mid X\} = 0\) (for the first, condition on \(X\) and use \(\E\{TY\mid X\} = \pi(X)\,\E(Y\mid T{=}1,X) = \pi(X)\mu_1(X)\)), which eliminate the perturbation in \(\pi\).
Perturbation in \(\pi\). Replace \(\pi\) by \(\pi + rh\) and differentiate. The only terms in Equation 12.4 that depend on \(\pi\) are the two IPW residual terms, so \[\left.\frac{\partial}{\partial r}\E\!\left\{\varphi_{\mathrm{eff}}(O;\,\tau_0,\mu_0,\mu_1,\pi+rh)\right\}\right|_{r=0} = \E\!\left[-\frac{T\,h(X)}{\pi(X)^2}\{Y-\mu_1(X)\} - \frac{(1-T)\,h(X)}{(1-\pi(X))^2}\{Y-\mu_0(X)\}\right] = 0\] by the two residual identities above (factor out \(h(X)/\pi(X)^2\) and \(h(X)/(1-\pi(X))^2\) respectively, then condition on \(X\)).
Perturbation in \(\mu_1\). Replace \(\mu_1\) by \(\mu_1 + rg\). Only \(-(T/\pi(X))\mu_1(X)\) and \(\mu_1(X)\) in Equation 12.4 depend on \(\mu_1\), so \[\left.\frac{\partial}{\partial r}\E\!\left\{\varphi_{\mathrm{eff}}(O;\,\tau_0,\mu_0,\mu_1+rg,\pi)\right\}\right|_{r=0} = \E\!\left[g(X)\left(1 - \frac{T}{\pi(X)}\right)\right] = 0\] by the first identity in Equation 12.5.
Perturbation in \(\mu_0\). Replace \(\mu_0\) by \(\mu_0 + r\ell\). Symmetrically, \[\left.\frac{\partial}{\partial r}\E\!\left\{\varphi_{\mathrm{eff}}(O;\,\tau_0,\mu_0+r\ell,\mu_1,\pi)\right\}\right|_{r=0} = \E\!\left[\ell(X)\left(\frac{1-T}{1-\pi(X)} - 1\right)\right] = 0\] by the second identity in Equation 12.5. This completes the proof. \(\square\)
The score in Equation 12.4 simultaneously fulfills three purposes:
- its zero-moment equation \(\E\{\varphi_{\mathrm{eff}}(O;\,\tau,\eta)\} = 0\) represents, under the causal identification assumptions of Chapter 5, the identified observed-data functional for the ATE (the score does not create causal identification by itself);
- it is doubly robust, yielding a consistent estimator whenever either the outcome model or the propensity score model is correctly specified; and
- it is the efficient influence function, so an estimator that is regular, asymptotically linear with this influence function, and whose nuisance remainders are asymptotically negligible achieves the semiparametric efficiency bound (Theorem 11.8).
This is why the score in Equation 12.4 is central in modern causal inference.
12.5 Why Reusing the Same Data Can Be Problematic
Section 12.2 identified two obstacles to valid root-\(n\) inference: the population drift and the empirical process term in decomposition Equation 12.3. We showed there that Neyman orthogonality reduces the population drift to a second-order quantity in \(\|\hat\eta - \eta_0\|\) but leaves the empirical process term uncontrolled. This section makes the empirical process term precise in the context of the ATE estimating equation and introduces the Donsker condition — the classical tool for controlling it when the same data are reused for both nuisance estimation and score evaluation.
12.5.1 The Plug-In Remainder for the AIPW Estimator
Recall from Chapter 10 that we write \(\mathbb{P}_n f = n^{-1}\sum_{i=1}^n f(O_i)\) for the empirical average of a function \(f\), and define the centered empirical process \[\mathbb{G}_n f = \sqrt{n}\,(\mathbb{P}_n - P) f = \frac{1}{\sqrt{n}}\sum_{i=1}^n \{f(O_i) - \E f(O)\}.\] Consider the plug-in AIPW estimator \(\hat\tau_{\mathrm{plug}}\) defined by the moment equation \(\mathbb{P}_n\bigl\{\varphi_{\mathrm{eff}}(O;\,\hat\tau_{\mathrm{plug}},\hat\eta)\bigr\} = 0\), where the same sample is used for both nuisance estimation and score evaluation. Because \(\varphi_{\mathrm{eff}}\) is linear in its \(\tau\)-argument with \(\partial_\tau\varphi_{\mathrm{eff}}=-1\), we may rearrange and add and subtract \(\varphi_{\mathrm{eff}}(O;\,\tau_0,\eta_0)\) to obtain \[\hat\tau_{\mathrm{plug}} - \tau_0 = \mathbb{P}_n\{\varphi_{\mathrm{eff}}(O;\,\tau_0,\eta_0)\} + \underbrace{\mathbb{P}_n\{\varphi_{\mathrm{eff}}(O;\,\tau_0,\hat\eta) - \varphi_{\mathrm{eff}}(O;\,\tau_0,\eta_0)\}}_{=:\,R_n}. \tag{12.6}\] The first term is a sample average of a fixed function, satisfies the CLT, and is \(O_p(n^{-1/2})\). The remainder \(R_n\) is exactly the left-hand side of decomposition Equation 12.3 specialized to \(\phi = \varphi_{\mathrm{eff}}\) at \((\tau_0,\eta_0)\) — together with the leading CLT term, this is precisely the three-term roadmap Equation 12.1 of Section 12.1: \[R_n = \underbrace{P\{\varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\hat\eta) - \varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\eta_0)\}}_{\text{population drift}} + \underbrace{(\mathbb{P}_n - P)\{\varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\hat\eta) - \varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\eta_0)\}}_{\text{empirical process term}}. \tag{12.7}\] By Lemma 12.1, Neyman orthogonality eliminates the first-order term in the expansion of the population drift around \(\eta_0\), leaving a remainder bilinear in \((\hat\mu_t - \mu_t,\,\hat\pi - \pi)\). Bounding this bilinear remainder by Cauchy–Schwarz under strong overlap and the product-rate condition Equation 12.13 then yields population drift \(= o_p(n^{-1/2})\) (the explicit calculation appears as Step 1 in the proof of Theorem 12.1). Writing \(f_\eta(\cdot) = \varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\eta)\) (the object written \(\varphi_\eta\) in the roadmap Equation 12.1), the empirical process term takes the form \[(\mathbb{P}_n - P)\{f_{\hat\eta} - f_{\eta_0}\} = \frac{1}{\sqrt{n}}\,\mathbb{G}_n\{f_{\hat\eta} - f_{\eta_0}\}. \tag{12.8}\] This term depends on \(\hat\eta\), which is estimated from the same sample. Whether it is \(o_p(n^{-1/2})\) depends on how complex the class \(\{f_\eta : \eta \in \mathcal{H}\}\) is — a question the Donsker condition answers.
12.5.2 The Donsker Condition \(\star\)
This subsection provides empirical-process background and may be treated as second-pass material. For a first reading it suffices to know: (i) what same-sample stochastic equicontinuity is supposed to accomplish; (ii) that highly adaptive learners may fall outside classical fixed-complexity regimes; and (iii) that cross-fitting supplies a different route (Section 12.6–Section 12.7).
The Donsker condition is a technical way of asking whether the function class used by the nuisance learner is tame enough that same-sample score evaluation remains asymptotically well behaved.
Definition 12.2 (Donsker Class) A class of measurable functions \(\mathcal{F}\) is a Donsker class (with respect to \(P\)) if the empirical process \(\{\mathbb{G}_n f : f \in \mathcal{F}\}\) converges weakly in \(\ell^\infty(\mathcal{F})\) to a tight Gaussian process indexed by \(\mathcal{F}\). In particular, every Donsker class satisfies the uniform bound \[\sup_{f \in \mathcal{F}} \bigl|(\mathbb{P}_n - P) f\bigr| = O_p(n^{-1/2})\] and the empirical process \(\mathbb{G}_n = n^{1/2}(\mathbb{P}_n - P)\) is asymptotically equicontinuous over \(\mathcal{F}\) with respect to an appropriate semimetric. Conversely, weak convergence in \(\ell^\infty(\mathcal{F})\) follows from convergence of the finite-dimensional distributions — which holds by the multivariate CLT whenever every \(f \in \mathcal{F}\) is square-integrable — together with asymptotic equicontinuity under a totally bounded semimetric and the relevant measurability conditions. The displayed uniform bound is a consequence of tightness; it does not by itself replace the finite-dimensional convergence requirement. See Vaart (1998), Chapter 19, for a complete treatment.
A key sufficient condition for \(\mathcal{F}\) to be Donsker involves its bracketing entropy. The bracketing number \(N_{[\,]}(\epsilon, \mathcal{F}, L_2(P))\) counts the minimum number of \(\epsilon\)-brackets \([\ell_j, u_j]\) needed to cover \(\mathcal{F}\), where a bracket is a pair of functions with \(P(u_j - \ell_j)^2 \leq \epsilon^2\); its logarithm \[H_{[\,]}(\epsilon, \mathcal{F}, L_2(P)) = \log N_{[\,]}(\epsilon, \mathcal{F}, L_2(P))\] is the bracketing entropy. Subject to measurability and a square-integrable envelope, the class \(\mathcal{F}\) is Donsker whenever \[\int_0^{\delta_0} \sqrt{H_{[\,]}(\epsilon, \mathcal{F}, L_2(P))}\, d\epsilon < \infty\] for some \(\delta_0 > 0\) (the integral is over the relevant range of bracket sizes; classes are typically bounded in \(L_2(P)\) so the upper limit can be taken as the \(L_2\)-diameter of \(\mathcal{F}\)). Intuitively, the integral is finite when the class is not too large: a sufficient condition is that the bracketing entropy grow at most polynomially with exponent below two, \(H_{[\,]}(\epsilon, \mathcal{F}, L_2(P)) \lesssim \epsilon^{-v}\) for some \(v < 2\). Note carefully that it is the entropy, not the bracketing number itself, that may grow polynomially in \(1/\epsilon\); the bracketing number then grows roughly exponentially in \(\epsilon^{-v}\).
Example 12.1 (Donsker and Non-Donsker Classes)
- Parametric classes (e.g., logistic regression models indexed by a finite-dimensional parameter) are Donsker under mild moment conditions, since the bracketing entropy grows only logarithmically.
- Hölder-smooth function classes on \([0,1]^d\) with smoothness index \(s > d/2\) are Donsker: a bounded Hölder ball has bracketing entropy \(H_{[\,]}(\epsilon, \mathcal{F}, L_2(P)) \lesssim \epsilon^{-d/s}\) for any \(P\) (via the sup-norm entropy bound), so \(s > d/2\) suffices for the entropy integral to converge. Under regular full-dimensional support conditions on \(P\), a matching lower bound holds, making \(s > d/2\) the sharp threshold for this class; for \(P\) concentrated on a lower-dimensional set, the \(L_2(P)\) entropy can be much smaller.
- Indicator classes \(\mathcal{F} = \{x \mapsto \mathbf{1}(x \leq t) : t \in \mathbb{R}\}\) are Donsker by the classical Donsker theorem for empirical distribution functions.
- Random forests and neural networks. Common modern implementations typically fall outside classical fixed Donsker regimes, especially when their complexity grows with \(n\), because the function classes they implicitly explore can approximate arbitrarily complex functions. Restricted or fixed-complexity variants can nevertheless be Donsker.
- High-dimensional lasso. With a growing number of selected variables the effective model dimension also grows with \(n\), so the class typically falls outside fixed Donsker regimes; such procedures are often analyzed using sparsity-based empirical-process arguments rather than Donsker theorems.
12.5.3 Connecting the Two Terms
Returning to the remainder Equation 12.7, the two terms correspond exactly to the two challenges identified in Section 12.2.3:
- Population drift. Its first-order nuisance derivative is eliminated by Neyman orthogonality. For the ATE score, the resulting exact bilinear remainder is \(o_p(n^{-1/2})\) under strong overlap and the product-rate condition, regardless of how \(\hat\eta\) is estimated.
- Empirical process term. Requires a separate argument. If \(\{f_\eta : \eta \in \mathcal{H}\}\) is Donsker, \(f_{\hat\eta}\) belongs to this class with probability approaching one, and the induced score converges in the sense \(P\{f_{\hat\eta} - f_{\eta_0}\}^2 = o_p(1)\) — convergence of \(\hat\eta\) in an unspecified norm does not by itself imply this — then by stochastic equicontinuity, \[\mathbb{G}_n\{f_{\hat\eta} - f_{\eta_0}\} = o_p(1), \tag{12.9}\] making the empirical process term \(o_p(n^{-1/2})\). Combined with the controlled population drift, the full remainder \(R_n = o_p(n^{-1/2})\), and root-\(n\) asymptotic linearity follows from Equation 12.6.
When the Donsker condition fails, Equation 12.9 need not hold: the empirical process term can remain non-negligible even as \(\hat\eta \to \eta_0\). Root-\(n\) asymptotic linearity of the plug-in estimator may then fail unless some other empirical-process or stability argument is available — for example, algorithm-specific bounds for sparse high-dimensional learners, or the sample-splitting arguments of Section 12.6–Section 12.7. The Donsker condition is sufficient for same-sample empirical-process control, not necessary, but absent some such argument the limiting distribution can degrade. This is why orthogonality and cross-fitting are complements rather than substitutes: orthogonality controls the population drift and cross-fitting controls the empirical process term, and the two together yield valid root-\(n\) inference with flexible nuisance learners under the broadest set of conditions.
The practical solution is therefore to separate nuisance estimation from score evaluation, which leads to sample splitting and, in its more efficient form, cross-fitting.
12.6 Sample Splitting
Section 12.5 showed that the plug-in remainder \(R_n\) in Equation 12.7 has two parts: the population drift, handled by Neyman orthogonality, and the empirical process term Equation 12.8, which is not controlled by orthogonality alone when the Donsker condition fails. Sample splitting resolves the second problem by construction.
Partition the indices \(\{1,\ldots,n\}\) into a training sample \(\mathcal{I}_{\mathrm{train}}\) and an evaluation sample \(\mathcal{I}_{\mathrm{eval}}\). Fit the nuisance estimators \(\hat\eta^{(\mathrm{train})} = (\hat\mu_0^{(\mathrm{train})}, \hat\mu_1^{(\mathrm{train})}, \hat\pi^{(\mathrm{train})})\) on \(\mathcal{I}_{\mathrm{train}}\), then define \(\hat\tau\) by solving the estimating equation on the held-out sample: \[\frac{1}{|\mathcal{I}_{\mathrm{eval}}|}\sum_{i \in \mathcal{I}_{\mathrm{eval}}} \varphi_{\mathrm{eff}}\!\bigl(O_i;\,\tau,\,\hat\eta^{(\mathrm{train})}\bigr) = 0.\]
To see why this removes the same-sample empirical-process dependence, write the analog of the remainder decomposition Equation 12.7 on the evaluation sample. Let \(n_{\mathrm{e}} = |\mathcal{I}_{\mathrm{eval}}|\) and \(\mathbb{P}_{n_{\mathrm{e}}}\) denote the empirical measure over \(\mathcal{I}_{\mathrm{eval}}\). The remainder on the evaluation fold is \[\begin{aligned} R_n^{(\mathrm{e})} &= \underbrace{P\{\varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\hat\eta^{(\mathrm{train})}) - \varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\eta_0)\}}_{\text{population drift: } o_p(n^{-1/2}) \text{ by orthogonality}} \\ &\quad+ \underbrace{(\mathbb{P}_{n_{\mathrm{e}}} - P)\{\varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\hat\eta^{(\mathrm{train})}) - \varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\eta_0)\}}_{\text{empirical process term}}. \end{aligned} \tag{12.10}\] The population drift is \(o_p(n^{-1/2})\) by the same orthogonality plus product-rate argument used in Section 12.5.1 and made explicit as Step 1 of the proof of Theorem 12.1. The empirical process term is now different in character: because \(\hat\eta^{(\mathrm{train})}\) is estimated on \(\mathcal{I}_{\mathrm{train}}\), it is independent of the observations \(\{O_i : i \in \mathcal{I}_{\mathrm{eval}}\}\). Conditioning on \(\hat\eta^{(\mathrm{train})}\), the summands in the empirical process term are independent and mean zero, so under \(L_2\) consistency of \(\hat\eta^{(\mathrm{train})}\), strong overlap, and finite-variance moment conditions, a conditional variance bound (made precise in the proof of Theorem 12.1) gives \[(\mathbb{P}_{n_{\mathrm{e}}} - P)\{\varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\hat\eta^{(\mathrm{train})}) - \varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\eta_0)\} = o_p(n^{-1/2}).\] No Donsker condition on the function class is needed: independence allows a conditional argument in place of the stochastic equicontinuity that Donsker theory would otherwise supply. The full remainder \(R_n^{(\mathrm{e})} = o_p(n^{-1/2})\), and root-\(n\) asymptotic linearity of the split estimator follows.
The main disadvantage of sample splitting is inefficiency: only \(|\mathcal{I}_{\mathrm{eval}}|\) observations contribute to the estimation of \(\tau\), and the estimate depends on the particular random partition chosen. In other words, sample splitting buys cleaner asymptotic theory by sacrificing training sample size for each nuisance learner, which can slow the convergence of \(\hat\eta\) and inflate the variance of the final estimator.
12.7 Cross-Fitting
Cross-fitting extends sample splitting to avoid the first-order information loss of a single evaluation split — permitting full-sample efficiency under the conditions of Theorem 12.1 — while preserving the independence argument that controls the empirical-process term. The key observation from Section 12.6 was that conditioning on \(\hat\eta^{(\mathrm{train})}\) makes the evaluation-fold summands independent and mean zero, so the empirical-process term can be made \(o_p(n^{-1/2})\) by a conditional variance argument under \(L_2\) consistency, overlap, and moment conditions. Cross-fitting applies this argument simultaneously to every observation by rotating the training and evaluation roles across \(K\) folds.
Partition the indices \(\{1,\ldots,n\}\) into \(K\) approximately equal folds \(\mathcal{I}_1,\ldots,\mathcal{I}_K\). For each fold \(k\):
- Estimate the nuisance functions using all observations not in fold \(k\), obtaining \(\hat\eta^{(-k)} = (\hat\mu_0^{(-k)}, \hat\mu_1^{(-k)}, \hat\pi^{(-k)})\).
- Evaluate the orthogonal score on the held-out fold \(\mathcal{I}_k\) using \(\hat\eta^{(-k)}\).
- Aggregate the score contributions across all \(K\) folds.
The cross-fitted estimator \(\hat\tau_{\mathrm{DML}}\) solves \[\frac{1}{n}\sum_{k=1}^K \sum_{i \in \mathcal{I}_k} \varphi_{\mathrm{eff}}\!\bigl(O_i;\,\tau,\,\hat\eta^{(-k)}\bigr) = 0. \tag{12.11}\]
Throughout this section we write \(n_k = |\mathcal{I}_k|\), so that \(\sum_k n_k = n\) and the fold proportions \(n_k/n\) sum to one. When folds are exactly balanced, \(n_k/n = 1/K\) and the weighted averages below collapse to the unweighted form \(K^{-1}\sum_k\). The figure below depicts the procedure for one representative fold.
To see why the same-sample empirical-process dependence is no longer an obstacle, write the remainder Equation 12.7 as a sum over folds, \[R_n = \sum_{k=1}^K \frac{n_k}{n}\,R_n^{(k)},\] where \(R_n^{(k)}\) is the fold-\(k\) remainder evaluated on \(\mathcal{I}_k\), decomposed exactly as in Equation 12.10 with \(\mathcal{I}_{\mathrm{eval}} = \mathcal{I}_k\) and \(\hat\eta^{(\mathrm{train})} = \hat\eta^{(-k)}\). For each \(k\), the nuisance estimate \(\hat\eta^{(-k)}\) is trained on \(\{1,\ldots,n\}\setminus\mathcal{I}_k\), which is independent of the observations in \(\mathcal{I}_k\).
Care is needed about what is conditionally mean zero. Writing \(f_k = \varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\hat\eta^{(-k)}) - \varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\eta_0)\), the centered variables \(f_k(O_i) - Pf_k\), \(i \in \mathcal{I}_k\), are conditionally independent and mean zero given the training sigma-field; the uncentered score differences have conditional mean \(Pf_k = B_n^{(k)}\), the population drift, which is controlled separately by orthogonality and the product-rate condition (Step 1 of the proof of Theorem 12.1 gives \(\sqrt{n}\,B_n^{(k)} = o_p(1)\)). For the centered part, a conditional variance argument under \(L_2\) consistency of \(\hat\eta^{(-k)}\), strong overlap, and finite-variance moment conditions (Step 2 of the proof) gives \(\sqrt{n}\,E_n^{(k)} = o_p(1)\) without any Donsker assumption on the nuisance class. Together, \(\sqrt{n}\,R_n^{(k)} = \sqrt{n}\,(B_n^{(k)} + E_n^{(k)}) = o_p(1)\) for each \(k\). Since \(R_n\) is a convex combination of the \(R_n^{(k)}\) with weights \(n_k/n \le 1\), the fold-wise bound transfers directly: \(\sqrt{n}\,R_n = o_p(1)\).
Cross-fitting therefore achieves two goals simultaneously: it preserves the held-out independence structure that removes the need for Donsker conditions, and it ensures every observation serves once as an evaluation point, so the full sample contributes to the estimation of \(\tau\). For this reason, cross-fitting has become the standard device for combining orthogonal scores with flexible nuisance estimation (Zheng and Laan 2011; Chernozhukov et al. 2018; Laan and Rose 2011). Cross-fitting removes the same-sample empirical-process obstruction, but it does not by itself guarantee good finite-sample performance if the nuisance estimators are inaccurate or overlap is poor.
12.8 Double Machine Learning for the Average Treatment Effect
The estimator in Equation 12.12 is simply the AIPW estimator from Chapter 11 with out-of-fold nuisance estimates substituted in place of estimates fit on the full sample. The term double machine learning (DML), introduced by Chernozhukov et al. (2018), refers to the combination of three ingredients:
- Neyman orthogonality, which reduces the population drift term in the remainder decomposition Equation 12.7 to a second-order product in nuisance estimation errors;
- cross-fitting, which removes the same-sample dependence in the empirical-process term of Equation 12.7 by separating training and evaluation folds, so the term can be controlled by a conditional variance argument in place of a Donsker assumption; and
- flexible machine-learning estimation of the nuisance functions \(\mu_0\), \(\mu_1\), and \(\pi\), which is made valid by the first two ingredients.
The word “double” should not be read as merely meaning “two models”; it refers to the combination of nuisance learning with orthogonalization and debiasing. The crucial feature is the orthogonal moment structure combined with cross-fitting, which together make flexible nuisance estimation compatible with root-\(n\) inference. In this sense, DML is not a new causal estimand and not a new identification strategy; it is a modern estimation protocol built around an orthogonal score.
12.9 Rate Conditions and Asymptotic Normality
The main theoretical advantage of orthogonality and cross-fitting is that valid root-\(n\) inference can hold even when nuisance estimators converge more slowly than \(n^{-1/2}\). The following result makes this precise.
Theorem 12.1 (Asymptotic Linearity Under Cross-Fitting) Let \(O_1,\ldots,O_n\) be i.i.d. from \(P\). Let the number of folds \(K\) be fixed, with fold sizes satisfying \(\min_k n_k/n\) bounded away from zero, and assume the fold partition is either deterministic or generated by randomization independent of \(O_1,\ldots,O_n\). Suppose that for each fold \(k\) all nuisance fitting, feature selection, preprocessing, and tuning use only observations outside \(\mathcal{I}_k\). Assume \(\E|Y|^{2+\delta} < \infty\) for some \(\delta > 0\) and \(0 < \sigma^2 = \E\{\varphi_{\mathrm{eff}}(O;\,\tau_0,\eta_0)^2\} < \infty\). (The causal identification assumptions of Chapter 5 are required if \(\tau_0\) is to be interpreted as the ATE; the distributional conclusion below concerns the observed-data functional.) Suppose further:
the score \(\varphi_{\mathrm{eff}}(O;\,\tau,\eta)\) is orthogonal at the truth \((\tau_0, \eta_0)\);
each component of \(\hat\eta^{(-k)}\) is \(L_2(P)\)-consistent for the corresponding component of \(\eta_0\), that is, \(\|\hat\eta^{(-k)} - \eta_0\|_{L_2(P)} = o_p(1)\) for each fold \(k\);
strong overlap and bounded fitted propensities hold: there exists a constant \(c > 0\) such that \(c \le \pi(X) \le 1-c\) almost surely (a property of the data-generating law), and, as an assumption on (or enforced constraint of) the nuisance estimator, \[P\!\left(c \;\leq\; \hat\pi^{(-k)}(X) \;\leq\; 1 - c \text{ for almost every } X\right) \to 1\] as \(n\to\infty\), for each fold \(k\);
the nuisance estimators satisfy the product-rate condition \[\|\hat\pi^{(-k)} - \pi\| \cdot \|\hat\mu_t^{(-k)} - \mu_t\| = o_p(n^{-1/2}), \qquad t = 0, 1, \tag{12.13}\] where \(\|\cdot\|\) denotes the \(L_2(P)\) norm.
Then the cross-fitted estimator satisfies \[\sqrt{n}(\hat\tau_{\mathrm{DML}} - \tau_0) = \frac{1}{\sqrt{n}}\sum_{i=1}^n \varphi_{\mathrm{eff}}(O_i;\,\tau_0,\eta_0) + o_p(1),\] and consequently \[\sqrt{n}(\hat\tau_{\mathrm{DML}} - \tau_0) \overset{d}{\longrightarrow} N\!\Bigl(0,\;\E\bigl[\varphi_{\mathrm{eff}}(O;\,\tau_0,\eta_0)^2\bigr]\Bigr).\]
The asymptotic variance equals the semiparametric efficiency bound (Theorem 11.8). Under the standard regularity conditions ensuring that the estimator is regular over the maintained semiparametric model, the influence-function expansion above therefore implies attainment of the semiparametric efficiency bound; pointwise asymptotic linearity alone does not constitute efficiency in the formal sense.
Proof sketch. In addition to the conditions stated in the theorem, we assume measurability of the nuisance estimators wherever it is needed; such conditions are standard in the double machine learning literature (Chernozhukov et al. 2018).
Write \(n_k = |\mathcal{I}_k|\) and \(\mathbb{P}_{n_k}\) for the empirical measure over fold \(k\). The cross-fitted moment equation Equation 12.11 can be written as \[0 = \frac{1}{n}\sum_{k=1}^K\sum_{i\in\mathcal{I}_k} \varphi_{\mathrm{eff}}(O_i;\,\hat\tau_{\mathrm{DML}},\hat\eta^{(-k)}) = \sum_{k=1}^K \frac{n_k}{n}\,\mathbb{P}_{n_k}\bigl\{\varphi_{\mathrm{eff}}(O;\,\hat\tau_{\mathrm{DML}},\hat\eta^{(-k)})\bigr\}.\] Adding and subtracting \(\varphi_{\mathrm{eff}}(O;\,\tau_0,\eta_0)\) inside each fold and rearranging, using the fact that the score is linear in \(\tau\) with \(\partial_\tau\varphi_{\mathrm{eff}} = -1\), \[\hat\tau_{\mathrm{DML}} - \tau_0 = \frac{1}{n}\sum_{i=1}^n \varphi_{\mathrm{eff}}(O_i;\,\tau_0,\eta_0) + \sum_{k=1}^K \frac{n_k}{n}\,R_n^{(k)},\] where each fold remainder \(R_n^{(k)}\) decomposes as in Equation 12.10: \[R_n^{(k)} = \underbrace{P\bigl\{\varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\hat\eta^{(-k)}) - \varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\eta_0)\bigr\}}_{\text{population drift } B_n^{(k)}} + \underbrace{(\mathbb{P}_{n_k} - P)\bigl\{\varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\hat\eta^{(-k)}) - \varphi_{\mathrm{eff}}(\,\cdot\,;\tau_0,\eta_0)\bigr\}}_{\text{empirical process term } E_n^{(k)}}.\] Since \(n_k/n \le 1\) and \(\sum_k n_k/n = 1\), it suffices to show \(\sqrt{n}\,R_n^{(k)} = o_p(1)\) for each \(k\).
Step 1: population drift. By iterated expectations conditioning on \(X\), and using \(\E(T\mid X) = \pi(X)\) and \(\E(Y\mid T{=}t, X) = \mu_t(X)\), \[\begin{aligned} B_n^{(k)} &= \E\!\left[\frac{\pi(X)}{\hat\pi^{(-k)}(X)}\bigl\{\mu_1(X)-\hat\mu_1^{(-k)}(X)\bigr\} - \frac{1-\pi(X)}{1-\hat\pi^{(-k)}(X)}\bigl\{\mu_0(X)-\hat\mu_0^{(-k)}(X)\bigr\}\right. \\ &\qquad\qquad\left. +\,\hat\mu_1^{(-k)}(X)-\mu_1(X) -\,\hat\mu_0^{(-k)}(X)+\mu_0(X)\right] \\ &= \E\!\left[\bigl(\hat\mu_1^{(-k)}-\mu_1\bigr)(X)\cdot\frac{\hat\pi^{(-k)}(X)-\pi(X)}{\hat\pi^{(-k)}(X)} + \bigl(\hat\mu_0^{(-k)}-\mu_0\bigr)(X)\cdot\frac{\hat\pi^{(-k)}(X)-\pi(X)}{1-\hat\pi^{(-k)}(X)}\right]. \end{aligned}\] Under strong overlap (condition (iii)), \(\hat\pi^{(-k)}\) is bounded away from \(0\) and \(1\) with probability approaching one, so the denominators are bounded below. Applying the Cauchy–Schwarz inequality, \[|B_n^{(k)}| \leq C\Bigl(\|\hat\mu_1^{(-k)} - \mu_1\|\cdot\|\hat\pi^{(-k)}-\pi\| + \|\hat\mu_0^{(-k)} - \mu_0\|\cdot\|\hat\pi^{(-k)}-\pi\|\Bigr) = o_p(n^{-1/2})\] by the product-rate condition (iv). Hence \(\sqrt{n}\,B_n^{(k)} = o_p(1)\).
Step 2: empirical process term. (The detailed calculation below may be skimmed on first reading; the two essential facts are that \(E_n^{(k)}\) has conditional mean zero given \(\hat\eta^{(-k)}\) and conditional variance bounded by \(\|f_k\|_{L_2}^2 / n_k = o_p(1/n_k)\).) Condition on \(\hat\eta^{(-k)}\). Since \(\hat\eta^{(-k)}\) is trained on \(\{1,\ldots,n\}\setminus\mathcal{I}_k\), the summands \(\bigl\{\varphi_{\mathrm{eff}}(O_i;\tau_0,\hat\eta^{(-k)}) - \varphi_{\mathrm{eff}}(O_i;\tau_0,\eta_0)\bigr\}\) for \(i\in\mathcal{I}_k\) are conditionally i.i.d. with conditional mean \(B_n^{(k)}\), which is removed by writing \(E_n^{(k)} = (\mathbb{P}_{n_k} - P)\{f_k\}\) as a centered empirical average of \(f_k = \varphi_{\mathrm{eff}}(\cdot;\tau_0,\hat\eta^{(-k)}) - \varphi_{\mathrm{eff}}(\cdot;\tau_0,\eta_0)\). The conditional variance of \(E_n^{(k)}\) satisfies \[\mathrm{Var}\bigl(E_n^{(k)}\mid\hat\eta^{(-k)}\bigr) = \frac{1}{n_k}\,\mathrm{Var}\bigl(f_k(O)\mid\hat\eta^{(-k)}\bigr) \leq \frac{1}{n_k}\,\|f_k\|_{L_2}^2.\] Under strong overlap and consistency (conditions (ii)–(iii)), the \(L_2\) norm satisfies \(\|f_k\|_{L_2}^2 = o_p(1)\), so \[\mathrm{Var}\bigl(\sqrt{n}\,E_n^{(k)}\mid\hat\eta^{(-k)}\bigr) = \frac{n}{n_k}\,\|f_k\|_{L_2}^2 = O(1)\cdot o_p(1) = o_p(1),\] using \(n/n_k = O(1)\) uniformly over the fixed number of folds, which holds because \(\min_k n_k/n\) is bounded away from zero by assumption. By conditional Chebyshev, \(\sqrt{n}\,E_n^{(k)} = o_p(1)\).
Conclusion. Combining Steps 1 and 2, \(\sqrt{n}\,R_n^{(k)} = o_p(1)\) for each \(k\), and hence \(\sqrt{n}\sum_{k=1}^K (n_k/n)\,R_n^{(k)} = o_p(1)\) since \(n_k/n\le 1\). The first term \(n^{-1/2}\sum_i\varphi_{\mathrm{eff}}(O_i;\tau_0,\eta_0)\) converges in distribution to \(N(0, \E[\varphi_{\mathrm{eff}}^2])\) by the ordinary CLT. Asymptotic linearity and asymptotic normality follow by Slutsky’s theorem. \(\square\)
12.10 Lab: Simulation Study of the DML Estimator
This lab uses simulation to illustrate the finite-sample behavior that motivates the DML estimator of Theorem 12.1. The central message of this chapter is that flexible nuisance estimation alone is not sufficient for valid inference: even when the nuisance estimators are consistent, same-sample score evaluation can leave a non-negligible empirical-process remainder at the root-\(n\) scale, so the resulting AIPW estimator may fail to be asymptotically linear with influence function \(\varphi_{\mathrm{eff}}\) — invalidating ordinary influence-function inference even where the point estimate remains consistent. Cross-fitting addresses this by removing the same-sample dependence in the empirical-process term of the remainder decomposition Equation 12.7, so that the term can be controlled by a conditional argument.
To avoid deliberate parametric misspecification, the three feasible estimators below use identical random-forest learners for the nuisance functions; the oracle estimator instead plugs in the true nuisance functions and serves only as an infeasible benchmark. The primary controlled comparison is between same-sample RF AIPW and cross-fitted RF AIPW — the only difference between them is whether the nuisance estimates are produced on the full sample (same data reused) or on held-out folds (cross-fitted) — with the single-split estimator added to separate held-out evaluation from fold rotation.
12.10.1 Data-Generating Process
Let \(n = 2000\) and generate covariates \(X_1, X_2, X_3 \overset{\mathrm{iid}}{\sim} \mathrm{Uniform}(-1, 1)\), collected as \(X = (X_1, X_2, X_3)^\top\). Define the true nuisance functions by \[\begin{aligned} \pi(X) &= \frac{\exp\!\bigl(\sin(\pi X_1) + X_2^2 - 0.5\bigr)}{1 + \exp\!\bigl(\sin(\pi X_1) + X_2^2 - 0.5\bigr)}, \\ \mu_1(X) &= 2\sin(\pi X_1) + X_2^2 + X_3, \\ \mu_0(X) &= \sin(\pi X_1) + X_2^2 - X_3. \end{aligned}\] The true ATE is \[\tau = \E\{\mu_1(X) - \mu_0(X)\} = \E\{\sin(\pi X_1) + 2X_3\} = 0,\] since \(X_1\) and \(X_3\) are symmetric about zero. Treatment and outcome are drawn as \[T_i \mid X_i \sim \mathrm{Bernoulli}\{\pi(X_i)\}, \qquad Y_i = \mu_{T_i}(X_i) + \varepsilon_i, \qquad \varepsilon_i \sim N(0, 1).\] The nonlinearity of \(\pi\) and \(\mu_t\) means that random forests are flexible enough to approximate them well in this simulation; formal \(L_2\) consistency would require additional conditions on the forest construction and the tuning sequence, which we do not verify here. The random-forest classes implemented in the lab are not covered by the classical Donsker sufficient conditions of Section 12.5 (we do not formally establish that the induced score classes are non-Donsker) — precisely the setting in which same-sample plug-in estimation lacks a supporting empirical-process argument and can become problematic.
12.10.2 The Four Estimators
- Oracle AIPW. Plug in the true nuisance functions \(\pi(X_i)\), \(\mu_1(X_i)\), \(\mu_0(X_i)\) directly. This estimator is infeasible but provides the semiparametric efficiency benchmark.
- Same-sample AIPW (often called “naive” AIPW). Estimate \(\pi\), \(\mu_1\), \(\mu_0\) using random forests on the full sample, then evaluate the AIPW score on the same sample. Data reuse means the empirical process term in Equation 12.7 is not controlled by any argument available to us: the fixed random-forest implementation used here is not covered by the classical Donsker sufficient conditions, and we make no formal claim that the fitted forests are \(L_2\)-consistent.
- DML (cross-fitted AIPW). Estimate \(\pi\), \(\mu_1\), \(\mu_0\) using the same random forest learners, but via \(K = 5\) fold cross-fitting. Training and evaluation folds are separated, so the empirical-process term is controlled by the conditional argument of Section 12.7 without any Donsker assumption on the nuisance class — under the \(L_2\)-consistency, overlap, and moment conditions of Theorem 12.1, which we do not verify for this implementation.
- Single-split AIPW. Randomly split the sample in half; fit the same random-forest learners on one half and evaluate the AIPW score on the other half only. This estimator enjoys the same independence argument as cross-fitting (Section 12.6) but uses only \(n/2\) evaluation points, so it isolates the role of held-out evaluation from the efficiency gain of rotating folds.
Estimators (2)–(4) use identical learners; the differences are data reuse versus held-out evaluation, and — between (3) and (4) — whether evaluation roles rotate over folds.
12.10.3 Implementation and Reproducibility
The simulation reported below was run in Python using scikit-learn random forests. The implementation uses fixed hyperparameters throughout; no nested cross-validation is used to tune the learners inside the cross-fitting loop. This keeps the comparison between the same-sample and DML estimators clean: any difference between them is attributable to cross-fitting alone, not to different tuning paths.
The pseudocode below describes one replication; the full procedure applies it \(n_{\mathrm{sim}}\) times and collects the four estimators per replication.
The reference implementation used to produce the numbers below, chapter12_lab.py, is available in the companion repository. For exact reproducibility, the repository records the Python and scikit-learn versions and the seed-derivation rule; “library defaults” are not stable across software versions, so version pinning is part of the specification. Readers can adapt the implementation to R (for example, using ranger for random forests and origami or a hand-coded KFold for the cross-fitting loop).
12.10.4 Simulation Results and Interpretation
Running the simulation with \(n = 2000\) and \(n_{\mathrm{sim}} = 200\) replications produces the point-estimation results below.
| Estimator | Bias | Monte Carlo variance | RMSE |
|---|---|---|---|
| Oracle AIPW | \(0.009\) | \(0.0029\) | \(0.055\) |
| Same-sample RF AIPW | \(0.039\) | \(0.0031\) | \(0.068\) |
| DML (\(K=5\)) | \(0.004\) | \(0.0036\) | \(0.060\) |
| Single-split AIPW | \(0.004\) | \(0.0084\) | \(0.092\) |
The simulation illustrates the asymptotic logic of cross-fitting: it reduces the bias created by same-sample nuisance fitting, but finite-sample performance still depends on the quality of the nuisance learners, the degree of overlap, and the sample size.
- Oracle AIPW is nearly unbiased with small variance, serving as the semiparametric efficiency benchmark.
- Same-sample RF AIPW exhibits a clear positive bias (0.039, roughly ten Monte Carlo standard errors from zero). This behavior is consistent with the mechanism analyzed in Section 12.5: reusing the same sample for nuisance estimation and score evaluation allows overfitting in the random forests to propagate into the AIPW score through the empirical process term Equation 12.8.
- DML reduces the bias to 0.004 — within Monte Carlo error of zero and indistinguishable from the oracle — with a slightly larger Monte Carlo variance (0.0036 versus 0.0031). The bias reduction is the intended effect of cross-fitting: by separating training and evaluation folds (Section 12.7), the empirical-process term is controlled by a conditional variance argument rather than by a Donsker assumption on the random-forest class, under the \(L_2\)-consistency, overlap, and moment conditions of Theorem 12.1. The smaller nuisance-training sample within each fold (\((K{-}1)n/K = 1600\) observations rather than \(n = 2000\)) and the additional randomness from fold assignment are plausible contributors to the variance difference, but the experiment does not identify a variance decomposition; in particular, the lower Monte Carlo variance of the same-sample estimator should not be attributed to overfitting without a separate analysis. Under the assumptions of Theorem 12.1, the first-order efficiency comparison between the two estimators is asymptotic rather than a general finite-sample variance ordering. The net effect here is nevertheless a lower RMSE for DML (0.060) than for same-sample RF AIPW (0.068): the bias reduction outweighs the variance difference.
- Single-split AIPW is also essentially unbiased (0.004): held-out evaluation alone removes the same-sample problem. But its Monte Carlo variance (0.0084) is more than twice DML’s, because only \(n/2\) observations contribute to the target average. The comparison between the single-split and cross-fitted estimators isolates the second role of cross-fitting: rotating the folds restores the full sample as evaluation points without giving up the independence argument.
12.10.5 Inference Performance
The chapter promises valid inference, not merely accurate point estimation, so the lab also evaluates the plug-in influence-function standard errors of Section 12.11. For each estimator and each replication we compute \(\hat V\) from Equation 12.14 (using in-sample plug-in score values for the same-sample estimator, out-of-fold values for DML, and evaluation-half values with denominator \(n_e = n/2\) for the single-split estimator), the nominal 95% Wald interval \(\hat\tau \pm 1.96\sqrt{\hat V}\), and its length.
| Estimator | aveSE | empSD | aveSE/empSD | Coverage | CI length |
|---|---|---|---|---|---|
| Oracle AIPW | \(0.057\) | \(0.054\) | \(1.06\) | \(0.960\) | \(0.223\) |
| Same-sample RF AIPW | \(0.045\) | \(0.056\) | \(0.81\) | \(0.790\) | \(0.178\) |
| DML (\(K=5\)) | \(0.066\) | \(0.060\) | \(1.10\) | \(0.950\) | \(0.257\) |
| Single-split AIPW | \(0.093\) | \(0.092\) | \(1.02\) | \(0.955\) | \(0.365\) |
Three features stand out. First, the same-sample estimator fails at the inferential level even more clearly than at the point-estimation level: its Wald intervals cover the truth in only \(79\%\) of replications. The failure has two reinforcing sources: the interval is centered at a biased estimate, and the estimated standard error is too small (\(\mathrm{aveSE}/\mathrm{empSD} = 0.81\)). The understatement is consistent with in-sample overfitting: fitting and evaluating the nuisance functions on the same observations can compress the residual and inverse-weight contributions appearing in the plug-in score, and the lower mean maximum absolute plug-in score value (\(8.4\) for the same-sample estimator versus \(13.6\) for the oracle) is a diagnostic consistent with this explanation, not a proof or a variance decomposition. Because the same-sample estimator has not been shown to be asymptotically linear with influence function \(\varphi_{\mathrm{eff}}\), its plugged-in quantities \(\varphi_{\mathrm{eff}}(O_i;\,\hat\tau,\hat\eta)\) should be read as plug-in score values rather than automatically as valid estimated influence values.
Second, DML delivers coverage at the nominal level (\(0.950\), ratio \(1.10\)), consistent with the Wald theory of Section 12.11. Third, in this experiment the single-split estimator also attains nominal coverage (\(0.955\), ratio \(1.02\)). Held-out evaluation controls the same-sample empirical-process remainder, but valid Wald inference additionally relies on the population-drift, nuisance-consistency, overlap, and moment conditions of Theorem 12.1; the simulation shows the procedure works here, not that held-out independence is sufficient in general. The wider single-split intervals (\(0.365\) versus \(0.257\), \(42\%\) longer than DML’s) reflect the fact that only the evaluation half contributes directly to the target average, together with any finite-sample effect of fitting the nuisances on the smaller training half. Held-out evaluation addresses the same-sample dependence problem; fold rotation recovers evaluation-sample efficiency; valid inference requires the full collection of conditions in Theorem 12.1.
12.10.6 Nuisance and Influence Diagnostics
Because the true nuisance functions are known in a simulation, the lab can also report the nuisance diagnostics that practice cannot observe. The reported nuisance errors are empirical root-mean-square errors against the true functions, computed at the observed covariate values: all \(n\) points with in-sample predictions for the same-sample estimator, the same \(n\) points with out-of-fold predictions for DML, and the \(n_e = n/2\) evaluation points for the single-split estimator. Each is a Monte Carlo approximation to the \(L_2(P_X)\) norm; the same-sample and DML errors are evaluated at identical design points, so their comparison is design-comparable, while the single-split errors use half the points.
| Estimator | \(\hat\pi\) | \(\hat\mu_0\) | \(\hat\mu_1\) | \(\hat\mu_1\!\cdot\!\hat\pi\) | \(\hat\mu_0\!\cdot\!\hat\pi\) | Clip | \(\max_i \lvert\hat\varphi_i\rvert\) |
|---|---|---|---|---|---|---|---|
| Oracle AIPW | – | – | – | – | – | – | \(13.6\) |
| Same-sample RF AIPW | \(0.139\) | \(0.363\) | \(0.374\) | \(0.052\) | \(0.050\) | \(0\) | \(8.4\) |
| DML (\(K=5\)) | \(0.107\) | \(0.347\) | \(0.372\) | \(0.040\) | \(0.037\) | \(0\) | \(26.6\) |
| Single-split AIPW | \(0.111\) | \(0.369\) | \(0.404\) | \(0.045\) | \(0.041\) | \(0\) | \(22.5\) |
Three lessons emerge. First, both error products for DML (\(0.040\) and \(0.037\)) exceed \(n^{-1/2} = 0.022\) at this sample size, so the Cauchy–Schwarz bound on the population drift is not small relative to the target root-\(n\) scale — and yet the realized bias of DML is negligible. This is a useful reminder that the bound is conservative: the exact drift is a signed bilinear integral in which cancellation can occur. It is equally a reminder that the product-rate condition is not verified here, consistent with the caveats below. Second, the same-sample in-sample fits show a larger empirical error against the true nuisance functions than DML’s out-of-fold predictions (\(0.139\) versus \(0.107\) for \(\hat\pi\)), which is consistent with overfitting toward the realized labels and away from the smooth truth, although the experiment does not isolate that mechanism. Third, the propensity clipping to \([0.01, 0.99]\) never binds — zero fitted propensities fall outside the interval across all estimators and all \(200\) replications — confirming that clipping plays no role in these results.
This finite-sample experiment illustrates the behavior motivating cross-fitting; it does not by itself verify the assumptions of Theorem 12.1. With the fixed random-forest implementation used here, we do not establish \(L_2\)-consistency, the product-rate condition, or membership outside every possible Donsker regime. The contrast between the same-sample and cross-fitted estimators is consistent with the removal of same-sample overfitting effects, but the simulation does not separately identify the population-drift term and the empirical-process term of the remainder decomposition Equation 12.7. In this particular DGP, the true logit lies in \([-1.5, 1.5]\), so the true propensity score is bounded in approximately \([0.18, 0.82]\); consistent with this, the clipping of \(\hat\pi\) to \([0.01, 0.99]\) never binds in the reported runs.
12.11 Variance Estimation and Confidence Intervals
Inference for \(\hat\tau_{\mathrm{DML}}\) proceeds exactly as in Chapter 11, except that the nuisance estimates are now out-of-fold. Cross-fitting changes how the nuisance functions are estimated, not the asymptotic variance formula itself; the variance is still the variance of the efficient score, estimated empirically using the out-of-fold score contributions. Let \(k(i)\) denote the fold containing observation \(i\), and define the estimated influence value as the score contribution \(\hat s_i\) from the algorithm box, evaluated at the solution \(\hat\tau_{\mathrm{DML}}\): \[\hat\varphi_i = \hat s_i(\hat\tau_{\mathrm{DML}}) = \varphi_{\mathrm{eff}}\!\bigl(O_i;\,\hat\tau_{\mathrm{DML}},\,\hat\eta^{(-k(i))}\bigr).\] A natural variance estimator is \[\hat{V} = \frac{1}{n(n-1)}\sum_{i=1}^n (\hat\varphi_i - \bar\varphi)^2, \qquad \bar\varphi = \frac{1}{n}\sum_{i=1}^n \hat\varphi_i. \tag{12.14}\] The Wald confidence interval at level \(1 - a\) is \(\hat\tau_{\mathrm{DML}} \pm z_{1-a/2}\sqrt{\hat{V}}\). (We write \(a\) for the significance level to avoid collision with other uses of \(\alpha\).)
Theorem 12.2 (Consistency of the Variance Estimator) Suppose the conditions of Theorem 12.1 hold and, in addition, the estimated influence values converge in the sense \[\frac{1}{n}\sum_{i=1}^n \bigl\{\hat\varphi_i - \varphi_{\mathrm{eff}}(O_i;\,\tau_0,\eta_0)\bigr\}^2 = o_p(1).\] Then the variance estimator \(\hat{V}\) in Equation 12.14 is consistent for the asymptotic variance of \(\hat\tau_{\mathrm{DML}}\); equivalently, \(n\hat{V} \overset{p}{\longrightarrow} \E[\varphi_{\mathrm{eff}}(O;\,\tau_0,\eta_0)^2]\). The Wald confidence interval \(\hat\tau_{\mathrm{DML}} \pm z_{1-a/2}\sqrt{\hat V}\) is therefore asymptotically valid.
Theorem 12.2 is deliberately stated in a general form: the estimated-influence-value convergence condition, not the particular estimator, is what drives variance consistency. For the cross-fitted AIPW estimator specifically, the condition is implied by the assumptions of Theorem 12.1 — strong overlap bounds the fitted inverse weights, cross-fitted \(L_2\) consistency gives \(\|f_k\|_{L_2} = o_p(1)\) fold by fold exactly as in Step 2 of the proof, and consistency of \(\hat\tau_{\mathrm{DML}}\) handles the \(\tau\)-argument, which enters the score linearly — so variance consistency for \(\hat\tau_{\mathrm{DML}}\) holds as a corollary under those assumptions alone.
From a practical standpoint, Equation 12.14 is the sample variance of the out-of-fold estimated influence values divided by \(n\), i.e., the usual plug-in estimate of the variance of a sample mean. The formula is identical to the one in Chapter 11; the only difference is that \(\hat\varphi_i\) now uses the out-of-fold nuisance estimate \(\hat\eta^{(-k(i))}\) rather than a full-sample nuisance estimate.
12.12 A Practical Workflow
The theory above leads to the following workflow for applied cross-fitted orthogonal-score estimation. The steps are grouped into three phases: identification and setup, estimation, and diagnostics and interpretation.
Step 11 is the most important step in the workflow. The core of identification — conditional exchangeability — sits outside what any algorithm can verify, and a careful discussion of identification often does more to make a causal estimate credible than any choice of nuisance learner. We return to this point in Section 12.13.
12.13 What Machine Learning Does Not Solve
Flexible nuisance estimation is a genuine improvement over parametric methods, but it does not resolve the fundamental difficulties of causal inference. In particular, machine learning does not solve:
- Unmeasured confounding. If important confounders are absent from \(X\), conditional exchangeability \(Y(t) \indep T \mid X\) fails. No amount of flexibility in estimating \(\pi(x)\) or \(\mu_t(x)\) can correct this.
- Ill-defined interventions, hidden versions, or interference. These problems invalidate the simple unit-level representation \(Y_i(t)\) and the corresponding ATE used in this chapter. They do not invalidate the potential-outcomes framework itself; rather, the intervention and estimand must be reformulated, for example by indexing \(Y_i\) by the treatment vector \(\mathbf{T}\) or by a specified exposure mapping.
- Violations of consistency. Even after the intervention is well defined, one must justify that the observed outcome equals the potential outcome corresponding to the treatment version actually received. No amount of flexibility in nuisance estimation bears on this claim.
- Exact positivity failure. If \(\pi(x) = 0\) or \(\pi(x) = 1\) on a set of positive probability relevant to the target population, the standard nonparametric ATE is not identified without extrapolation assumptions.
- Weak overlap or near-positivity. If \(\pi(x)\) is positive but approaches \(0\) or \(1\) in part of the covariate space, the ATE may remain identified, but influence values become unstable, the efficient variance can be very large or infinite, and coverage of Wald confidence intervals can degrade severely.
- Ambiguity about the estimand. Flexible estimation cannot resolve disagreement about what causal quantity is scientifically meaningful.
- Invalid instrumental variable assumptions. The exclusion restriction and exogeneity conditions (Chapter 7) are not testable from the data; machine learning cannot substitute for subject-matter justification.
- Fragile mediation assumptions. Identification of natural direct and indirect effects requires the sequential ignorability assumptions of Chapter 8 (Section 8.6) — treatment ignorability, mediator–outcome ignorability, and a cross-world independence — together with consistency, composition, and suitable positivity. A treatment-induced mediator–outcome confounder, whether measured or not, generically invalidates the cross-world independence and with it the standard mediation formula. Flexible nuisance learning does not supply any of these assumptions.
Machine learning improves the estimation of nuisance functions within a given identification strategy. It does not create identification where none exists. Machine learning can improve estimation after the causal problem has been formulated and identified, but it cannot repair a failure of identification assumptions.
12.14 Chapter Summary
- Flexible methods are attractive for nuisance estimation when the true nuisance functions are complex, but naive plug-in estimators can fail for two reasons: orthogonality may be absent, and even when orthogonality holds, same-sample nuisance fitting can leave an additional empirical-process remainder (Section 12.2).
- A score is Neyman orthogonal if the Gateaux derivative of its population moment with respect to the nuisance vanishes at the truth (Definition 12.1). Under suitable local smoothness, this removes the first-order nuisance term from the population moment drift; it does not by itself control the same-sample empirical-process remainder.
- The ATE score Equation 12.4 is orthogonal and doubly robust and is the efficient influence function in the maintained nonparametric model; a regular estimator that is asymptotically linear with this influence function attains the semiparametric efficiency bound. It serves as the orthogonal score throughout this chapter (Section 12.4).
- Even with an orthogonal score, reusing the same data for nuisance estimation and score evaluation can leave a non-negligible empirical-process remainder, especially when the induced score class falls outside classical fixed Donsker regimes (Section 12.5).
- Sample splitting separates nuisance fitting from score evaluation, allowing the empirical-process remainder to be controlled conditionally without a Donsker assumption, at the cost of using only the evaluation subsample for the target estimate (Section 12.6).
- Cross-fitting rotates training and evaluation roles across \(K\) folds, so each observation is evaluated only with nuisance estimates trained on other observations; this avoids the first-order information loss of a single split and permits efficiency under the conditions of Theorem 12.1 (Section 12.7).
- The cross-fitted AIPW estimator \(\hat\tau_{\mathrm{DML}}\) is the standard double machine learning estimator for the ATE (Section 12.8).
- Under the full conditions of Theorem 12.1 — i.i.d. sampling, fixed data-independent folds, moment conditions, \(L_2(P)\)-consistency, strong overlap with bounded fitted propensities, and the product-rate condition Equation 12.13 — orthogonality and cross-fitting together yield asymptotic linearity, root-\(n\) consistency, and asymptotic normality of \(\hat\tau_{\mathrm{DML}}\), with asymptotic variance equal to the semiparametric efficiency bound. A sufficient symmetric condition is that each nuisance estimator converges at rate \(o_p(n^{-1/4})\) in \(L_2(P)\) norm, equivalently that its squared \(L_2(P)\) error is \(o_p(n^{-1/2})\).
- The asymptotic variance is estimated consistently from the empirical variance of the out-of-fold estimated influence values (Theorem 12.2).
- Machine learning improves nuisance estimation, but it does not resolve identification problems: unmeasured confounding, positivity failures, and untestable structural assumptions remain the researcher’s responsibility (Section 12.13).
12.15 Exercises
1. Verifying orthogonality.
- Write out the efficient influence function \(\varphi_{\mathrm{eff}}(O;\,\tau,\eta)\) for the ATE and compute its expectation. Confirm it equals zero at the truth.
- Consider a perturbation \(\pi \mapsto \pi + r\cdot h\) for a bounded function \(h(x)\). Differentiate \(\E\{\varphi_{\mathrm{eff}}(O;\,\tau, \mu_0, \mu_1, \pi+rh)\}\) with respect to \(r\) and evaluate at \(r=0\). Show the derivative is zero, confirming Neyman orthogonality in the propensity score direction.
- Repeat the calculation for a perturbation in \(\mu_1\).
2. First-order bias of the plug-in estimator. Suppose \(\hat\mu_1 = \mu_1 + \delta\) for a deterministic function \(\delta(x)\), and \(\hat\mu_0 = \mu_0\), \(\hat\pi = \pi\).
- Show that the bias of the resulting prediction estimator \(\hat\tau_{\mathrm{pred}} = n^{-1}\sum_i \{\hat\mu_1(X_i) - \hat\mu_0(X_i)\}\) is \(\E\{\delta(X)\}\).
- Show that the population drift of the plug-in AIPW estimator (using \(\hat\mu_1 = \mu_1 + \delta\) with the true \(\pi\) and \(\mu_0\)) is zero. Explain which property of the AIPW score is responsible. Does this zero population drift guarantee that the same-sample AIPW estimator is root-\(n\) asymptotically linear with influence function \(\varphi_{\mathrm{eff}}\) when \(\hat\mu_1\) is estimated flexibly from the same sample used to evaluate the score? Explain why the empirical-process term must still be controlled.
3. Cross-fitting with \(K=2\). Partition a sample of \(n=200\) observations into two halves \(\mathcal{I}_1\) and \(\mathcal{I}_2\).
- Write down the cross-fitted moment equation Equation 12.11 explicitly for \(K=2\).
- Explain why the contribution from \(\mathcal{I}_1\) uses nuisance estimates trained on \(\mathcal{I}_2\) and vice versa.
- Compare this procedure to single sample splitting with \(\mathcal{I}_1\) as the training fold. What is gained and what is lost?
4. The product-rate condition. Suppose \(\|\hat\pi - \pi\|_{L_2} = o_p(n^{-\alpha})\) and \(\|\hat\mu_t - \mu_t\|_{L_2} = o_p(n^{-\beta})\) for \(\alpha, \beta > 0\).
- State the condition on \(\alpha\) and \(\beta\) under which the product-rate condition Equation 12.13 holds.
- Show that \(\alpha = \beta = 1/4\) satisfies your condition.
- Now suppose that the propensity score is estimated at the parametric rate, \(\|\hat\pi - \pi\|_{L_2} = O_p(n^{-1/2})\), while the outcome model is only known to be consistent, \(\|\hat\mu_t - \mu_t\|_{L_2} = o_p(1)\) (no rate is specified). Show that the product-rate condition Equation 12.13 still holds. What does this say about the standard situation in which the propensity score is estimated by a parametric logistic regression while the outcome model is fit by a flexible learner?
5. Variance estimation. Using the explicit formula Equation 12.12, write out the estimated influence value \(\hat\varphi_i\) for a generic observation \(i \in \mathcal{I}_k\). Show that \(n^{-1}\sum_i \hat\varphi_i = 0\) exactly when \(\hat\tau_{\mathrm{DML}}\) solves the cross-fitted moment equation, and explain why the centering in Equation 12.14 is therefore numerically inconsequential.
6. What machine learning cannot do. For each of the following scenarios, identify the identification assumption that is violated and explain why flexible nuisance estimation cannot remedy the problem.
- A study of job-training effects on earnings omits pre-program earnings, which is a pretreatment common cause of program participation and subsequent earnings and is not otherwise captured by the measured covariates \(X\).
- In a study of a medical treatment assigned within hospital wards, a patient’s outcome may depend on both that patient’s own treatment and the proportion of other patients treated in the same ward. Which component of SUTVA fails? How does this differ from ordinary within-ward correlation of outcomes that does not represent treatment interference?
- An instrument is used to estimate the effect of education on wages, but the instrument also directly affects wages through a channel not related to education.
7. Cross-fitting leakage. A researcher creates five folds but selects variables, tunes hyperparameters, and standardizes covariates using the full sample before fitting the fold-specific nuisance models. Does the independence argument that justifies cross-fitting (Section 12.7) remain valid? Identify which operations must be repeated within each training complement, and explain what goes wrong in the conditional variance argument of Step 2 of the proof of Theorem 12.1 when they are not.
8. Coverage experiment (optional computational project). Using the reference implementation chapter12_lab.py (record your software versions; results vary slightly across library releases). A full run of parts (a)–(c) involves thousands of random-forest fits — on the order of 10–15 minutes per configuration at \(n = 2000\) on a single core, and several times longer at \(n = 8000\) — so reducing \(n_{\mathrm{sim}}\) to \(100\) and reporting the correspondingly larger Monte Carlo standard errors is acceptable, as is completing a single part.
- Rerun the simulation with \(n = 500\) and \(n = 8000\), keeping \(K = 5\). Report the point-estimation and inference tables for each \(n\), including the Monte Carlo standard error of each coverage estimate. How do the same-sample coverage and the DML aveSE/empSD ratio change with \(n\), and why?
- At \(n = 2000\), rerun DML with \(K \in \{2, 10\}\). Relate any changes in bias and interval length to the effective nuisance-training sample size \((K{-}1)n/K\).
- The out-of-fold nuisance error products reported in the diagnostics table exceed \(n^{-1/2}\), yet DML’s realized bias is negligible. Reconcile these two facts using the exact drift identity in Step 1 of the proof of Theorem 12.1 and the role of Cauchy–Schwarz.