A method of supervised classification that takes many features, assumed to be independent, and uses Bayes' Theorem to determine given those independent variables how likely the likelihood of another event occurring.

It's called "Naive" because it assumes that features are conditionally independent, given the class, i.e., for all instances of a given class, the features have little or no correlation with each other.

(Naive) Bayes Classifier

According to Bayes' theorem, we have
$$
P\left(Y=k | X=x\right)=\frac{P\left(Y=k\right) P\left(X=x | Y=k\right)}{\sum_{l=1}^{K} P\left(Y=l\right) P\left(X=x | Y=l\right)}\\
=\frac{\pi_{k} f_{k}(x)}{\sum_{l=1}^{K} \pi_{l} f_{l}(x)}.
$$
In LDA with the assumption of Gaussian distribution in every class, we have the Gaussian density 
$$
f_k(x)=\frac{1}{(2 \pi_k)^{p / 2}|\boldsymbol{\Sigma}|^{1 / 2}} \exp \left(-\frac{1}{2}(x-\mu_k)^{T} \boldsymbol{\Sigma}^{-1}(x-\mu_k)\right)
$$
for a $p$-dimensional random vector $X$ with the distribution $N(\mu_k, \mathbf{\Sigma})$. All we need to do is to plug $f_k(x)$ into Bayes' theorem above and find out the $k$ which maximize $P\left(Y=k | X=x\right)$. LDA is a Bayes classifier with the assumption of Gaussian distribution in every class.

Using Bayes' theorem in LDA

Bayesian networks were developed to understand reasoning under conditions of uncertainty. It is based on Bayes Theorem and is a probabilistic model that represents a set of variables and their conditional dependencies by being able to find the probability of facts being true/false given already defined facts. A Bayesian network is represented by a directed acyclic graph which helps to visualize the probabilistic model for reviewing relationships between random variables and reasonings for causal probabilities given certain facts



Bayesian Networks

Example 1: Imagine a billiard table, and suppose that the billiard track bounces off the table many times, so we don't know where the ball will end up. Let's say the billiard table is L, and the probability that the ball ends up x feet from the edge is x/L.

The inverse probability problem: Suppose we observe that the ball's final stop is a foot away from the table, but we don't know the length of the table, L.

Bayes' main work is to break this cognitive asymmetry and propose a method to estimate the inverse probability.


Inverse-probability: break cognitive asymmetry 

This example in the book is a discussion on how we can update our beliefs using Bayes's rule.  The question the example tries to address is, "What is the probability that she has breast cancer, given that the test came out positive?", which can be expressed as P(D | T), where D is the hypothesis, disease, and T is the evidence, test.

$$
        $$(Updated probability of D) = P(D | T) = (likelihood ratio)) × (prior probability of D)$$
      $$

(Note) The "likelihood ratio" is given by P(T | D)/P(T). 



Mammogram (Breast Cancer Screening Example)

There are three out of four binary classification metrics mentioned in the Book of Why; true positive, false positive, and false negative.  The fourth one that is not mentioned in the book is true negative.  The following are the two primary building blocks of such four metrics: 

1. True vs. False: These are about prediction's correctness whether a prediction outcome was made accurately/ correctly.  The correct prediction is True, and the incorrect prediction is False.
2. Positive vs. Negative: These are about a binary classification -(e.g.) In the mammogram example in the book, "positive" is positive test results (breast cancer), whereas "negative" is negative test results (not breast cancer).  

The combinations of the above concepts result in True positive, True negative, false positive, and false negative, each of which is explained in the subsequent nodes to follow.  


Binary Classification Metrics

Inverse probability, as supposed to forward probability, is a conditional probability that a hypothesis (cause), H, is true given that the evidence (effect), E is true.  Inverse probabiity is harder to estimate than estimating a forward probability.  

It can be expressed as:

	 $$ P(H | E) $$


Inverse Probability

Forward probability, as supposed to inverse probability, is the conditional probability that an evidence (effect), E, is true given that the hypothesis (cause), H is true.  For the definition of inverse probability, see the node, "Inverse Probability Defined." 

It can be expressed as:

  $$P(E | H)$$

Estimating the forward probability is easier than estimating the inverse probability because, as long as we know the cause, it is easy to estimate the probability of the effect.  

Pearl calls forward probability, calculating conditional probability in the direction, in which "our judgment is more reliable."


Forward Probability

Bayes' Theorem can equivalently be written using the definition of joint probability: $$P(A \mid B) = \frac{P(A \cap B)}{P(B)}$$ This follows directly from the definition of conditional probability: the probability that event $$A$$ occurs given that $$B$$ has occurred equals the probability both events occur together, divided by the probability that $$B$$ occurs. If $$A$$ and $$B$$ are independent, then $$P(A \cap B) = P(A) \cdot P(B)$$, so the equation simplifies to: $$P(A \mid B) = \frac{P(A) \cdot P(B)}{P(B)} = P(A)$$ This confirms that for independent events, knowing that $$B$$ occurred gives no information about $$A$$.

Conditional-Probability Form of Bayes' Theorem

Bayes Theorem is about predicting the probability of an event A occurring given an event B has already occurred. We can use the inverse probability of this statment and the probability of each event to find this value. 
P(A) is called prior probability, P(B|A) is Likelihood, P(B) is evidence, and P(A|B) is posterior probability.


Johns Hopkins University

University of Michigan - Ann Arbor

Causal inference is the process of drawing conclusions about the causal relationships between variables based on observational data or experimental designs. Unlike association or correlation, which only describe how variables vary together, causal inference seeks to determine whether and how a change in one variable directly produces a change in another.

Causal Inference

https://medium.com/@seevee/machine-learning-bayes-theorem-2f48c33d51e5

Bayes Theorem in Machine Learning

A machine passes this test if a human could perceive their interactions with the machine as human, indicating that the machine has successfully emulated human reasoning. 

Turing Test

- Causal Inference Books
- Causal Inference Papers

Causal Inference References

The calculus of causation is formed of:
1. Causal diagrams (the language of knowledge)
2. Causal symbolic language (the language of queries)

The calculus of causation

Bayes Theorem Overview

Most of the statistical methods are entirely objective, which means reasoning exclusively from data and experiments. But causal analysis and Bayesian analysis are different in that they involves some subjectivity.

Causal analysis requires some subjective commitments based on one's knowledge in the field before the causal digram is drawn and data is explored.

Similarly, Bayesian analysis combines objective evidence with the prior subjective belief.

From objectivity to subjectivity

Logical reasoning is the process of using a rational, systematic series of steps based on sound certain mathematical procedures and statements to arrive at a conclusion. 

There are three types of the reasoning:
- Abductive Reasoning 
- Deductive Reasoning 
- Inductive Reasoning

Reasoning

Hill's Criteria is a collection of nine informal guidelines ("viewpoints" as supposed to requirements) used to determine whether a causal relationship exists within an observed association.  Initially, five out of the nine viewpoints were proposed by the surgeon general's advisory committee, to which Bradford Hill later added four more.  Hill's Criteria is not meant to be the necessary conditions to support or refute a cause-and-effect hypothesis.  The debate over the causality between cigarette and lung cancer prompted the criteria, which were used for the proponent of the causal relationship to justify their claim that cigarette smoking is indeed causally related to lung cancer.  

Hill's Criteria

1. necessary causation (but-for causation)
2. sufficient causation
3. necessary-and-sufficient causation

Three different kinds of causation

1. The Law of Counterfactuals
2. The Law of Conditional Independence

The Two Fundamental Laws of Causal Inference

A Randomized Controlled Trial (RCT) or a Controlled Experiment is a method to eliminate unmeasurable confounders effect on causal inference by implementing an intervention on a randomly selected group of individuals (treatment/intervened group) while keeping other individuals unchanged (controlled group, placebo, not-intervened group) to observe the changes in outcomes due to the intervention effect. The RCTs are intended to "disarm" confounders (confounding variables).

Randomized Controlled Trial (RCT) = Controlled Experiment

List of techniques that help solve intractable inference problems in machine learning that usually arise from interactions between latent variables in a structured graphical model.

-Expectation Maximization

-MAP Inference and Sparse Coding

-Variational Inference and Learning

-Learned Approximate Inference



Approximate Inference 

"The estimand is the target quantity—the purpose of the statistical analysis ... the thing we are estimating" " that connects statistical evidence to theory."

It refers "to both the theoretical and the empirical estimands."

Estimand

1- "Set a **theoretical estimand**, clearly connecting this quantity to theory"

2- "Link to an **empirical estimand**, which is informative about the theoretical estimand under some identification assumptions"

3- Use "an **estimation strategy** to learn the empirical estimand (e.g., a regression model) ... from data."

Three Critical Choices in Causal Inference

- Correlation does not imply causation.
- Causation does not imply correlation either.

Correlation vs. Causation

Establishing causality in economics is a significant challenge for two main reasons. First, unlike in many natural sciences, it is often impossible to conduct controlled experiments to isolate a single variable. The economy is a dynamic system where numerous factors change simultaneously, creating confounding variables that can lead to misleading correlations. Second, the economy's fundamental complexity, arising from the interactions of millions of people, makes it impossible to measure and understand every variable. This inherent complexity means economists must be skeptical of simple claims that one complex phenomenon, like capitalism, is the direct cause of another, such as a rise in living standards. To address these challenges, economists use methods like natural experiments and advanced statistical techniques to infer causal relationships from observational data.

The Challenge of Establishing Causality in Economics

Instrumental Variables (IV) estimation is a causal-inference method used when a regressor of interest is endogenous, i.e., correlated with the error term. An instrument is a variable that (i) is correlated with the endogenous regressor (relevance) and (ii) affects the outcome only through that regressor (exclusion restriction). Under these conditions, IV recovers a consistent estimate of the causal effect that ordinary least squares cannot.

Instrumental Variables Estimation

An encouragement design is a randomized experiment in which units are randomly assigned an encouragement to take a treatment rather than the treatment itself. Because compliance is partial, the random encouragement serves as an instrument for the realized exposure: it satisfies random assignment, is associated with uptake (relevance), and is assumed to affect the outcome only through that uptake (exclusion). The encouragement-design framework links the intention-to-treat effect of encouragement to the causal effect of the treatment among compliers, providing the standard bridge between randomization and instrumental-variables identification.

Encouragement Design (Randomized Encouragement)

Heteroskedasticity-consistent (HC) standard errors are 'sandwich' variance estimators for regression coefficients that remain valid when the error variance is not constant across observations. Introduced by White (1980), the basic HC0 estimator replaces the usual homoskedastic variance with $$\widehat{V}=(X^\top X)^{-1}\left(\sum_i \hat e_i^2\, x_i x_i^\top\right)(X^\top X)^{-1}$$, using squared OLS residuals $$\hat e_i$$. This delivers asymptotically correct standard errors under arbitrary heteroskedasticity without requiring the analyst to model the variance structure.

Heteroskedasticity-Consistent (HC) Standard Errors

The Intent-to-Treat (ITT) effect is the average causal effect of being **assigned** to a treatment condition, regardless of whether the assigned unit actually complied with or received the treatment. In a randomized experiment with outcome $$Y$$ and binary assignment $$Z\in\{0,1\}$$, the ITT effect is $$\mathrm{ITT}=\mathbb{E}[Y\mid Z=1]-\mathbb{E}[Y\mid Z=0]$$. Because $$Z$$ is randomized, ITT is unbiased for the effect of the **offer/assignment** itself, but under partial compliance it underestimates the effect of the treatment when actually received.

Intent-to-Treat (ITT) Effect

The Treatment-on-the-Treated (TOT) effect, also called the average treatment effect on the treated (ATET), is the average causal effect of **actually receiving** the treatment among the subpopulation that took it up. Under one-sided non-compliance (no-shows) with random assignment $$Z$$ to treatment $$D$$, Bloom (1984) shows it can be estimated as $$\widehat{\mathrm{TOT}}=\dfrac{\mathbb{E}[Y\mid Z=1]-\mathbb{E}[Y\mid Z=0]}{\Pr(D=1\mid Z=1)}$$, i.e., the ITT scaled by the compliance rate. With two-sided non-compliance and a monotonicity assumption, this is generalized to the Local Average Treatment Effect (LATE) among compliers (Imbens & Angrist, 1994).

Treatment-on-the-Treated (TOT) Effect

When a randomized experiment has imperfect compliance, two distinct causal estimands separate: the **Intent-to-Treat (ITT)** effect captures the impact of **assignment**, while the **Treatment-on-the-Treated (TOT)** effect captures the impact of **actually receiving** the treatment among takers. Under randomization, ITT is unbiased for the effect of the offer but is attenuated toward zero relative to the effect of exposure as compliance shrinks. TOT can be recovered from ITT by scaling by the compliance rate (Bloom, 1984) or, more generally, by instrumental-variables estimation using assignment as an instrument for realized exposure, which identifies the LATE among compliers under monotonicity (Imbens & Angrist, 1994; Angrist, Imbens & Rubin, 1996). Reporting ITT alone hides the effect of exposure; reporting TOT/IV alone hides what happens at scale when many assigned users do not comply. The distinction is what justifies running an IV on a randomized default rather than just reporting the assignment-level contrast.

Intent-to-Treat vs. Treatment-on-the-Treated (Compliance-Adjusted Effects)

The ladder of causation classifies causal reasoning in organisms and machine learning into three levels and corresponding abilities: association (seeing), intervention (doing), and counterfactual reasoning (imagining).

Ladder of Causation: Association, Intervention, and Counterfactuals

An estimation strategy specifies the method or model used to estimate an empirical estimand from observed data; for example, it may use a regression model.

Estimation Strategy in Causal Inference

Inductive reasoning moves from empirical observations to broader generalizations or hypotheses. Deductive reasoning starts with a hypothesis or general premise and derives predictions that can be tested against observations. Used together, they form an iterative cycle in causal inquiry: evidence motivates hypotheses, and deductions from those hypotheses guide further empirical testing.

Learn Before

Related

Learn After