Two features $$x_i$$ and $$x_j$$ are considered statistically independent if the occurrence or value of one does not affect the probability of the other. Mathematically, this is expressed as $$p(x_i|x_j) = p(x_i)$$.

San Diego State University

Google

1. _Sample Space_ is set of all values that observation x can take.
2. _PDF_ (Probability Density Function) is the function mapping point from sample space to the number between 0 and 1. Area under PDF is equal to 1.
3. _Parametric Models_ are density functions having finite number of parameters.
4. _Likelihood _ is defined as $L(\theta | x) = p_{\theta}(x)$
5. _Maximum Likelihood Estimation_ is defined as $\theta = argmax L(\theta | X)$.
It is used to measure the set of parameters that are more apt to explain observed data X.

Quick Recap For Some Probability Concepts

$ \theta_{kl}= p(x_k = l)$
$ \theta_kl = \frac {n_{kl}}{N}$ 
Number of times feature k takes value l in dataset




Maximum Likelihood

The formula $\mathcal{L}(\mathbf{p}_{i+1}^{\theta}, \mathbf{p}_{i+1}^{\text{gold}})$ represents a loss function that quantifies the difference between a model's predicted probability distribution, $\mathbf{p}_{i+1}^{\theta}$, and the ground truth or "gold" probability distribution, $\mathbf{p}_{i+1}^{\text{gold}}$. The predicted distribution is parameterized by $\theta$, which are the model's parameters that are updated during training. The goal of training is typically to minimize this loss, thereby making the predicted distribution as close as possible to the true distribution. The subscript $i+1$ suggests this is often used in sequential contexts, like predicting the next element in a sequence.

Learn Before

Related