Reward-Weighted Probability Distribution

Consider a scenario where for a given input $\mathbf{x}$, there are only two possible outputs, $\mathbf{y}_1$ and $\mathbf{y}_2$. A reference model assigns probabilities $\pi_{\text{ref}}(\mathbf{y}_1|\mathbf{x}) = 0.6$ and $\pi_{\text{ref}}(\mathbf{y}_2|\mathbf{x}) = 0.4$. A reward function gives scores $r(\mathbf{x}, \mathbf{y}_1) = 2$ and $r(\mathbf{x}, \mathbf{y}_2) = 1$. Assuming the scaling factor $\beta$ is 1, what is the value of the normalization factor $Z(\mathbf{x})$, which is calculated as $Z(\mathbf{x}) = \sum_{\mathbf{y}} \pi_{\text{ref}}(\mathbf{y}|\mathbf{x}) \exp(r(\mathbf{x}, \mathbf{y}))$?

Consider the calculation of a normalization factor using the formula: $Z(\mathbf{x}) = \sum_{\mathbf{y}} \pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x}) \exp \left(\frac{1}{\beta}r(\mathbf{x}, \mathbf{y})\right)$ If the reward function $r(\mathbf{x}, \mathbf{y})$ consistently returns a value of 0 for all possible outputs $\mathbf{y}$, the normalization factor $Z(\mathbf{x})$ will always be equal to 1.

Impact of Scaling Factor on Normalization