Recall that Newton's update is given by $\theta^\ast=\theta_0-H^{-1}\nabla_{\theta}J(\theta_0)$ where $H$ is the Hessian of $J$ with respect to $\theta$ evaluated at $\theta_0$.

The approach adopted by the BFGS algorithm is to approximate the inverse with a matrix $M_t$ that is iteratively refined by low-rank updates to become a better approximation of $H^{-1}$.

Once the inverse Hessian approximation $M_t$ is updated, the direction of descent $\rho_t$ is determined by $\rho_t=M_tg_t$.

A line search is performed in this direction to determine the size of the step, $\epsilon^{\ast}$, taken in this direction. The final update to the parameters is given by: $\theta_{t+1}=\theta_t + \epsilon^{\ast}\rho_t$.