7. Numerical Derivatives with Finite Difference Approach

Taylor Expansion of a Point

If we apply a small difference to $x$ value such as $h \sim 10^{-6}$, one can approximate the value of $f(x+h)$ by employing Taylor series in terms of $f(x)$ as: $$f(x+h) \approx f(x) + h f^{\prime}(x) + \frac{h^2}{2} f^{\prime \prime}(x) + \frac{h^3}{6} f^{\prime \prime \prime}(x) + \ldots$$

First Forward Difference Approximation

Only the first one or two terms have a significant effect in the Taylor expansion given above. Therefore the first derivative of a function can be written as follows: $$f^{\prime}(x) = \frac{f(x+h) - f(x)}{h}$$ This approximation's name is "First Forward" because of using only the first term after $f(x)$ in the forward direction regarding to the x-axis.

First Backward Difference Approximation

Similar to the "First Forward", if the difference is used as $-h$, this time the approximation gets the name as "First Backward". The equation can be found by using Taylor expansion again. $$f(x-h) \approx f(x) - h f^{\prime}(x)$$ Therefore the first derivative of a function can be written as follows: $$f^{\prime}(x) = \frac{f(x) - f(x-h)}{h}$$

Central Difference Approximation

For a more precise approximation, one can use the both directions and still keep the difference $h$. In such approximation the points will be $f(x-h/2)$ and $f(x+h/2)$ and the first derivative of the function: $$f^{\prime}(x) = \frac{f(x+\frac{h}{2}) - f(x-\frac{h}{2})}{h}$$

Custom Approximations

According to the situation that is raised by the definition of the function, such as sharp extremum points, one may want to increase the number of points.

Three Point Forward Approximation with the points $f(x)$, $f(x+h)$, $f(x+2h)$: $$f^{\prime}(x) = \frac{-3 f(x) + 4 f(x+h) - f(x+2h)}{2h}$$

Three Point Backward Approximation with the points $f(x-2h)$, $f(x-h)$, $f(x)$: $$f^{\prime}(x) = \frac{3 f(x) - 4 f(x-h) + f(x-2h)}{2h}$$

Four Point Central Approximation with the points $f(x-2h)$, $f(x-h)$, $f(x+h)$, $f(x+2h)$: $$f^{\prime}(x) = \frac{f(x-2h) - 8 f(x-h) + 8 f(x+h) - f(x+2h)}{12h}$$

Note: Do these points always have to be in the same direction?