convex-functions

Practice with Convex Functions

Part A

Please write a careful mathematical proof of the following statement. You may cite without proving any of the statements from the reading up to page 7. As suggested in the reading, Prop 2. (the second derivative test) is likely to be especially useful.

Claim: Let \(f:\mathbb{R}\rightarrow\mathbb{R}\) be any twice-differentiable and convex function. Then, the function \(g(x) = f(ax + b)\) is also convex for any constants \(a\) and \(b\).

Part B

The logistic sigmoid \(\sigma\) is the function \(\sigma: \mathbb{R} \rightarrow \mathbb{R}\) given by the formula \[ \begin{aligned} \sigma(z) = \frac{1}{1 + e^{-z}}\;. \end{aligned} \]

B.1

Use matplotlib with either numpy or torch to make a graph of this function on the interval \(z \in [-1, 1]\).

B.2

Compute \(\frac{d\sigma(s_0)}{ds}\).

\(\frac{d\sigma(s_0)}{ds}\) is the first derivative of \(\sigma\) evaluated at the point \(s_0\). You may need to recall things like the chain and quotient rules.

B.3

Using your computation of \(\frac{d\sigma(s_0)}{ds}\) from the previous part, check that the following formula holds. You should do so by computing both sides of the equation and verifying that they are equal.

\[ \begin{aligned} \frac{d\sigma(s_0)}{ds} = \sigma(s_0)\left(1 - \sigma(s_0)\right)\;. \end{aligned} \]

B.4

Using any methods from the reading, write a careful proof that the following two functions are convex:

\(f(s) = - \log \sigma(s)\)
\(g(s) = - \log (1 - \sigma(s))\)

B.5

Let \(y \in \{0,1\}\). Explain why the function

\[ \ell(s, y) = - y \log \sigma(s) - (1 - y) \log(1-\sigma(s)) \]

is convex as a function of \(s\).