Classification — Cost Function
A recap of Andrew Ng course (Machine Learning Specialization) on Coursera
Course 1 Week 3
Cost Function for logistic regression
cost function gives you a way to choose better parameters.
if we were to use the squared error cost for logistic regression, we would get a non-convex shape, and we will get lots of local minima that we will get stuck in. this is because logistic regression got a non-linear factor, sigmoid function
we use another cost function so that the cost function shape will be convex again. let’s call the part inside the blue box loss, and the summation of losses will be the cost function.
Definition Note: In this course, these definitions are used:
Loss is a measure of the difference of a single example to its target value while the
Cost is a measure of the losses over the training set
The loss function, 𝑙𝑜𝑠𝑠(𝑓𝐰,𝑏(𝐱(𝑖)),𝑦(𝑖)), takes in two parameters, the logistic regression output and the true label y. the loss function outputs -log(f(x)) when the true label is 1 and outputs -log(1-f(x)) when true label is 0. the feature of this loss function is that it uses two separate curves.
Get Aimyo’s stories in your inbox
Join Medium for free to get updates from this writer.
if we plot the shape of log(f) and -log(f), the two functions intersect when f = 1. and since logistic regression always output values in between 0 and 1, so we can focus on the pink zone of -log(f). when true label is 1, we will see the loss shape as the plot in the lower left corner.
the second part of the loss function corresponding to true label being 0. we are still using the -log(f) function to visualize the loss. but we flip the curve, so that when the logistic regression output (f) is closer to one, 1-f should be closer to 0, thus we penalize the wrong output.
we can put the two curves side by side for easy read. the curves are similar to the quadratic curve of the squared error loss, so it is now a convex function and we can use gradient descent to find the best parameters.
we can also re-write the loss function so that it could be written in just one line
Remember that y can have only two values, 0 and 1. so we can consider this equation in two pieces.
we can visualize the logistic loss function, it does not have plateaus, local minima, or discontinuities.
we combine all losses across all data points and divide it by the total data count to get the final cost for the dataset.
this cost function was derived from a statistical principle, maximum likelihood, which is an idea from statistics on how to efficiently find parameters for different models.
We can find the best parameters by applying gradient descent.
we repeat the derivative part for each feature from 1 to N. We do these updates simultaneously, meaning we do the equations on the right first and then simultaneously update all the values in W and B
we can monitor gradient descent at each execution to see if the model converges. also, applying feature scaling so that different features take on similar ranges of values will speed up gradient descent for logistic regression
