손실 함수, 경사 하강법, 역전파

Loss function(손실 함수)

- Loss function is a metric that shows how bad is the model prediction.

- [0, INF]

- Binary Cross-Entropy Loss(Log Loss), 2개의 카테고리로 구분 시 이용

- Negative log likelihood(NLL), 2개 이상의 카테고리로 구분 시 이용

 

Gradient Descent(경사 하강법)

- Gradient: Slope of a function for a given point

- Descent: Going down

 

- 손실 함수의 최솟값을 찾기 위해 경사 하강법을 이용한다.

  x좌표의 차이가 매우 작은 두 점의 좌표를 이용해, 현재의 기울기를 구한다. 

 

- Amount of change is decided by gradient magnitude and learning rate. (proportion, 비례)

   

- What if we apply too high learning rate?

   It will oscillate or diverge rather than converge.

 

- What if we apply too low learning rate?

   1. Take too many updates to arrive minima

   2. Can be stuck into local minima

 

- Step of Gradient Descent

1. Initialize model parameters at random

2. Calculate loss in current state

3. Find a direction to reduce the loss(Gredient)

4. Change the parameter values slightly to the direction

5. Repeat 2 - 4 

 

https://angeloyeo.github.io/2020/08/16/gradient_descent.html

경사하강법(gradient descent) - 공돌이의 수학정리노트

Error BackPropagation(역전파)

- Each neuron sends back its current error to the its input to tell whether the value in the previous step has to be increased or decreased.

 

- For every parameter, we can calculate the gredient based on the input of parameters and error of its input.

 

- The gradient of each parameter is calculated by Backpropagation.

 

- After the calculation, we update the entire parameter simultaneously.

 

https://re-code-cord.tistory.com/entry/%ED%95%B4%EB%AC%BC%ED%8C%8C%EC%A0%84%EB%A7%90%EA%B3%A0-%EC%97%AD%EC%A0%84%ED%8C%8C

딥러닝의 핵심, 역전파