About the gradient descent

[queqichao]

Hi, I think your code is very useful. But 'l-bfgs' seems to out perform 'sgd' consistently, which seems counter-intuitive to me. One thing I have in my mind is for 'sgd' it does not include the momentum to accumulate the past gradients. I would like to add that into your code and maybe try to merge it to your code. Is that ok to you?

[IssamLaradji]

Hi @queqichao , l-bfgs uses the hessian (the second derivative) to compute the direction to which the parameters update. It uses more information about the function than SGD for the update, so intuitively it makes sense that l-bfgs performs better in general.

I implemented one for scikit-learn and I believe it would be better if you could help Andy complete the finishing touches here (scikit-learn/scikit-learn#3939) :).

Thanks!! :)

[queqichao]

Thanks for pointing out it. They seems to have added the function I am interested. Hope it to be release soon.

About the l-bfgs and sgd, actually I feel that l-bfgs is prone to converge to a local minimum quickly, while a properly tuned sgd could converge to a better model than l-bfgs in many case. Of course in theory there's no any guarantee about this.