this is a work in progress…please view later.
I shall discuss about various optimizers. First, as usual, I would provide source of all original material and encourage you to read the original content instead of this blog. Next, I shall introduce each optimizer, the mathematical framework, from what I know about them. Then I shall compare and analyse them in greater detail to have some basic understanding.
Maybe someday I shall develop greater insights.
Credits: https://arxiv.org/pdf/1803.08494