Gradients via Reverse Accumulation

July 14th, 2010

We extend the ideas of from Automatic Differentiation with Scala to include the reverse accumulation. Reverse accumulation is a non-obvious improvement to automatic differentiation that can in many cases vastly speed up calculations of gradients.
As the tables, diagrams and equations do not translate well into HTML, our full article is available here in PDF: http://www.win-vector.com/dfiles/ReverseAccumulation.pdf.

The purpose of our article is to explain reverse accumulation automatic differentiation clearly (and to release some sample code and timing results). A side effect of the article is to make sense of the following two diagrams:

If the following is picture of standard or forward differentiation:

cutFwd.png

then the following is a picture of reverse accumulation:

cutRev.png


Example code now distributed from: github.com/WinVector/AutoDiff.



Be Sociable, Share!
Comments are closed.