07-28-2023-1204 - In mathematics and computer algebra, automatic differentiation (auto-differentiation, autodiff, or AD), also called algorithmic differentiation, computational differentiation,[1][2] is a set of techniques to evaluate the partial derivative of a function specified by a computer program. (DRAFT)

From Wikipedia, the free encyclopedia

In mathematics and computer algebra, automatic differentiation (auto-differentiation, autodiff, or AD), also called algorithmic differentiation, computational differentiation,^[1][2] is a set of techniques to evaluate the partial derivative of a function specified by a computer program.

Automatic differentiation exploits the fact that every computer calculation, no matter how complicated, executes a sequence of elementary arithmetic operations (addition, subtraction, multiplication, division, etc.) and elementary functions (exp, log, sin, cos, etc.). By applying the chain rule repeatedly to these operations, partial derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor of more arithmetic operations than the original program. 

https://en.wikipedia.org/wiki/Automatic_differentiation

Figure 1: How automatic differentiation relates to symbolic differentiation

Automatic differentiation is distinct from symbolic differentiation and numerical differentiation. Symbolic differentiation faces the difficulty of converting a computer program into a single mathematical expression and can lead to inefficient code. Numerical differentiation (the method of finite differences) can introduce round-off errors in the discretization process and cancellation. Both of these classical methods have problems with calculating higher derivatives, where complexity and errors increase. Finally, both of these classical methods are slow at computing partial derivatives of a function with respect to many inputs, as is needed for gradient-based optimization algorithms. Automatic differentiation solves all of these problems. 

https://en.wikipedia.org/wiki/Automatic_differentiation

Categories:

• Classification algorithms
• Neural network architectures

https://en.wikipedia.org/wiki/Multilayer_perceptron

 

Differentiable computing
General	Differentiable programming Information geometry Statistical manifold Automatic differentiation Neuromorphic engineering Pattern recognition Tensor calculus Computational learning theory Inductive bias
Concepts	Gradient descent SGD Clustering Regression Overfitting Hallucination Adversary Attention Convolution Loss functions Backpropagation Normalization (Batchnorm) Activation Softmax Sigmoid Rectifier Regularization Datasets Augmentation Diffusion Autoregression
Applications	Machine learning In-context learning Artificial neural network Deep learning Scientific computing Artificial Intelligence Language model Large language model
Hardware	IPU TPU VPU Memristor SpiNNaker
Software libraries	TensorFlow PyTorch Keras Theano JAX LangChain
Implementations	Audio–visual AlexNet WaveNet Human image synthesis HWR OCR Speech synthesis Speech recognition Facial recognition AlphaFold DALL-E Midjourney Stable Diffusion Verbal Word2vec Seq2seq BERT LaMDA Bard NMT Project Debater IBM Watson GPT-2 GPT-3 ChatGPT GPT-4 GPT-J Chinchilla AI PaLM BLOOM LLaMA Decisional AlphaGo AlphaZero Q-learning SARSA OpenAI Five Self-driving car MuZero Action selection Auto-GPT Robot control
People	Yoshua Bengio Alex Graves Ian Goodfellow Stephen Grossberg Demis Hassabis Geoffrey Hinton Yann LeCun Fei-Fei Li Andrew Ng Jürgen Schmidhuber David Silver
Organizations	Anthropic EleutherAI Google DeepMind Hugging Face OpenAI Meta AI Mila MIT CSAIL
Architectures	Neural Turing machine Differentiable neural computer Transformer Recurrent neural network (RNN) Long short-term memory (LSTM) Gated recurrent unit (GRU) Echo state network Multilayer perceptron (MLP) Convolutional neural network Residual network Autoencoder Variational autoencoder (VAE) Generative adversarial network (GAN) Graph neural network
Portals Computer programming Technology Categories Artificial neural networks Machine learning