29,259 technical terms and definitions
activation checkpointing, memory efficient training, recomputation training, checkpointing deep learning
Gradient checkpointing's recompute-versus-retain memory tradeoff can be tuned by choosing how many, and which, forward-pass activation layers to actually checkpoint and retain…
activation recomputation, memory efficient training, checkpoint segment, rematerialization
Gradient Checkpointing (Activation Recomputation) is the memory optimization technique for training deep neural networks that trades compute for memory by storing only a subset…
max norm, stability
Gradient clipping limits gradient magnitude during training to prevent exploding gradients, stabilizing optimization of deep networks and recurrent architectures. Methods: (1)…
model training
Gradient clipping caps gradient magnitude to prevent exploding gradients that destabilize training. The problem: Large gradients cause huge weight updates, loss spikes, or NaN…
training techniques
Gradient Clipping is operation that limits gradient magnitude to a fixed norm before optimization updates - It is a core method in modern semiconductor AI serving…
gradient explosion, clip grad norm
Gradient Clipping — a technique that limits the magnitude of gradients during backpropagation to prevent exploding gradients from destabilizing training. The Problem In deep…
clip gradient, global norm clipping, clip grad norm, exploding gradients
Gradient clipping limits gradient magnitude before an optimizer update to prevent rare extreme steps from destabilizing training. It is standard protection for recurrent networks…
Gradient Clipping Strategies Introduction & Motivation Gradient Clipping Strategies addresses a central problem in optimization algorithm tuning and improvement: how to Optimize…
communication
Gradient Compression is a distributed training optimization that reduces the communication overhead of synchronizing gradients across GPU workers — using quantization (reducing…
gradient sparsification, powersgd, topk gradients, communication compression
Gradient Compression is a distributed training optimization technique that reduces the communication volume of gradients — sending only the most important gradient information…
gradient compression communication-efficient prediction, gradient compression communication-efficient, compression communication-efficient prediction
Gradient Compression Communication-Efficient Prediction with Machine Learning Introduction & Motivation Gradient Compression Communication-Efficient Prediction with Machine…
privacy
Gradient Compression for Privacy is the use of gradient compression techniques (sparsification, quantization) to reduce privacy leakage in distributed training — by transmitting…
gradient compression and quantization for communication efficiency, gradient compression quantization, compression quantization communication, quantization communication efficiency
Gradient Compression and Quantization for Communication Efficiency Introduction & Motivation Gradient Compression and Quantization for Communication Efficiency addresses a central…
distributed training
Gradient compression techniques is the communication-reduction methods that lower distributed training bandwidth demand by encoding or sparsifying gradients - they reduce…
top k sparsification, gradient sparsity training, magnitude based pruning, sparse gradient communication
Gradient Compression Techniques are the family of methods that reduce gradient communication volume by transmitting only the most important gradient components — using…
learning rate, what is gradient descent
Gradient descent is the algorithm that actually does the "learning" in machine learning: it repeatedly nudges a model's parameters in the direction that reduces the loss function…
gradient descent, descent optimizers, cifar glue lamb lars nlp sgd
Gradient Descent Optimizers Introduction & Motivation Optimization: update parameters to minimize loss. SGD, Adam, AdamW. Applications: training convergence, hyperparameter…
gradient descent variants batch mini batch stochastic, gradient descent, variants batch mini, bert cifar gd gpu lr mnist, gradient descent variants batch mini-batch & stochastic
Gradient Descent Variants: Batch, Mini-Batch & Stochastic Introduction & Motivation Gradient descent update policy: determines optimization trajectory. Batch GD: full dataset per…
Gradient Descent Algorithm Variants Introduction & Motivation Gradient Descent Algorithm Variants addresses a central problem in optimization algorithm tuning and improvement: how…
gem, continual learning
Gradient episodic memory is a continual-learning algorithm that constrains new-task gradients so they do not increase loss on stored past-task examples - Projected gradients…
computer vision
Gradient flow in deep ViTs is the mechanism that determines whether supervision signals can propagate across many transformer layers without vanishing or exploding - controlling…
model training
Gradient Flow Preservation is a design principle for pruning and sparse training — ensuring that removing weights does not disrupt the backpropagation signal, keeping gradient…
gradient flow stability in deep networks prediction, gradient flow stability, flow stability deep, stability deep networks, deep networks prediction, gradient flow
Gradient Flow Stability in Deep Networks Prediction with Machine Learning Introduction & Motivation Gradient Flow Stability in Deep Networks Prediction with Machine Learning…
Evolution Strategies for Gradient-Free Meta-Learning Introduction & Motivation Evolution Strategies for Gradient-Free Meta-Learning addresses a central problem in fewshot…
Nelder-Mead Simplex Direct Search Algorithm Introduction & Motivation Nelder-Mead Simplex Direct Search Algorithm addresses a central problem in optimization algorithm tuning…
Score-CAM Gradient-Free Visual Activation Mapping Introduction & Motivation Score-CAM Gradient-Free Visual Activation Mapping addresses a central problem in explainability…
ai safety
Gradient Masking is a phenomenon where a defense accidentally or intentionally makes the model's gradients uninformative — causing gradient-based attacks to fail while the model…
Gradient Masking Pitfalls Introduction & Motivation Gradient Masking Pitfalls addresses a central problem in robustness of advrobust models under adversarial conditions: how…
Gradient Masking Pitfalls (Extended Study) Introduction & Motivation Gradient Masking Pitfalls (Extended Study) addresses a central problem in robustness of advrobust models under…
Gradient Masking Pitfalls (Extended Study) Introduction & Motivation Gradient Masking Pitfalls (Extended Study) addresses a central problem in robustness of advrobust models under…
optimization
Gradient Noise is the deliberate addition of noise to gradient updates during training — typically Gaussian noise with decaying variance, which helps escape local minima, improves…
Gradient Norm Expectation for Deep Model Active Selection Introduction & Motivation Gradient Norm Expectation for Deep Model Active Selection addresses a central problem…
optimization
Gradient Normalization is the practice of normalizing gradient magnitudes during training — either by clipping the gradient norm to a maximum value (gradient clipping) or…
generative models
Gradient Penalty is a regularization technique used primarily in GAN training (WGAN-GP) — penalizing the norm of the discriminator's gradient with respect to its input, enforcing…
distributed training
Gradient quantization for communication reduces the precision of gradient tensors before transmitting them between workers in distributed training, dramatically reducing network…
domain adaptation
The Gradient Reversal Layer (GRL) is the ingenious mathematical trick at the beating heart of Adversarial Domain Adaptation (specifically DANN), functioning as a simple, custom…
optimization
Gradient scaling, the general technique underlying loss scaling, multiplies gradients by a chosen factor before any precision-limited numeric operation that might otherwise lose…
Gradient Similarity and Dot-Product Task Affinity Introduction & Motivation Gradient Similarity and Dot-Product Task Affinity addresses a central problem in transfer learning…
optimization
Gradient Sparsification is a communication reduction technique for distributed training that transmits only a subset of gradient components — sending only the most important…
Gradient Surgery for Multi-Task Learning (PCGrad) Introduction & Motivation Gradient Surgery for Multi-Task Learning (PCGrad) addresses a central problem in multitask learning…
Gradient Surgery for Multi-Task Learning (PCGrad) (Extended Study) Introduction & Motivation Gradient Surgery for Multi-Task Learning (PCGrad) (Extended Study) addresses a central…
Gradient Surgery for Multi-Task Learning (PCGrad) (Extended Study) Introduction & Motivation Gradient Surgery for Multi-Task Learning (PCGrad) (Extended Study) addresses a central…
PCGrad Projected Conflicting Gradients Multi-Task Introduction & Motivation PCGrad Projected Conflicting Gradients Multi-Task addresses a central problem in optimization algorithm…
distributed training
Gradient synchronization is the distributed operation that aligns per-worker gradients into a shared update before parameter step - it ensures data-parallel replicas remain…
analog front end, sensor signal conditioning, instrumentation amplifier, sensor adc, gradio
sensor interface is an analog and mixed-signal signal chain that excites, conditions, digitizes, calibrates, and communicates a physical sensor output. It is the bridge that…
GradNorm Loss Balancing Introduction & Motivation GradNorm Loss Balancing addresses a central problem in multitask learning across several related prediction objectives: how…
GradNorm Loss Balancing (Extended Study) Introduction & Motivation GradNorm Loss Balancing (Extended Study) addresses a central problem in multitask learning across several…
GradNorm Loss Balancing (Extended Study) Introduction & Motivation GradNorm Loss Balancing (Extended Study) addresses a central problem in multitask learning across several…
Gradual Domain Adaptation along Intermediate Trajectories Introduction & Motivation Gradual Domain Adaptation along Intermediate Trajectories addresses a central problem…
percentage, traffic
Gradual Rollout Gradual rollout (also called canary deployment or progressive delivery) incrementally increases traffic to a new model or system version—1%, 5%, 10%, 25%, 50%…