Core Concepts

• Information content
• Entropy
• Joint entropy
• Conditional entropy
• Mutual information
• Cross-entropy
• KL divergence (Kullback-Leibler Divergence)

Applications in Large Models

Loss Function

• Cross-entropy loss is a measure of the difference between the predicted distribution and the true distribution.

Attention Mechanism

• When computing attention weights, the softmax operation relates to probability distributions and entropy.

Reinforcement Learning

• The optimization objective in policy gradient may include an entropy regularization term to encourage exploration.
• The core of TRPO / PPO algorithms is a KL divergence constraint.

Model Compression and Quantization

• Evaluating quantization information loss.

---

贡献者

• Mira190贡献 2 次 · 最近 2025/09/13
• longsizhuo贡献 0 次 · 最近 2026/04/18

---

最近更新

Involution Hell© 2026 byCommunityunderCC BY-NC-SA 4.0