<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Silicon & Sockets]]></title><description><![CDATA[This publication documents my independent work in cybersecurity and deep learning. It covers traffic analysis, security walkthroughs, protocol breakdowns, devlo]]></description><link>https://siliconandsockets.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Sun, 22 Nov 2026 16:32:14 GMT</lastBuildDate><atom:link href="https://siliconandsockets.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[From Thresholds to Probabilities]]></title><description><![CDATA[a comprehensive blog on the Gaussian Error Linear Unit activation function.]]></description><link>https://siliconandsockets.hashnode.dev/from-thresholds-to-probabilities</link><guid isPermaLink="true">https://siliconandsockets.hashnode.dev/from-thresholds-to-probabilities</guid><category><![CDATA[Deep Learning]]></category><category><![CDATA[Mathematics]]></category><category><![CDATA[Probability Distributions]]></category><category><![CDATA[AI]]></category><category><![CDATA[llm]]></category><dc:creator><![CDATA[Vedant]]></dc:creator><pubDate>Wed, 25 Jun 2025 00:50:17 GMT</pubDate><enclosure url="https://cdn.hashnode.com/res/hashnode/image/upload/v1750533867841/8da85873-d9b5-4043-a134-6a242e017b16.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>In the previous post, we looked at Softmax and NLL loss, both critical for output interpretation and learning in Transformers. Now let’s dive into what happens within the network: activation functions. Specifically, GELU.</p>
<hr />
<h2>What is GeLU?</h2>
<p>GeLU, or, Gaussian Error Linear Unit can be simply defined as a modified, more flexible version of ReLU. By “soft-gating” rather than hard clamping, GeLU avoids abrupt transitions that can destabilize deep stacks of self-attention layers and Feed-Forward Network layers. Abrupt transitions in ReLU arise when a neuron's input is near zero, causing the output and gradient to become zero — a phenomenon known as the <strong>"dying ReLU" problem</strong>. This leads to dead neurons and disrupts learning. One way to address this is by introducing noise to the input before applying ReLU.</p>
<p>GeLU can be interpreted as the <strong>expected</strong> output of a ReLU when its input is corrupted by Gaussian noise. Instead of a hard 0/1 gate like ReLU, GELU smoothly weights inputs by their probability of being significant, effectively allowing more nuanced gradient flow.</p>
<hr />
<h2>Deriving and explaining GeLU</h2>
<p>We just defined GeLU as the <strong>expectation of ReLU applied to input corrupted with Gaussian noise</strong>. But what exactly is this Gaussian noise?</p>
<p>Gaussian noise refers to a <strong>normal distribution with mean 0 and standard deviation 1</strong>. In the context of GeLU, it appears when we compute Φ(x), the cumulative distribution function (CDF) of the standard normal. This value represents the <strong>probability that a standard normal variable is less than</strong> <code>x</code> essentially quantifying how significant the input <code>x</code> is.</p>
<p>$$\phi(x)=P(ε≤x)$$</p>
<p>$$\phi(x) = \int^x_{-\infty}\frac{1}{\sqrt{2\pi}}e^{\frac{-t^2}{2}}dt$$</p>
<p>where</p>
<p>$$t= \varepsilon \sim N(0,1)$$</p>
<p>Now, we know that ReLU outputs <code>x</code> when it is greater than 1 and 0 when it is less than 1. Therefore, the <em>probability</em> that the neuron is <strong>on</strong> can be mathematically defined as</p>
<p>$$P(x+\varepsilon&gt;0)$$</p>
<p>$$=P(\varepsilon&gt;-x)$$</p>
<p>Now flip the inequality</p>
<p>$$=1-P(\varepsilon\le-x)$$</p>
<p>since</p>
<p>$$\phi(x)=P(ε≤x)$$</p>
<p>by symmetry of the Gaussian curve</p>
<p>$$P(\varepsilon\le-x) = \phi(-x) \Longrightarrow P(\varepsilon&gt;-x) = 1-\phi(-x)$$</p>
<p>$$P(x+\varepsilon &gt;0) =1-\phi(-x)$$</p>
<p>again, using the symmetry of Gaussian distribution</p>
<p>$$P(x+\varepsilon&gt;0) = 1-(1-\phi(x))$$</p>
<p>$$P(x+\varepsilon &gt;0) =\phi(x)$$</p>
<p>Therefore, the CDF Φ(x), a major component of the function, can be interpreted as the probability that a <strong>Gaussian noise ε∼N(0,1) exceeds −x</strong>, or equivalently, the probability that the <strong>neuron’s perturbed input x+ε is positive</strong>.</p>
<p>We are more interested in the latter.</p>
<p>We can also visualise both the interpretations of Φ(x) with the help of this graph:</p>
<img src="https://pbs.twimg.com/media/GsCLe1DaQAAppRb?format=jpg&amp;name=large" alt="Image" />

<p>Both <code>Area &lt;= x</code> and <code>Area &gt;= -x</code> means CDF Φ(x).</p>
<p>Now, lets introduce ReLU to our gaussian corrupted input.</p>
<p>$$ReLU = max(0, x+\varepsilon)$$</p>
<p>taking Expectation of ReLU</p>
<p>$$\mathbb{E}[ReLU(x+\varepsilon)]$$</p>
<p>$$=\mathbb{E}[max(0, x+\varepsilon)]$$</p>
<p>Why is this useful?</p>
<ul>
<li>For very negative value of <code>x</code>, the perturbed ReLU input samples (x + ɛ) are &lt; 0 meaning the average is approximately 0.</li>
<li>For very positive value of <code>x</code>, the perturbed ReLU input samples (x + ɛ) are &gt; 0 meaning the average is approximately <code>x</code> itself.</li>
<li>When <code>x</code> ≈ 0, we get a smooth transition from 0 to <code>x</code>.</li>
</ul>
<p>$$\therefore \mathbb{E}[max(0,x+ϵ)]=\int^\infty_{-\infty} ​ max(0,x+\varepsilon)⋅p(t)dt$$</p>
<p>after some more math, we get our final equation:</p>
<p>$$\mathbb{E}[max(0, x+\varepsilon)] = x\cdot\phi(x) +\varphi(x)$$</p>
<p>The extra term ϕ(x) (PDF of standard normal at x) is small and disappears fast as x → ±∞. So for <strong>efficient computation</strong> and <strong>very similar performance</strong>, Expectation is <strong>approximated</strong> as:</p>
<p>$$\mathbb{E}[max(0, x+\varepsilon)] \approx x\cdot\phi(x)$$</p>
<p>By our definition of GeLU,</p>
<p>$$GeLU(x) = x\cdot\phi(x)$$</p>
<p>The graph of GeLU(x) is:</p>
<img src="https://onlinelibrary.wiley.com/cms/asset/8d963cbd-ebf8-4778-b05c-97847e135136/jom4229924-fig-0001-m.jpg" alt="Mathematical Analysis and Performance Evaluation of the GELU Activation  Function in Deep Learning - Lee - 2023 - Journal of Mathematics - Wiley  Online Library" />

<p>That smooth portion near <code>x=0</code> is due to Φ(x) handling a Standard Normal variable (ε)<strong>.</strong></p>
<p>To understand how Φ(x) handling a Standard Normal variable (ε) provides a smooth transition, we can look at another Φ(x) graph:</p>
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1750809669170/f1e42f31-2b66-436d-893b-4acb5cb518a8.png" alt="" style="display:block;margin:0 auto" />

<p>Even when <code>x</code> is 0 or very near to 0, Φ(x) outputs a positive value leading to a very smooth transition instead of a very abrupt, hard-clamped transition which is seen in ReLU.</p>
<hr />
<h2>Why is GeLU widely used in modern-day LLMs?</h2>
<p>GeLU averages ReLU over all possible small Gaussian jitters meaning the output gradually changes as <code>x</code> moves. This makes it naturally capable of handling noise in data and is less sensitive to tiny changes, i.e, more stable gradients.</p>
<p>It has been observed that models with GeLU converge faster as compared to other activation functions.</p>
<hr />
<h2>Approximations and Implementations of GeLU</h2>
<p>Although we derived GeLU(x) as <code>x⋅Φ(x)</code>, in practice we use a Gaussian error function to define it:</p>
<p>$$GeLU(x) = \frac{1}{2}(1+erf(\frac{x}{\sqrt2}))$$</p>
<p>To make GeLU even more computationally efficient, we use approximations of it. The most used approximation is:</p>
<p>$$GeLU(x) \approx 0.5x(1+tanh[\sqrt\frac{2}{\pi}(x+0.044715x^3)])$$</p>
<p>The sigmoid based approximation is:</p>
<p>$$GeLU(x) \approx x\cdot\sigma(1.702x)$$</p>
<hr />
<h2>Gradient of GeLU</h2>
<p>We know that</p>
<p>$$GeLU(x) = x\cdot\phi(x)$$</p>
<p>Therefore,</p>
<p>$$\frac{\partial GeLU(x)}{\partial x} =\frac{\partial x\phi(x)}{\partial{x}}$$</p>
<p>$$= \phi(x) + x\cdot\phi^`(x)$$</p>
<p>$$= \phi(x) + x\cdot\varphi(x)$$</p>
<p>Or</p>
<p>$$= \phi(x) + x\cdot p(X=x)$$</p>
<p>where <code>p(X=x)</code> means the value of PDF at <code>x</code>.</p>
<hr />
<h2><strong>Conclusion</strong></h2>
<p>The Gaussian Error Linear Unit (GeLU) stands out by blending deterministic input with probabilistic reasoning. GeLU offers a compelling balance between mathematical elegance and practical performance making it a popular choice in modern deep learning architectures like Transformers. Unlike ReLU’s hard cutoff, GeLU softly gates inputs based on the likelihood that they remain positive under Gaussian noise. This results in a smoother, more stable activation function that better reflects real-world uncertainty in neural activations.</p>
]]></content:encoded></item><item><title><![CDATA[Logits and Likelihoods]]></title><description><![CDATA[a comprehensive blog on the Softmax activation function and Negative Log-Likelihood Loss]]></description><link>https://siliconandsockets.hashnode.dev/logits-and-likelihoods</link><guid isPermaLink="true">https://siliconandsockets.hashnode.dev/logits-and-likelihoods</guid><category><![CDATA[Deep Learning]]></category><category><![CDATA[llm]]></category><category><![CDATA[AI]]></category><category><![CDATA[Mathematics]]></category><dc:creator><![CDATA[Vedant]]></dc:creator><pubDate>Fri, 30 May 2025 22:42:52 GMT</pubDate><enclosure url="https://cdn.hashnode.com/res/hashnode/image/upload/v1748683910893/d060df0b-33dd-4bb7-b73e-b303411d5caf.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>In the heart of modern neural networks capable of understanding the sentence semantics and generating thousands of words per second, Transformers, as we call them, lie 2 core mathematical operations that often go unnoticed, namely the Softmax and Negative Log-Likelihood. While the overall Transformer architecture and attention mechanisms get most of the spotlight, its these foundational functions that translate raw model outputs, which cant be decoded statistically, into structured probability distributions and guide the model’s learning through the desired gradient based optimization.</p>
<p>This blog aims to demystify the roles of Softmax and NLL, tracing how they work, why they matter, and how they affect the behavior of models, specifically autoregressive models.</p>
<hr />
<h1>The Softmax activation function</h1>
<h3>Deriving Softmax</h3>
<p>Softmax, put simply, takes the output of a neural network, often called <strong>logits</strong>, and converts them into a <strong>probability distribution</strong>. This means each value is transformed into a probability between 0 and 1, and all the resulting probabilities add up to 1. In that sense, Softmax creates a <strong>normalized distribution</strong> over the possible output classes.</p>
<p>But here's the catch: logits can be <strong>positive</strong>, <strong>negative</strong>, or close to <strong>zero</strong>. So how do we convert these arbitrary values into something non-negative and meaningful as a probability? The answer lies in the <strong>exponential function</strong>. No matter what number you input, negative, zero, or positive, the exponential function always outputs a <strong>positive</strong> value.</p>
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1748626092425/ca83cefb-34cd-436d-8c27-de71e41cac24.jpeg" alt="" style="display:block;margin:0 auto" />

<p>Now that we’ve used the exponential function to make all the logits positive, the next step is to normalize them.</p>
<p>In the formula for Softmax, the denominator consists of a summation term that sums up all the exponential values across all classes. This step is what turns a bunch of unbounded numbers into a proper probability distribution. Hence, it is safe to think of the denominator as a kind of <strong>normalization constant.</strong> it balances the scores so that the largest exponentiated value doesn’t just dominate, and every class gets its fair share in the final distribution.</p>
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1748627761587/d13c9300-55d1-4af1-93ae-867cbce63b65.jpeg" alt="" style="display:block;margin:0 auto" />

<p><em>By examining the y-coordinate of each point on both graphs above, you can see the difference between them. The y-coordinates of the points on the second graph add up to 1, while those on the first graph are not normalized.</em></p>
<p>Now, we can construct the Softmax function from the above points</p>
<p>$$\text{Softmax}(x_i) = \frac{e^{x_i}}{\sum_{j=1}^{k} e^{x_j}}$$</p>
<p>where <code>xi</code> = logit at index i, <code>xj</code> = logit at index j, <code>k</code> = number of classes</p>
<hr />
<h3>Temperature</h3>
<p>Since Softmax transforms raw logits into probabilities relative to each other, we can scale them arbitrarily by multiplying all the probabilities with a scaling factor. This scaling can change the model behaviour by changing how deterministic or random the outputs can be.</p>
<p>Mathematically, the Softmax with temperature T is written as:</p>
<p>$$\text{Softmax}(x_i; T) = \frac{e^{x_i / T}}{\sum_{j=1}^{k} e^{x_j / T}}$$</p>
<p>When <strong>T&lt;1</strong>, the logits are divided by a small number, making the exponentials more extreme. This sharpens the probability distribution, making the model <strong>more confident and deterministic</strong>.</p>
<p>When <strong>T&gt;1</strong> the logits are divided by a larger number, flattening the distribution. The model becomes <strong>less confident and more random</strong> in its choices.</p>
<p>This technique is usually seen in generative models like LLMs where temperature helps balance between creativity and accuracy during generations.</p>
<hr />
<h3>Why Softmax?</h3>
<p>You might wonder: why not just use the raw logits directly? Why bother with Softmax at all?</p>
<p>The reason is tied to what a <strong>classification task</strong> actually requires. To decide which class the model believes is the most likely, we need a way to <strong>compare the relative importance</strong> of each output. Raw logits don't give us a clear sense of how confident the model is. Softmax helps by <strong>scaling the values</strong> in a way that reflects their importance, turning them into probabilities that are easy to interpret and compare.</p>
<p>In a transformer model, the goal is simple, to come up with a probability distribution, which says, at each time step, how much importance or attention should be paid to the input words. Hence, Softmax is typically applied to raw attention scored obtained from dot product of Query and Key vectors in the self-attention mechanism.</p>
<hr />
<h1>Negative Log-Likelihood Loss</h1>
<p>In classification or token-prediction tasks, we pair the Softmax output with a loss function (Negative Log-Likelihood) that penalizes incorrect probabilities and pushes up the probability of the true class while pushing down the others. This involves maximizing the likelihood of seeing correct output from the model.</p>
<h2>Deriving Negative Log-Likelihood Loss</h2>
<p><em>Likelihood refers to the chances of some calculated parameters producing some known data.</em></p>
<p>We can define Likelihood l(θ) as</p>
<p>$$l(\theta) = p(x_1, \ldots, x_n \mid \theta) = \prod_{i=1}^n p(x_i \mid \theta).$$</p>
<p>where <code>x1, x2, … , xn</code><strong>​</strong> are individual observations and <code>n</code> is the number of independent observations.</p>
<p>To maintain efficiency in computation and ease in differentiation, we can introduce Logarithm to our expression of l(θ)</p>
<p>$$\log l(\theta) = \sum_{i=1}^n p(x_i \mid \theta)$$</p>
<p>Now, our goal is to maximize this log-likelihood for each observation.</p>
<p>It is safe to say that our goal of maximizing log-likelihood is equivalent to minimizing the loss.</p>
<p>Why? Because maximizing a function f(θ) is the same as minimizing any monotonically decreasing transformation of it.</p>
<p>Hence, we can define our Loss, L(θ) as</p>
<p>$$L(\theta) = -l(\theta) = -\sum_{i=1}^n \log p(x_i \mid \theta)$$</p>
<p>Now, we require the θ that makes this loss, L(θ) as small as possible</p>
<p>$$\hat{\theta} = \arg \min_{\theta} L(\theta)$$</p>
<p>Because L(θ) = -l(θ), flipping the sign swaps <strong>max</strong> for <strong>min</strong></p>
<p>$$\arg \max_{\theta} l(\theta) = \arg \min_\theta[-l(\theta)] = \arg \min_\theta L(\theta)$$</p>
<p>Hence, finding the θ that maximizes our Likelihood, or Log-Likelihood, is exactly the same as finding the θ that minimizes NLL loss.</p>
<p>Therefore, the final expression for Negative Log-Likelihood Loss is</p>
<p>$$L(\theta) = - \sum_{i=1}^N \log p(label_i \mid input_i, \theta)$$</p>
<p>For a Softmax setting where true labels are one-hot encoded vectors, we can derive NLL loss as follows</p>
<p>Let <code>true class label be y</code> and <code>logit(s) be z</code> then the probability assigned to that correct class by the model is</p>
<p>$$p_y = \frac{e^{z_y}}{\sum_je^{z_j}}$$</p>
<p>$$\log p_y = z_y - \log\sum_je^{z_j}$$</p>
<p>Hence, our loss L(θ) for a Softmax setting will be</p>
<p>$$L(z, y) = -z_y + \log\sum_je^{z_j}$$</p>
<p>Minimizing <code>L(z, y)</code> or NLL loss simultaneously raises <code>z_y</code> and lowers other logits so as to maximize likelihood of desired output.</p>
<hr />
<h1>Binding Softmax and NLL together</h1>
<h2>Derivative of NLL with respect to Logit(s)</h2>
<p>We know that</p>
<p>$$L = -\sum_{i=1}y_i\log p_i$$</p>
<p>Therefore</p>
<p>$$\frac{\partial L}{\partial p_i}=-\frac{y_i}{p_i}$$</p>
<p>Our end goal is to find how much Loss changes if Logits also change by infinitesimal values, hence, we use <strong>chain rule</strong> as follows</p>
<p>$$\frac{\partial L}{\partial z_j}=\sum^C_{i=1}\frac{\partial L}{\partial p_i}\frac{\partial p_i}{\partial z_j}$$</p>
<p>From the Softmax jacobian</p>
<p>$$= \sum^C_{i=1}(-\frac{y_i}{p_i})p_i(\delta_{ij}-p_j)$$</p>
<p>$$= -\sum_iy_i\delta_{ij} + \sum_iy_ip_j$$</p>
<p>$$= -y_j + p_j\sum_{i}y_i$$</p>
<p>$$= p_j - y_j$$</p>
<p>where <code>i, j</code> are class-indices from 1 to C(total number of classes), <code>z_j</code> is logit for class j, <code>p_j</code> is softmax probability of class j, <code>y</code> is true label under one-hot encoding.</p>
<p>When true class <code>j = y</code>, <code>y_j</code> = <code>y_y</code> = 1</p>
<p>$$\frac{\partial L}{\partial z_j} = -1 +p_y$$</p>
<p>Whenever <code>p_y</code> &lt; 0, gradient descent will increase the logit z_j, pushing up the predicted probability for the true class.</p>
<p>When other classes <code>j ≠ y</code>, <code>y_j</code> = 0</p>
<p>$$\frac{\partial L}{\partial z_j}=p_j -0 =p_j$$</p>
<p>Since <code>p_j</code> &gt; 0, these gradients are positive, so gradient descent will decrease non-true logits <code>z_j</code>, pushing down the predicted probability for non-true classes.</p>
<hr />
<h1>Conclusion</h1>
<p>Together, Softmax and Negative Log-Likelihood form the mathematical backbone of almost every modern classifier and autoregressive model. By understanding not just <strong>what</strong> Softmax + NLL does, but <strong>why</strong> it works, from the probabilistic foundations to the gradient derivation, you’ll be equipped to diagnose training issues, innovate new loss formulations, and appreciate the statistical guarantees underpinning your models.</p>
<p><em>Happy modeling!</em></p>
]]></content:encoded></item></channel></rss>