✦ ANN Module 01 · CI (CS-3030)

Activation Functions
Shaping the Neuron's Output

From the hard binary step to smooth sigmoids — an interactive calculator for every standard transfer function, plus the classic AND/OR sigmoid network.

Quick Recap — The Neuron Pipeline
Every artificial neuron follows the same three-stage pipeline: weighted summation → activation function → output.
Neuron K PIPELINE
u_k = Σⱼ xⱼ w_kj  →  v_k = u_k + b_k  →  y_k = f(v_k)

Where w_kj is the connection weight from neuron j to neuron k, b_k is the bias, and f(·) is the activation (transfer) function — the subject of this module.

📌 For a single-input, single-output neuron in the simplified form: a = f(w·p + b), where p is the input, w the weight, b the bias.
DIAGRAM — MULTI-INPUT NEURON K
1 2 ⋮ N w_k1 w_k2 w_kN b_k Σ v_k f(·) y_k (1/0) summation unit activation function
Standard Activation Functions
The complete catalog of classic transfer functions f(sum_k), each with its defining equation.
Table 2.3 — Standard & Common Activation Functions REFERENCE
FunctionEquation
Identity (purelin)f(sum_k) = sum_k
Binary step1 if sum_k ≥ θ_k, else 0
Binary sigmoid (logsig)1 / (1 + exp(−s·sum_k))
Bipolar sigmoid (tansig)(1−exp(−s·sum_k)) / (1+exp(−s·sum_k))
Hard limit (hardlim)1 if sum_k ≥ 0, else 0
Symmetric hard limit (hardlims)1 / sum_k / 0 for >1 / [0,1] / <0
Saturating linear (satlin)1 if sum_k ≥ 0, else −1
Symmetric sat. linear (satlins)1 / sum_k / −1 for >1 / [−1,1] / <−1
The Logistic (Sigmoid) Function Family S-CURVE

The slope parameter s controls steepness: s=0 gives a flat line at 0.5, while larger s pushes the curve toward a hard step. This is why the sigmoid is often called a "smoothed" binary step function — it enables gradual weight updates, unlike the discontinuous step function.

LOGISTIC ACTIVATION FUNCTION FOR VARYING SLOPE s
Interactive Activation Calculator
Enter weights, inputs, and a bias — pick a transfer function and watch sum_k and f(sum_k) compute live, exactly like the handbook's worked problems.
Neuron Configuration LIVE
COMPUTATION
FUNCTION SHAPE (sum_k from −10 to 10)
Worked Problems (from lecture notes)
Try each one in the Interactive Calculator tab to reproduce these exact answers.
Q1 — Single Input, Bipolar Sigmoid Q1

A single-input, single-output neuron has weight = 5 and bias = −3. What is sum_k for input 1? What is the output using bipolar sigmoid with slope s=0.5?

sum_k = 5×1 − 3 = 2
f(sum_k) = 1/(1+exp(−0.5×2)) = 0.731  [binary sigmoid, logsig]
Q2 — 2-Input Identity Function Q2

A 2-input, single-output NN has weights [1.3, 2.7] and bias 1.6. Input = [3.1, 2.4]ᵀ. Output using the identity function?

sum_k = 1.3×3.1 + 2.7×2.4 + 1.6 = 12.11
f(sum_k) = sum_k = 12.11  [identity / purelin]
Q3 — Symmetric Saturating Linear Q3

Same network as Q2, symmetric saturating linear activation, input = [−6, 4.1]ᵀ.

sum_k = 1.3×(−6) + 2.7×4.1 + 1.6 = 4.87
f(sum_k) = 1  (since sum_k ≥ 0)  [satlins]
Q4 — 3-Input, 2-Output Network, Binary Step Q4

w₁₁=0.6, w₁₂=1.1, w₂₁=0.7, w₂₂=0.5, w₃₁=0.8, w₃₂=0.2. Input=[0.3, 0.7, 1.6]ᵀ, threshold=1.5.

sum_(k=1) = 0.6×0.3 + 0.7×0.7 + 0.8×1.6 = 1.95 → f = 1 (≥1.5)
sum_(k=2) = 1.1×0.3 + 0.5×0.7 + 0.2×1.6 = 1.00 → f = 0 (<1.5)
Q5 — Bipolar Sigmoid, Slope s=1.1 Q5

Using the sum_k values from Q4 (1.95 and 1.00), apply the bipolar sigmoid (tansig) with slope s=1.1.

f(sum_(k=1)) = (1−exp(−1.1×1.95)) / (1+exp(−1.1×1.95)) ≈ 0.79
f(sum_(k=2)) = (1−exp(−1.1×1)) / (1+exp(−1.1×1)) ≈ 0.5
Solving AND / OR with a 2-Input, 1-Output Sigmoid Neuron
Unlike the equal-weight MCP neuron, a sigmoid neuron uses large signed weights and a bias to push sum_k far into the saturated region — approximating a hard decision boundary.
Simplified 2-Input Neuron with Bias DIAGRAM
SIMPLIFIED FORM — x₀ (BIAS) + x₁, x₂
+1 x₀ bias x₁ x₂ w₀ w₁ w₂ Σ sum_k f(·) h(x)
AND Network w₀=−30, w₁=w₂=20
h_AND(x) = f(−30 + 20x₁ + 20x₂),   f = sigmoid
x₁x₂sum_kh_AND(x)
00−30≈ 0
01−10≈ 0
10−10≈ 0
1110≈ 1
OR Network w₀=−10, w₁=w₂=20
h_OR(x) = f(−10 + 20x₁ + 20x₂),   f = sigmoid
x₁x₂sum_kh_OR(x)
00−10≈ 0
0110≈ 1
1010≈ 1
1130≈ 1
📌 Try these exact weights in the Interactive Calculator (weights: 20,20 · bias: −30 or −10 · function: Binary Sigmoid) to reproduce the ≈0 / ≈1 outputs.