Data Analytics with Deep Learning

Multilayer Perceptrons: Architecture and Theoretical Foundations

The multilayer perceptron (MLP) is a feedforward neural network that extends the simple perceptron by introducing one or more hidden layers between the input and output layers. Each hidden unit applies a smooth, differentiable nonlinear activation function to a weighted sum of its inputs. The presence of these nonlinear hidden layers enables the network to learn complex mappings from input features to target variables, making MLPs universal approximators: a single hidden layer with a sufficient number of neurons can approximate any continuous function to arbitrary accuracy, provided the activation function is not polynomial. In practice, deeper architectures with multiple hidden layers often achieve superior representational efficiency and generalisation.

Formally, consider an $L$-layer MLP (where layer $0$ is the input and layer $L$ the output). Let $\mathbf{x} \in \mathbb{R}^d$ be the input vector. For each layer $l = 1,\dots,L$, denote the pre-activation vector as $\mathbf{a}^{(l)}$ and the activation vector as $\mathbf{h}^{(l)}$. The computation proceeds recursively:

$$\mathbf{a}^{(l)} = \mathbf{W}^{(l)} \mathbf{h}^{(l-1)} + \mathbf{b}^{(l)}, \qquad \mathbf{h}^{(l)} = \sigma^{(l)}(\mathbf{a}^{(l)}),$$

with $\mathbf{h}^{(0)} = \mathbf{x}$, weight matrices $\mathbf{W}^{(l)} \in \mathbb{R}^{n_l \times n_{l-1}}$, bias vectors $\mathbf{b}^{(l)} \in \mathbb{R}^{n_l}$, and activation function $\sigma^{(l)}$. Common choices for hidden layers include the Rectified Linear Unit (ReLU) $\sigma(a) = \max(0,a)$ and its variants, the hyperbolic tangent $\tanh(a)$, and the logistic sigmoid $\sigma(a) = 1/(1+e^{-a})$. The differentiability of $\sigma$ is essential for gradient-based training.

Without nonlinearities, the composition of multiple linear transformations would mathematically collapse into a single linear mapping, stripping the network of its representational power.

Regression with MLPs

For regression tasks, the output layer typically employs a linear activation, so $\hat{\mathbf{y}} = \mathbf{a}^{(L)}$. Given a training set $\{ (\mathbf{x}_i, \mathbf{y}_i) \}_{i=1}^N$, the network parameters $\boldsymbol{\Theta} = \{\mathbf{W}^{(l)}, \mathbf{b}^{(l)}\}_{l=1}^L$ are learned by minimising a loss function. The most common choice is the Mean Squared Error (MSE):

$$\mathcal{L}_{\text{MSE}}(\boldsymbol{\Theta}) = \frac{1}{2N} \sum_{i=1}^N \|\mathbf{y}_i - \hat{\mathbf{y}}_i\|^2.$$

The factor $1/2$ simplifies subsequent gradient calculations.

Training proceeds by iteratively updating the parameters in the opposite direction of the gradient of the loss with respect to each parameter, a process known as gradient descent. Because the network is a composition of differentiable functions, the gradient can be computed efficiently using the backpropagation algorithm, which applies the chain rule of calculus from the output layer backward to the input.

For a single data point, define the error signal at the output layer: $\boldsymbol{\delta}^{(L)} = \frac{\partial \mathcal{L}}{\partial \mathbf{a}^{(L)}}$. In the case of MSE with linear output activation, one obtains

$$\boldsymbol{\delta}^{(L)} = \hat{\mathbf{y}} - \mathbf{y}.$$

For any hidden layer $l$, the error is backpropagated via

$$\boldsymbol{\delta}^{(l)} = \sigma'( \mathbf{a}^{(l)} ) \odot \big( (\mathbf{W}^{(l+1)})^\top \boldsymbol{\delta}^{(l+1)} \big),$$

where $\odot$ denotes element-wise multiplication and $\sigma'$ is the derivative of the activation function. The gradients with respect to the weights and biases are then

$$\frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(l)}} = \boldsymbol{\delta}^{(l)} (\mathbf{h}^{(l-1)})^\top, \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{b}^{(l)}} = \boldsymbol{\delta}^{(l)}.$$

All parameters are updated using the chosen gradient-based optimiser. The ability of backpropagation to efficiently compute all partial derivatives in one forward and one backward pass is the cornerstone of deep learning.

Classification with MLPs

For classification, the output layer must produce a valid probability distribution over $K$ classes. The softmax function accomplishes this:

$$\hat{y}_k = \text{softmax}(\mathbf{a}^{(L)})_k = \frac{e^{a^{(L)}_k}}{\sum_{j=1}^{K} e^{a^{(L)}_j}}.$$

The natural loss function is the categorical cross-entropy (negative log-likelihood):

$$\mathcal{L}_{\text{CE}} = -\sum_{k=1}^{K} y_k \log(\hat{y}_k),$$

where $\mathbf{y}$ is a one-hot encoded vector of the true class. A remarkable simplification occurs when computing the gradient of the cross-entropy loss with respect to the pre-activation $\mathbf{a}^{(L)}$:

$$\frac{\partial \mathcal{L}_{\text{CE}}}{\partial a^{(L)}_k} = \hat{y}_k - y_k.$$

This elegant result, formally similar to that of MSE regression, stems from the cancellation of terms when differentiating the log-softmax. It means that the output-layer error signal is simply the difference between predicted probabilities and the one-hot target, making the implementation of backpropagation for classification both compact and numerically stable. For binary classification, a single logistic output with binary cross-entropy reproduces the same gradient form $\hat{y} - y$.

Economic and Analytical Intuition: In microeconomic theory, a firm seeks to minimize its cost function subject to technological constraints, adjusting input factors based on their marginal productivity. Similarly, an MLP minimizes its empirical risk (the loss function) by adjusting its weights. The gradient of the loss with respect to a specific weight represents the marginal cost of that parameter. By moving in the direction of the steepest descent, the network efficiently allocates its "representational resources" to minimize prediction errors, akin to a firm optimizing its production inputs to minimize financial costs.

Practical Implementation with PyTorch

import torch
import torch.nn as nn
import torch.optim as optim

# 1. Define the MLP Architecture
class MLPRegressor(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(MLPRegressor, self).__init__()
        self.network = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, output_dim)  # Linear output for regression
        )
    
    def forward(self, x):
        return self.network(x)

# 2. Instantiate Model, Loss Function, and Optimizer
input_features = 15
model = MLPRegressor(input_dim=input_features, hidden_dim=64, output_dim=1)
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001, weight_decay=1e-5)

# 3. Training Loop
num_epochs = 100
for epoch in range(num_epochs):
    model.train()
    optimizer.zero_grad()
    
    predictions = model(X_train)
    loss = criterion(predictions, y_train)
    
    loss.backward()
    optimizer.step()
    
    if (epoch + 1) % 10 == 0:
        print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')

For a classification task, the architectural modifications are minimal but critical: the final nn.Linear layer should output $C$ logits, the activation function is removed from the final layer (as nn.CrossEntropyLoss internally applies the softmax), and the criterion is changed to nn.CrossEntropyLoss().

Gradient Descent Optimizers

Standard Stochastic Gradient Descent (SGD) updates parameters according to $\boldsymbol{\Theta}_{t+1} = \boldsymbol{\Theta}_t - \eta \nabla_{\boldsymbol{\Theta}} \mathcal{L}$, where $\eta$ is the learning rate. While simple, SGD suffers from slow convergence in flat regions, oscillations along directions of high curvature, and difficulty escaping saddle points. More sophisticated optimisers introduce adaptive learning rates and momentum to address these issues.

Momentum accumulates an exponentially decaying moving average of past gradients, effectively smoothing the update trajectory and accelerating convergence along directions of consistent gradient:

$$\mathbf{v}_t = \beta \mathbf{v}_{t-1} + (1-\beta) \nabla_{\boldsymbol{\Theta}} \mathcal{L}_t, \qquad \boldsymbol{\Theta}_{t+1} = \boldsymbol{\Theta}_t - \eta \mathbf{v}_t.$$

The hyperparameter $\beta \in [0,1)$ (commonly $0.9$) controls the memory. Momentum helps to dampen oscillations in narrow ravines of the loss landscape and climbs out of plateaus more quickly. Intuition: Analogous to a heavy ball rolling down a physical hill, momentum builds speed in directions with consistent gradients.

RMSProp (Root Mean Square Propagation) adapts the learning rate for each parameter individually by maintaining a moving average of the squared gradients:

$$\mathbf{s}_t = \gamma \mathbf{s}_{t-1} + (1-\gamma) (\nabla_{\boldsymbol{\Theta}} \mathcal{L}_t)^2, \qquad \boldsymbol{\Theta}_{t+1} = \boldsymbol{\Theta}_t - \frac{\eta}{\sqrt{\mathbf{s}_t + \epsilon}} \odot \nabla_{\boldsymbol{\Theta}} \mathcal{L}_t.$$

Here $\gamma$ is usually set to $0.9$ and $\epsilon$ (e.g., $10^{-8}$) prevents division by zero. Parameters with large gradient magnitudes receive smaller effective learning rates. Economic Intuition: This mirrors volatility-adjusted investment strategies in finance. Parameters exhibiting high historical variance receive smaller marginal updates to prevent catastrophic divergence.

Adam (Adaptive Moment Estimation) combines the ideas of Momentum and RMSProp. It maintains both a first moment estimate $\mathbf{m}_t$ (the mean) and a second moment estimate $\mathbf{v}_t$ (the uncentered variance) of the gradients:

$$\mathbf{m}_t = \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \nabla_{\boldsymbol{\Theta}} \mathcal{L}_t,$$ $$\mathbf{v}_t = \beta_2 \mathbf{v}_{t-1} + (1-\beta_2) (\nabla_{\boldsymbol{\Theta}} \mathcal{L}_t)^2.$$

To correct the bias introduced by initialising the moment estimates to zero, Adam applies a bias correction:

$$\hat{\mathbf{m}}_t = \frac{\mathbf{m}_t}{1-\beta_1^t}, \qquad \hat{\mathbf{v}}_t = \frac{\mathbf{v}_t}{1-\beta_2^t}.$$

The parameter update then is:

$$\boldsymbol{\Theta}_{t+1} = \boldsymbol{\Theta}_t - \frac{\eta}{\sqrt{\hat{\mathbf{v}}_t + \epsilon}} \odot \hat{\mathbf{m}}_t.$$

Default values: $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$. Adam is currently the default optimizer for most deep learning applications due to its robust performance across a wide range of architectures and problem domains.

From Feedforward Networks to Recurrent Architectures

Standard multilayer perceptrons (MLPs) assume that each input $x$ is an independent, fixed-dimensional vector and that no temporal or sequential dependencies exist among successive samples. In many data analytics applications — financial time series, sensor streams, natural language — the ordering of observations carries essential information. A feedforward network that maps $x_t$ to an output $y_t$ treats each time step in isolation, ignoring the historical context required for accurate forecasting or classification.

A Recurrent Neural Network (RNN) addresses this limitation by introducing a hidden state that serves as the network's memory of previous inputs. At each time step $t$, the RNN receives an input $x_t$ and updates its hidden state $h_t$ using both $x_t$ and the previous hidden state $h_{t-1}$:

$$h_t = \phi\,(W_{h} h_{t-1} + W_{x} x_t + b_h),$$

where $\phi$ is a nonlinear activation (typically $\tanh$ or ReLU), $W_{h}$ and $W_{x}$ are weight matrices, and $b_h$ a bias vector. An output $y_t$ can be generated from $h_t$ through a linear layer:

$$\hat{y}_t = W_{y} h_t + b_y.$$

The same weights are shared across all time steps, making the RNN capable of processing sequences of arbitrary length with a fixed number of parameters. Parameter tying is what gives the RNN its power: the network learns a transition function rather than a separate mapping for each horizon, mirroring the way an economic agent applies the same decision rule repeatedly as new information arrives.

Because the hidden state $h_t$ summarizes all past inputs $x_1, \dots, x_t$, RNNs are, in theory, able to capture long-range dependencies. In practice, training such networks by backpropagation through time (BPTT) exposes crucial numerical difficulties that severely limit their memory horizon.

The Vanishing and Exploding Gradient Problems

Training an RNN involves unrolling the computational graph over $T$ time steps and applying standard backpropagation. For a loss $\mathcal{L}$ computed at the final step (or summed over steps), the gradient of $\mathcal{L}$ with respect to an initial hidden state $h_0$ involves repeated multiplication by the recurrent weight matrix. Ignoring activation functions and biases for clarity, the recurrence simplifies to $h_t = W_h h_{t-1}$, and the gradient via the chain rule is

$$\frac{\partial \mathcal{L}}{\partial h_0} = \prod_{k=1}^{T} \frac{\partial h_k}{\partial h_{k-1}} \,\frac{\partial \mathcal{L}}{\partial h_T} = \left(W_h^{\top}\right)^T \frac{\partial \mathcal{L}}{\partial h_T}.$$

More precisely, with activation functions included:

$$\frac{\partial \mathbf{h}_t}{\partial \mathbf{h}_k} = \prod_{i=k+1}^{t} \frac{\partial \mathbf{h}_i}{\partial \mathbf{h}_{i-1}} = \prod_{i=k+1}^{t} \operatorname{diag}\!\bigl(\phi'(\mathbf{z}_i)\bigr)\,\mathbf{W}_{hh}.$$

If the spectral radius of $W_h$ is less than 1, the term $(W_h^{\top})^T$ shrinks exponentially with $T$, causing the gradient to vanish. Consequently, weights influencing long-term dependencies receive negligible updates; the network cannot remember relevant events that occurred many steps earlier. If the spectral radius exceeds 1, gradients explode, leading to unstable weight updates and divergence during training.

From a signal-processing perspective, an RNN resembles an infinite impulse response (IIR) filter applied to the input sequence. The eigenvalues of $W_h$ determine how quickly past inputs decay; persistent memory requires that some eigenvalues be close to 1 in magnitude, but then gradients explode. Standard activation functions like $\tanh$ further compress the gradient flow because their derivatives are bounded below 1 (saturating near 0 for large activations). These intertwined effects make standard RNNs ineffective for tasks that demand learning dependencies beyond about 10–20 time steps.

In an economic forecasting context, vanishing gradients mean the model cannot learn that a monetary policy shock twelve quarters ago still propagates through today's inflation — a failure to capture the very long-run dependencies that make sequence analytics valuable. Exploding gradients, conversely, manifest as training divergence: loss spikes, parameter norms blow up, and the optimizer must be rescued by gradient clipping, which rescales the gradient vector whenever $\|\nabla L\| > \tau$ for some threshold $\tau$.

Long Short-Term Memory (LSTM)

The Long Short-Term Memory (LSTM) architecture, introduced by , was explicitly designed to overcome the vanishing gradient problem by introducing a cell state $C_t$ that acts as an internal long-term memory and a gating mechanism that controls the flow of information. The LSTM maintains two parallel state vectors: the hidden state $h_t$ (short-term memory, exposed to downstream layers) and the cell state $C_t$ (long-term memory, protected from nonlinear saturation).

An LSTM unit consists of three gates: the forget gate $f_t$, the input gate $i_t$, and the output gate $o_t$. All gates are sigmoid neurons that output values between 0 and 1, thereby regulating how much information passes through. At each time step:

$ \begin{aligned} f_t &= \sigma\big(W_f \,[h_{t-1}, x_t] + b_f\big) \\[2pt] i_t &= \sigma\big(W_i \,[h_{t-1}, x_t] + b_i\big) \\[2pt] \tilde{C}_t &= \tanh\big(W_C \,[h_{t-1}, x_t] + b_C\big) \\[2pt] C_t &= f_t \odot C_{t-1} \;+\; i_t \odot \tilde{C}_t \\[2pt] o_t &= \sigma\big(W_o \,[h_{t-1}, x_t] + b_o\big) \\[2pt] h_t &= o_t \odot \tanh(C_t) \end{aligned} $

where $[\,\cdot\,,\cdot\,]$ denotes concatenation and $\odot$ is element-wise multiplication.

• The forget gate $f_t$ decides which parts of the previous cell state $C_{t-1}$ to retain. A value close to 1 retains the information, while a value near 0 discards it.
• The input gate $i_t$ determines how much of the new candidate state $\tilde{C}_t$ should be added to the cell state.
• The cell state update $C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$ is linear in $C_{t-1}$, meaning that gradients can flow along the cell state without being repeatedly multiplied by a saturating activation function. This design, often called the constant error carousel, is what mitigates vanishing gradients: the recurrence over $C$ involves only element-wise gating and addition, not a full weight matrix multiplication followed by a squashing nonlinearity.
• The output gate $o_t$ controls how much of the cell state (after a $\tanh$ transformation) is exposed to the hidden state $h_t$.

The economic intuition is illuminating. Think of $C_t$ as a firm's capital stock — a slowly evolving, cumulative quantity — and $h_t$ as the observable output (production, dividends, reported earnings) that the market actually sees. The forget gate corresponds to depreciation and divestment: the firm chooses which assets to write down. The input gate corresponds to investment: which new projects to add. The output gate is the disclosure policy: which aspects of the underlying capital are revealed to external observers. Because the cell update is additive, $\frac{\partial C_t}{\partial C_{t-1}} = \operatorname{diag}(f_t)$, and when the forget gate is close to one, the gradient passes through essentially unchanged.

Gated Recurrent Unit (GRU)

The Gated Recurrent Unit (GRU), proposed by , is a streamlined variant that achieves comparable empirical performance to LSTM with fewer parameters and faster training. The GRU collapses the cell and hidden states into a single vector $h_t$ and replaces the three LSTM gates with two:

$ \begin{aligned} z_t &= \sigma\big(W_z \,[h_{t-1}, x_t] + b_z\big) && \text{(update gate)}\\[2pt] r_t &= \sigma\big(W_r \,[h_{t-1}, x_t] + b_r\big) && \text{(reset gate)}\\[2pt] \tilde{h}_t &= \tanh\big(W\,[r_t \odot h_{t-1},\, x_t] + b\big) \\[2pt] h_t &= (1 - z_t) \odot h_{t-1} \;+\; z_t \odot \tilde{h}_t \end{aligned} $

• The reset gate $r_t$ determines how much of the previous hidden state to "forget" when computing the candidate update $\tilde{h}_t$. If $r_t$ is near 0, the candidate is computed almost solely from the current input $x_t$, effectively allowing the unit to discard all past history when irrelevant.
• The update gate $z_t$ controls the interpolation between the old hidden state $h_{t-1}$ and the new candidate $\tilde{h}_t$. A value close to 1 retains the candidate, while a value close to 0 keeps the old state unchanged. This is analogous to the combined effect of the LSTM's forget and input gates, but without a separate memory cell.

The GRU's update rule is linear in $h_{t-1}$ (thanks to the $(1 - z_t) \odot h_{t-1}$ term), which again provides a direct path for gradients and alleviates the vanishing gradient problem. Because the GRU has fewer parameters (no output gate, no separate cell state), it trains faster and generalizes better on smaller datasets, while often matching LSTM performance on larger corpora.

Concept of Transformer: Self-Attention Mechanism

Originally developed for natural language processing, Transformer Architectures have revolutionized sequential and tabular data analytics. The core innovation of the transformer is the Self-Attention Mechanism, which allows the model to weigh the importance of different parts of the input sequence dynamically. Transformers discard recurrence and convolution entirely, relying instead on the self-attention mechanism to model global dependencies between input and output sequences.

Given an input matrix $X$, the model computes three linear projections: Query ($Q$), Key ($K$), and Value ($V$):

$$Q = XW^Q, \quad K = XW^K, \quad V = XW^V.$$

The core mathematical operation is the Scaled Dot-Product Attention, defined as:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V.$$

Here, the dot product $QK^T$ computes the raw alignment scores between all queries and keys. The scaling factor $\frac{1}{\sqrt{d_k}}$, where $d_k$ is the dimensionality of the key vectors, is mathematically critical. Without it, as $d_k$ grows, the variance of the dot products increases, pushing the softmax function into regions where its gradients are extremely small (the saturation regime), thereby halting learning. The softmax operation normalizes these scaled scores into a probability distribution, which is then used to compute a weighted sum of the value vectors $V$.

To capture diverse linguistic relationships (e.g., syntactic dependencies vs. semantic coreference), Transformers employ Multi-Head Attention. Instead of performing a single attention function, the model linearly projects $Q$, $K$, and $V$ $h$ times with different learned weights to $h$ distinct subspaces:

$$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V),$$ $$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O.$$

Economic Intuition: In behavioral economics and decision theory, agents operate under bounded rationality and must allocate limited cognitive resources to the most relevant market signals. Traditional recurrent networks process information sequentially with decaying memory, akin to an agent who rigidly relies on recent events. In contrast, the self-attention mechanism mathematically formalizes optimal attention allocation. It dynamically assigns higher weights to historically distant but contextually critical events (e.g., a past market crash or a structural policy shift), allowing the model to capture long-range dependencies in time-series forecasting and complex customer journey analytics.

The Transformer architecture enables parallelized training (since there is no sequential dependency between time steps) and captures arbitrarily long-range dependencies through direct attention connections. Pre-trained models (BERT, DistilBERT, GPT) are fine-tuned for downstream tasks via transfer learning — the dominant paradigm in modern NLP.

Autoencoders: Mathematical Foundations and Latent Representations

An autoencoder is an unsupervised neural network trained to reconstruct its input, forcing the learning of a compact latent representation or code. Architecturally, it consists of an encoder $f_\theta: \mathbb{R}^d \rightarrow \mathbb{R}^k$ and a decoder $g_\phi: \mathbb{R}^k \rightarrow \mathbb{R}^d$, with parameters $\theta$ and $\phi$, typically chosen such that $k < d$. Given a dataset $\{x^{(i)}\}_{i=1}^N$, the model is trained to minimize the expected reconstruction error:

$$\mathcal{L}(\theta,\phi) = \frac{1}{N}\sum_{i=1}^N \ell\big(x^{(i)},\, g_\phi(f_\theta(x^{(i)}))\big).$$

For continuous data, $\ell$ is often the mean squared error (MSE): $\ell(x,\hat{x}) = \|x - \hat{x}\|^2$.

Linear Autoencoders and PCA: When both $f$ and $g$ are affine maps and the loss is MSE, the autoencoder's latent space coincides with the principal subspace of the data. Concretely, let $f(x) = W x + b$, $g(z) = W' z + b'$, with $W \in \mathbb{R}^{k \times d}$ and $W' \in \mathbb{R}^{d \times k}$. If we impose tied weights, $W' = W^\top$, the optimal solution yields $W$ whose rows span the top-$k$ principal directions of the covariance matrix. Thus, a linear autoencoder performs linear dimensionality reduction equivalent to PCA.

Non-linear Autoencoders and Manifold Learning: To capture non-linear structures, the encoder and decoder incorporate element-wise non-linearities (e.g., ReLU, sigmoid). The resulting latent code $z = f_\theta(x)$ defines a non-linear embedding of the data manifold. Optimising the reconstruction loss learns a differentiable manifold unwrapping: the decoder maps a low-dimensional code back to the high-dimensional observation space, while the encoder provides the inverse mapping.

Economic Intuition: The autoencoder's information bottleneck mirrors the behavior of a central planner or a financial market facing severe bandwidth constraints. Just as a central bank must distill millions of microeconomic indicators into a single, actionable policy interest rate, the encoder must compress high-dimensional data into a dense latent signal that preserves only the most critical variance. This compressed representation strips away idiosyncratic noise, leaving the systematic factors necessary for downstream predictive modeling.

Applications in data analytics:

• Dimensionality reduction for visualisation: Projecting data to $k=2$ or $3$ dimensions and plotting the latent codes reveals clusters, outliers, and trends. Unlike PCA or t-SNE, autoencoders provide a parametric mapping that can be applied to unseen data.
• Feature extraction for predictive modelling: The latent representation $z$ often disentangles factors of variation better than hand-crafted features.
• Anomaly detection: An autoencoder trained on normal data will yield high reconstruction errors for anomalous samples. Defining an anomaly score as $\|x - g(f(x))\|^2$ is a powerful non-linear extension of the Mahalanobis distance.
• Data denoising: A denoising autoencoder is trained to reconstruct a clean sample $x$ from a corrupted version $\tilde{x}$.

Generative Adversarial Networks (GANs)

While autoencoders compress data, Generative Adversarial Networks excel at synthesizing new, realistic data points. A GAN comprises two competing neural networks: a Generator $G$ and a Discriminator $D$. The generator takes a random noise vector $z \sim p_z(z)$ and maps it to the data space, producing synthetic samples $G(z)$. The discriminator evaluates the probability that a given sample originated from the true data distribution $p_{data}(x)$ rather than the generator.

The training process is formulated as a Minimax Game with the following value function:

$$\min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))].$$

The discriminator seeks to maximize this objective by correctly classifying real and fake data, while the generator seeks to minimize it by producing samples that fool the discriminator.

Optimal Discriminator and the Jensen–Shannon Divergence

For a fixed generator, the functional $V$ is a binary cross-entropy that can be maximised point-wise, yielding the optimal discriminator

$$D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)},$$

where $p_g$ denotes the distribution implicitly defined by $G$. Substituting $D^*$ back into $V$ gives

$$C(G) = \max_D V = -\log 4 + 2\; \text{JS}(p_{\text{data}} \,\|\, p_g),$$

where $\text{JS}$ denotes the Jensen–Shannon divergence. Therefore, the generator's task is equivalent to minimising the JS divergence between $p_{\text{data}}$ and $p_g$. The game reaches equilibrium when $p_g = p_{\text{data}}$.

Training Dynamics and Wasserstein GANs

In practice, simultaneous gradient-based optimisation is fragile. The JS divergence may fail to provide meaningful gradients when the support of the two distributions has low overlap, leading to vanishing gradients and mode collapse (the generator produces only a few distinct samples). The Wasserstein GAN (WGAN) replaces the JS divergence with the Earth Mover (Wasserstein-1) distance:

$$W(p_r, p_g) = \inf_{\gamma \in \Pi(p_r,p_g)} \mathbb{E}_{(x,y)\sim \gamma}[\|x - y\|].$$

Using the Kantorovich–Rubinstein duality, $W(p_r, p_g) = \sup_{\|f\|_L \le 1} \mathbb{E}_{x \sim p_r}[f(x)] - \mathbb{E}_{x \sim p_g}[f(x)]$, where $f$ is a 1-Lipschitz function. The discriminator (here called the critic) is trained to approximate this supremum, and the Lipschitz constraint is enforced via a gradient penalty. WGAN provides more stable training and meaningful loss curves that correlate with sample quality.

Economic Intuition: The GAN framework is a mathematical formalization of a regulatory cat-and-mouse game, akin to the dynamic between counterfeiters and auditors in a financial system. The generator acts as the counterfeiter, continuously innovating to exploit loopholes and produce indistinguishable fake assets. The discriminator acts as the auditor, updating its detection heuristics in response. This adversarial competition drives the system toward a Nash equilibrium where the synthetic data perfectly mimics the empirical distribution. In data analytics, this is invaluable for generating synthetic tabular data or augmenting minority classes in imbalanced datasets.