Pattern recognition is the science of assigning a label or category to an input: is this email spam, is that digit a 7, does this scan show a tumour, is this transaction fraudulent? It is the classical foundation of machine learning. The models are older than today’s deep learning, but the vocabulary, training and test sets, features, overfitting and decision boundaries is the same one used in every ML pipeline. This article revisits the five units of the JNTUK course and ends the Engineering Revisited series.
Unit 1: Introduction and Bayesian decision theory
The pattern recognition system
| Stage | Activity |
|---|---|
| Sensing and acquisition | Collect raw data: images, audio, sensor readings, logs |
| Preprocessing | Remove noise, normalise, segment |
| Feature extraction | Compute measurements that capture what matters |
| Classification | Decide the class from the feature vector |
| Post-processing | Combine results, apply context and costs |
A pattern is represented as a feature vector x = (x₁, …, xd) in feature space. Approaches: statistical (probability models), syntactic or structural (grammars and graphs), neural and template matching. Learning can be supervised (labelled examples), unsupervised (no labels) or semi- and reinforcement learning. The workflow: collect data, choose features, choose a model, train, evaluate on separate test data, then deploy.
Bayesian decision theory
For classes ω₁, …, ωc, Bayes’ rule gives the posterior probability:
P(ωᵢ | x) = p(x | ωᵢ) P(ωᵢ) / p(x)
where P(ωᵢ) is the prior, p(x | ωᵢ) the likelihood and p(x) the evidence (a normaliser). The Bayes classifier picks the class with the largest posterior. It minimises the probability of error and is the best any classifier can do, if the probabilities are known.
| Idea | Meaning |
|---|---|
| Loss function and risk | Assign costs to mistakes (missing a disease is worse than a false alarm) and choose the action with minimum expected risk |
| Zero-one loss | All errors cost the same; gives the maximum-posterior rule |
| Discriminant functions gᵢ(x) | Choose the class with the largest gᵢ(x); the decision boundary is where two are equal |
| Gaussian case | If class densities are normal, the boundary is a line, plane or quadratic surface; with equal covariances it is linear |
| Reject option | Refuse to decide when the posteriors are close |
Receiver operating characteristic (ROC) curves plot the true-positive against the false-positive rate as the threshold varies.
Today: a spam filter or fraud model with a probability threshold is a Bayes decision with unequal costs. The choice of an alert threshold, how many false alarms to accept to catch real incidents, is the same trade-off.
Unit 2: Parameter estimation
The Bayes classifier needs p(x | ωᵢ). In practice we assume a form (usually Gaussian) and estimate its parameters from training data.
Maximum likelihood estimation (MLE)
Choose the parameters θ that make the observed training data most probable. For a Gaussian, the MLE of the mean is the sample mean and of the variance the sample variance (with the divisor n; the unbiased version divides by n − 1). The likelihood is usually maximised through its logarithm.
Bayesian estimation
Treat θ as a random variable with a prior p(θ), and use the data to compute a posterior p(θ | data). With a lot of data the two approaches agree; with little data the prior matters, which is useful when data is scarce.
Other topics
| Topic | Idea |
|---|---|
| Gaussian mixture models (GMM) and EM | Model a distribution as a weighted sum of Gaussians; the expectation-maximisation algorithm alternates between assigning points to components (E-step) and updating parameters (M-step) |
| Hidden Markov models (HMM) | Model sequences with hidden states and observed outputs; three problems: evaluation (forward algorithm), decoding (Viterbi), learning (Baum-Welch) |
| Naive Bayes | Assume features are independent given the class, which makes estimation easy even in many dimensions |
| Bias-variance trade-off | Simple models have high bias; complex ones have high variance and overfit |
| Curse of dimensionality | With many features, the data needed grows exponentially, and distances become less meaningful |
Today: MLE is the principle behind training logistic regression and neural networks (minimising cross-entropy loss is maximising likelihood). HMMs powered early speech recognition and are still used for sequence tagging. EM and GMMs are used for clustering and anomaly detection.
Unit 3: Non-parametric methods and linear discriminant functions
Non-parametric techniques
When no assumed form fits, estimate the density or the decision directly from the data.
| Method | Idea |
|---|---|
| Histogram | Count samples in bins |
| Parzen windows (kernel density) | Put a small “bump” (kernel) on each sample and add them up; the window width controls smoothness |
| k-nearest neighbour (k-NN) estimation | Density from the volume needed to enclose k samples |
| k-NN classifier | Label a point by the majority class of its k nearest training samples; k = 1 is the nearest-neighbour rule, whose error is at most twice the Bayes error with enough data |
Distance measures: Euclidean, Manhattan, Mahalanobis (accounts for correlation), cosine. k-NN is simple and needs no training but is slow at prediction (use KD-trees, approximate search) and sensitive to feature scaling.
Linear discriminant functions
A linear discriminant g(x) = wᵀx + w₀ separates classes with a hyperplane; the sign of g(x) gives the class. Learning means finding w.
| Algorithm | Idea |
|---|---|
| Perceptron | Start with any w; for each misclassified sample update w ← w + η y x; converges if the classes are linearly separable |
| Least-squares (MSE) and Widrow-Hoff (LMS) | Minimise the squared error; works when not separable |
| Fisher’s linear discriminant | Project the data onto the line that maximises the ratio of between-class to within-class scatter |
| Logistic regression | Model the class probability with a sigmoid of wᵀx; trained by gradient descent |
| Support vector machine (SVM) | Choose the hyperplane with the maximum margin between the classes; only the support vectors matter; soft margins allow errors; the kernel trick (polynomial, RBF) handles non-linear boundaries |
A single perceptron cannot learn XOR, which is not linearly separable; this limitation motivates multi-layer networks.
Today: logistic regression is the workhorse baseline of industry classification, and SVMs were the best general-purpose classifiers before deep learning. k-NN is the basis of vector search for embeddings, now used in recommendation, semantic search and retrieval for AI applications.
Unit 4: Feature extraction and dimensionality reduction
Good features matter more than the choice of classifier. Dimensionality reduction removes redundancy, noise and the curse of dimensionality, and makes data easier to visualise.
| Approach | Idea |
|---|---|
| Feature selection | Pick a subset of the original features: filter methods (correlation, mutual information), wrapper methods (try subsets with the classifier), embedded methods (L1 regularisation, tree importance) |
| Feature extraction | Create new features as combinations of the old ones |
Principal component analysis (PCA)
PCA finds the directions of maximum variance in the data.
- Subtract the mean from the data.
- Compute the covariance matrix.
- Find its eigenvalues and eigenvectors; the eigenvectors are the principal components.
- Keep the k components with the largest eigenvalues and project the data onto them.
The share of total variance kept is the sum of the chosen eigenvalues divided by the sum of all. PCA is unsupervised and linear, and it is sensitive to feature scaling.
Linear discriminant analysis (LDA)
LDA is supervised: it finds directions that best separate the classes. For c classes it gives at most c − 1 useful dimensions.
Other methods
ICA (independent components, separating mixed signals), t-SNE and UMAP (non-linear visualisation of high-dimensional data), autoencoders (neural networks that learn compressed representations), and manifold learning. Domain-specific features: edges, corners and HOG/SIFT in images, MFCCs in speech, n-grams and TF-IDF in text. Deep learning largely learns features automatically instead of requiring hand-designed ones.
Today: PCA is used to compress and visualise data and to speed up models, and the eigenvalue computation from Mathematical Methods is exactly what it uses. t-SNE and UMAP plots show embeddings of text, images and logs. Feature stores and feature engineering are central parts of production ML pipelines.
Unit 5: Clustering, neural networks and evaluation
Unsupervised learning and clustering
With no labels, clustering groups similar patterns.
| Method | Idea |
|---|---|
| k-means | Choose k centres; assign each point to the nearest, recompute centres, repeat until stable; needs k; local optima |
| Hierarchical (agglomerative) | Repeatedly merge the two closest clusters; read a dendrogram; linkage choices: single, complete, average |
| Mixture models (EM) | Soft assignments to Gaussian components |
| DBSCAN | Density-based; finds clusters of any shape and marks noise |
| Spectral clustering, SOM | Graph-based and neural methods |
Choosing the number of clusters uses the elbow method or the silhouette score. Related: anomaly (outlier) detection.
Neural networks
A neuron computes an activation of a weighted sum: y = f(wᵀx + b). A multi-layer perceptron (MLP) has input, hidden and output layers and can learn non-linear boundaries. Training uses backpropagation: compute the error at the output, propagate gradients backward with the chain rule and update the weights by gradient descent. Key ideas: activation functions (sigmoid, tanh, ReLU, softmax), loss functions (cross-entropy), learning rate, regularisation (weight decay, dropout) and early stopping.
| Deep architecture | Use |
|---|---|
| Convolutional neural networks (CNN) | Images and video |
| Recurrent networks, LSTM | Sequences |
| Transformers | Language, and increasingly images and audio; the basis of today’s large language models |
Evaluation and good practice
| Concept | Meaning |
|---|---|
| Train, validation, test split | Tune on validation data and report on unseen test data |
| Cross-validation | Rotate the held-out portion (k-fold) |
| Confusion matrix | True and false positives and negatives |
| Accuracy, precision, recall, F1, ROC-AUC | Choose by the cost of errors; accuracy misleads on imbalanced data |
| Overfitting and regularisation | Training accuracy far above test accuracy means the model memorised |
| Combining classifiers | Bagging, random forests, boosting; ensembles often win |
| Data drift | Real-world data changes; monitor and retrain |
Applications
Optical character recognition, speech and speaker recognition, face and fingerprint recognition, medical diagnosis, fraud and intrusion detection, recommendation, industrial inspection, handwriting and signature verification, and remote sensing.
Today: this is the base of MLOps: versioned data and models, training pipelines, evaluation gates, deployment (often on Kubernetes with GPUs), monitoring for drift, and retraining. A DevOps engineer who knows what a train-test split, overfitting and precision and recall mean can work with data-science teams effectively, and tools such as the ones taught in AI infrastructure courses (vLLM, GPUs on Kubernetes) serve the models described in this unit.
One-minute recap
| Unit | Key idea | You meet it in |
|---|---|---|
| 1. Bayesian decision theory | Posterior, priors, loss and risk, decision boundaries | Spam and fraud thresholds, alert trade-offs |
| 2. Parameter estimation | MLE, Bayesian estimation, EM, HMM, curse of dimensionality | Training with cross-entropy, sequence models |
| 3. Non-parametric and linear methods | Parzen, k-NN, perceptron, SVM, logistic regression | Baselines, vector search, embedding retrieval |
| 4. Feature extraction | PCA, LDA, feature selection, t-SNE | Data compression, embedding plots, feature stores |
| 5. Clustering and neural networks | k-means, DBSCAN, backpropagation, CNNs, transformers, evaluation | MLOps, drift monitoring, LLM serving |
That completes Engineering Revisited, from Applied Physics to Pattern Recognition.
Written by Claude, Anthropic’s AI assistant, as part of Engineering Revisited, a quick refresher of the B.Tech subjects one at a time.