COSMICOUS

About the series

You had to wait till this fourth part of my series for discussions on Neural Networks, even though Neural Networks were the first ones to come into the realm of ML/AI and enjoying leadership position now. I would personally refer to these algorithms based on Neural Networks as ‘Brain Works.’

You can read my earlier parts of this series:

 ‘Seen it before’ or Supervised algorithms were the subject of discussions in the second part  (https://ai-positive.com/2024/10/20/demystifying-ai-ml-algorithms-part-ii-supervised-algorithms-2/). The series started with my treatment of Good-Old-Fashioned-AI that gave a real start to practical use of AI (https://ai-positive.com/2024/08/28/understanding-gofai-rules-rule-and-symbolic-reasoning-in-ai/).

Neural Network’s Nobel Journey

Perceptron was one of the earliest incarnations of neural network models, developed by Frank Rosenblatt in 1958. Almost every decade starting from 1960s had newer developments in Neural Networks – Adaline in 1960, Backpropagation in 1974, Recurrent and Convolutional Neural Networks in 1980s, Long Short-Term Memory in 1997 followed by Generative Adversarial Networks in 2014, Diffusion Models in 2015 and Transformer in 2017, which transformed the AI scene into Generative AI, making Neural Networks the darling of today’s AI scene.  

To top it all, the 2024 Nobel Prizes in Physics and Chemistry both have fascinating connections to neural networks. John Hopfield and Geoffrey Hinton were awarded the Nobel Prize in Physics, recognizing Hopfield network invented by John Hopfield and Boltzmann machine developed by Geoffrey Hinton. David Baker, Demis Hassabis, and John Jumper received the Nobel Prize in Chemistry for their contributions to computational protein design and protein structure prediction. Hassabis and Jumper developed AlphaFold2, an AI model that predicts protein structures with remarkable accuracy. Nobel Prizes added noble stature to Neural Networks.

How does Neural Network Work?

Following points detail the structure and working of Neural Network:

1. Neurons (Nodes): Similar to biological neuron, basic unit of a Neural Network is a node appropriately named as a neuron. They are organized into layers.
2. Layers: Starting with an input layer, followed by one or more hidden layers, and an output layer form the Neural Network. Each layer contains multiple neurons. Input layer receives the input data. Hidden layers process the data through a series of transformations and output layer produces final output or prediction.
3. Input Data: When data enters the input layer, each feature is assigned to a neuron.

4. Weights and Biases: Connection between neurons has a weight associated with it that determines the strength of the connection. Neurons also have a bias value that adjusts the output along with the weighted sum.
5. Activation Function: Each neuron has an activation function which is applied to the weighted sum of its inputs plus the bias.
6. Forward Propagation: The input data passes through the layers of the network, with each neuron computing its output based on activation function and passing it to the neurons in the next layer.
7. Output: The last layer produces the output of the Neural Network.

Brain works

The reason I refer to Neural Networks as ‘Brain works’ is that Artificial Neural Networks (ANN) are inspired by the structure and workings of the brain of living beings as explained below:

1. Neurons and Nodes: In the brain, neurons are the fundamental units that process and transmit information. Similarly, in ANNs, nodes referred to as artificial neurons serve as the basic units of computation.
2. Synapses and Weights: Neurons in the brain are connected by synapses, which facilitate the transfer of information through neurotransmitters. In ANNs, connections between nodes are represented by weights, the strength of which influence the connection.
3. Layers: The brain is organized in a manner, with different areas responsible for distinct types of processing. Likewise, ANNs have layers where each layer performs specific computations.
4. Activation Function: In the brain, a neuron fires when it reaches a certain threshold of excitation. In ANNs, an activation function determines if a node should produce an output or not, simulating this firing mechanism.

Assembly Line Analogy

Before discussing aspects related to training of Neural Networks, let us look at assembly in a manufacturing unit as rough analogy to understand the concept behind how Neural Network works.

1. Input Layer (Starting Point):
   • Beginning of the assembly line is where raw materials / components (inputs) are introduced. This is like where data enters the Neural Network. In case of car manufacturing, the raw materials such as steel fabrications, tyres, engines may enter the assembly line.
2. Hidden Layers (Stations):
   • Each station on the assembly line represents a hidden layer in the neural network. At each station, workers (neurons) take the incoming materials (data), process them, and pass them on to the next station. (In case of building a car, the first station could frame the body of the car. The second station may install engine, while the third station could add wheels, and the next station may paint).
   • Weights (Tools and Techniques): The tools and techniques used by workers to process the materials represent the weights. They are like the influence each neuron carries.
   • Biases (Adjustments): Just like adjustments made by workers to ensure the specifications of the product, biases adjust the processing to improve accuracy.
   • Activation Function (Quality Check): Each station has a quality check mechanism (activation function) to decide if the processed material should move forward in the assembly line.

Due to highly automated nature of car manufacturing, there may be fewer workers in each station and automation has taken up the role of workers using the tools & applying techniques at each station. Automated process controls handle the movement from one station to the next according to the product specification and quality requirements.

3. Output Layer (End of the Line):
   • The final station on the assembly line is where the fully processed product comes out. The final, complete car rolls off the line, ready to be driven or test driven. This is the output layer where the final prediction or result of Neural Network is produced.

Training the Neural Network

“Cells that fire together, wire together,” is the core idea of how brain learns by adjusting the strength of synapses. When a neuron in brain repeatedly activates another neuron, the synaptic connection between them becomes stronger. Such repeated stimulation of a synapse leads to a long-lasting increase in its strength. Experiences, learning, and memory formation shape neural circuits in the brain. Adopting this idea, Neural Networks are made to learn through training using large data sets representing the context of the problem. Training involves the following steps:

1. Data Preparation:

Gather a dataset relevant to the problem to be solved. Clean the data by removing noise, handling missing values, and normalizing it to a suitable range.

2. Network Initialization:

Choose the type of neural network (see below for popular types of Neural Networks) and define its characteristics such as number of layers, types of layers, number of neurons in each layer. Initialize with random weights for the connections between neurons.

3. Forward Propagation to produce output:

Pass a batch of input data to the first layer. At each layer, compute the output by applying the activation functions to the weighted sum of inputs. Produce the final output of the network.

4. Improving Output:

Compare the network’s output with the actual target values using a loss function. Calculate the loss, which quantifies how far the outputs are from the actual values. Update the weights from the last layer to the first through Back Propagation. Technique called Gradient Descent is used for calculating the loss and updating the weights to minimize the loss.

5. Iteration:

Iterate through forward propagation, loss calculation, and backpropagation until the network’s performance improves. A complete pass through the dataset is referred to as an epoch. Usually, the data is divided into batches, and the weights are updated after processing each batch through multiple epochs.

6. Evaluation:

Evaluate the network on a separate validation data set to check for its performance. Adjust parameters such as number of neurons, number of layers, number of epochs/ data batches etc., and retrain if necessary to improve performance. These parameters are called Hyper Parameters.

Deep learning is the term used for neural networks with many layers to model and understand complex patterns in large datasets.

Popular Neural Networks:

1. Feedforward Neural Networks (FNN): The simplest type of artificial neural network where the information moves in one direction—from input nodes, through hidden nodes to output nodes. They are widely used for pattern recognition.

2. Convolutional Neural Networks (CNN): Primarily used for image and video recognition tasks. They are designed to learn spatial hierarchies of features automatically and adaptively from input images. This works like magnifying glass used to look at small parts of an image to recognize prominent parts. Feature maps of such prominent parts aid in deciding what the image is.

3. Recurrent Neural Networks (RNN): Suitable for sequential data or time series prediction. They have connections that form directed cycles, allowing them to maintain a ‘memory’ of previous inputs. Even in the world of Transformer Networks (see below), RNNs are more effective for applications where data arrives in a continuous stream and decisions need to be made on-the-fly such as Real-Time Speech Recognition and Stock Price prediction.

4. Long Short-Term Memory Networks (LSTM): A type of RNN that can learn long-term dependencies. They are particularly effective for tasks where the context of previous data points is important, such as language modelling. While Transformer Network (see below) does this better, LSTM is preferred for smaller datasets or simpler tasks, as it is easier to implement, and train compared to transformers.

5. Generative Adversarial Networks (GANs): Consist of two neural networks, a generator and a discriminator, which compete against each other. GAN works like iterative constructive criticism of a critic against a creator’s work to improve the creator’s output to make it more realistic. They are used for generating synthetic instances of data, restoring damaged photos by filling up missing parts and for predicting future frames in videos as required in autonomous driving.

6. Transformer Networks: They use mechanisms called attention to weigh the influence of various parts of the input data. Based on the famous research paper ‘Attention Is All You Need,’ from Google researchers, Transformer is the major component of today’s Generative AI – GenAI.

Brain Works or Selfies or Seen-It-Before?

While it is true that many problems solved by traditional statistical machine learning (ML) algorithms such as Selfies and Seen-It-Before, discussed in the previous two parts can also be tackled by neural networks, following are some nuances to consider:

1. Flexibility and Power: Neural networks, especially deep learning models, are highly flexible and powerful. They can model complex, non-linear relationships in data, making them suitable for a wide range of tasks, from image recognition to natural language processing.

2. Data Requirements: Neural networks typically require enormous amounts of data to perform well. Traditional statistical ML algorithms, like linear regression or decision trees is good enough for smaller datasets.

3. Interpretability: Traditional ML algorithms are more interpretable while Neural networks are often considered “black boxes” due to their complexity.

4. Computational Resources: Neural networks, especially deep learning models, require significant computational resources for training. Traditional ML algorithms are usually less resource-intensive and can be cost effective for certain applications.

5. Specific Use Cases: Some problems are better suited to traditional ML algorithms due to their simplicity and efficiency. For example, logistic regression is often used for binary classification tasks, and k-means clustering remains a popular choice for unsupervised learning tasks.

I hope the four parts would have provided a conceptual understanding of essential ML/AI algorithms. We will review how to make an optimum choice of ML/AI algorithms in a later part of this series.

About the series

In this third part of the series on the basics of AI/ML algorithms, I would deal with so called Un Supervised algorithms, which I refer to as Selfies. ‘Seen it before’ or Supervised algorithms were the subject of discussions in the second part ( https://ai-positive.com/2024/10/20/demystifying-ai-ml-algorithms-part-ii-supervised-algorithms-2/). The series started with my treatment of Good-Old-Fashioned-AI that gave a real start to practical use of AI (https://ai-positive.com/2024/08/28/understanding-gofai-rules-rule-and-symbolic-reasoning-in-ai/).

Getting rid of teacher

All algorithms we discussed in the second part require data with label, the outcome which the teacher as a trainer relate to the other input variables in the data to identify pattern. The output of the machine learning algorithms is a trained model which when fed with a new data can predict the label to which the new data belong, hence the outcome. Finding a good teacher is always a challenging task. What if we must automatically find the pattern in the data without explicit label?

Child learns within the first 3 years after born without a real teacher! They observe, listen, touch, taste, and smell everything they encounter which helps them learn about their environment. Imitation and play enable child to learn quickly. The logic of learning is already there in child’s mind. It becomes human nature to group things together or categorize them to make better sense of things. We see stars and constellations appear. Unsupervised algorithms in that sense are selfies which find hidden patterns, structures, and relationship within the data. There are several popular unsupervised learning algorithms widely used for machine learning. Let us look at most common ones.

K-Means Clustering: This algorithm partitions data into K distinct clusters based on the distance to the centroids of those clusters. This is like separating items of particular colour from mixed items of various colours. K indicates number of clusters to be formed. Popular implementations of K-Means clustering algorithms are:

Customer Segmentation: Grouping customers based on purchasing behaviour for targeted marketing.

Image Compression: Reducing the number of colours in an image by clustering similar colours together.

Document Classification: Organizing documents into topics or categories based on their content.

Hierarchical Clustering: It is a method of clustering that builds a hierarchy of clusters, which can be visualized as a dendrogram. Hierarchical clustering works something like this. Suppose you have group of students in a class and want to form groups based on similar interests for club activities. Initially each student is a cluster. Identify how similar each student is to every other student based on their interests. Group together two students who have most similar interests. Now find similarity of this new group to other students or other groups so formed. Repeat the process until everyone is part of any of the groups.

 Typical use cases of Hierarchical Clustering are:

Gene Expression Analysis: Grouping genes with similar expression patterns in biological research.

Market Research: Segmenting markets based on consumer preferences and behaviours.

Social Network Analysis: Identifying communities within social networks.

DBSCAN (Density-Based Spatial Clustering of Applications with Noise): DBSCAN groups together points that are close to each other based on their density and marks points that are in low-density regions as outliers/ noise. It uses parameters like radius and minimum points to define dense regions and expands clusters from core points.

Suppose you are at a crowded party, and you want to figure out who are loners. Starting with a person check who are all close by. Anyone within the close range is part of same group. For each of the new group members, you again see who is close to them and keep adding them to the group. If someone is not near enough to form a group, they are considered an outsider or noise. The process is to be repeated till everyone at the party is either part of a group or classified as noise.

It is quite natural that DBSCAN is used for these applications:

Anomaly Detection: Identifying outliers in datasets, such as fraudulent transactions.

Geospatial Analysis: Detecting clusters of geographic locations, like hotspots in crime data.

Astronomy: Clustering stars or galaxies based on their characteristics.

Apriori Algorithm: Apriori algorithm is a learning method which discovers frequent itemsets in data. It then generates several association rules for those set of items. By calculating two factors – namely ‘confidence’ and ‘lift,’ Apriori algorithm eliminates rules which do not meet the minimum requirement and retains only those rules that qualify.

This algorithm works in context of a supermarket to identify items which are bought together frequently. It looks first at individual items such as soap or shampoo and counts how often they are bought. Retaining items that are bought frequently enough (above a particular number of times in a period like week), the algorithm looks at pairs of these times such as ‘soap and shampoo’ to see how often they are bought together. Retaining further only those pairs bought together frequently enough, the algorithm looks for larger sets of items like soap, shampoo and possibly conditioner and repeats the process. It keeps expanding and counting sets of items filtering out the ones that are not bought often enough together. The process results in item sets that are frequently bought together which helps to understand customer behaviour to make decisions by supermarket management.

Apriori Algorithm is used for:

Market Basket Analysis: Finding frequent item sets in transactional data to understand buying patterns.

Recommendation Systems: Suggesting products to customers based on their purchase history as well as other customer’s purchase history.

Web Usage Mining: Identifying common patterns in web navigation behaviour.

Self-Organizing Maps (SOM): SOM is used for clustering and visualization of high-dimensional data, i.e., data with several features/ variables. Preserving the topological structure of the original data, it creates a lower dimensional grid of computational units (called neurons) making it easier to identify patterns, clusters, and relationships.

SOM can be used to visualize the similarities between songs breaking down higher dimensional features such as Tempo, Genre, Duration, Energy, Danceability, Loudness, Musical key, Acousticness, Valence, Instrumentalness and Speechiness into a 2 dimensional grid with top-left cluster containing songs with high energy, fast tempo and high danceability (dance and electronic music category) and bottom right with high acousticness, low energy and high instrumentalness (classical, acoustic music category) and the centre containing songs with moderate energy, positive valence and high speechiness (pop and hip-hop music category).

Typical real-life applications of SOMs include:

Speech/ Handwriting Recognition: Recognizing patterns in complex datasets, such as speech or handwriting.

Social Network Analysis: Visualizing the structure of social networks and identifying communities or influential individual within the network.

Manufacturing Process: Used to monitor the health of machinery and detect potential failures based on the pattern of sensor data on temperature, vibration, and acoustic emissions and identifying deviations from the normal patterns indicating potential issues.

Principal Component Analysis (PCA): Reduces the dimensionality of data by transforming it into a new set of variables (principal components) that capture the most variance. Dimensionality refers to the number of features or variables in a dataset. In other words, PCA simplifies the data while preserving the variance as much as possible so that resulting data is easier to visualize and analyse. PCA is used as pre-processing step to reduce the number of features prior to applying any other machine learning algorithm to build a model.

Imagine you have a huge photo album, and each photo has several details like people, locations, activities, attires, and dates. It would be overwhelming to look through every photo and find key moments. We can identify key features or common themes such as weddings, birth days, vacations that most photos share and group them according to the theme. Choose a few representative photos from each group that capture the essence of the theme, which will highlight the key moments and people. PCA works like this.

Most common use cases of PCA are:

Face Recognition: Identifying prominent features in facial images for recognition, typically used by the police to identify a criminal from the description by a witness.

Stock Market Analysis: PCA is used to analyse and reduce the dimensionality of financial data, to identify the most key factors affecting stock prices and make informed decisions.

Environmental Studies: Analysing environmental data such as air quality and water pollution to determine the main sources of pollution to develop strategy for environmental protection.

Seen-it-before or Selfies – which way to go?

Selfies are the best for exploratory tasks such as customer segmentation, anonymity detection and market basket analysis when you do not have labels.

Selfies focus on exploration and discovering insights from data without pre-defined labels, but not on accuracy that can be obtained from Seen-it-before algorithms.

Selfie algorithms can be used to pre-train a model or extract features from data which then can be fed into Seen-it-before algorithms for building models with more accurate prediction.

There is also a cross between Seen-it-before algorithms and Selfies where a small amount of labelled data is combined with a large amount of unlabelled data to improve learning accuracy iteratively.

Ensemble method of using multiple models from both categories are used to arrive at the most accurate final model.