Prerequisites
Before diving into GenAI, it is important to have a solid foundation. Only by having a sturdy base, will you be able to build upon your knowledge of AI. Skipping some of these fundamental elements can lead to confusion and hinder your progress. Here are some prerequisites that you should have
- 3+ years of software experience
- 1+ years of experience with Python
- 1+ years of experience with UNIX/LINUX
Level 1 Topics
Machine Learning
Machine learning is a type of artificial intelligence where computers learn from data without being explicitly programmed. By analyzing data, they can identify patterns and make predictions on new data, becoming increasingly accurate over time.
Perceptron
A perceptron is the building block of neural networks in machine learning. It’s a simple algorithm for binary classification tasks. Imagine a single neuron receiving inputs, like features of an image. The perceptron assigns weights to these inputs and sums them. If the sum is above a certain threshold, it outputs a 1, otherwise a 0. Perceptrons learn by adjusting weights to improve their classification accuracy. While limited to binary problems, they are a foundational concept for understanding more complex neural networks.
Supervised learning
Supervised learning is a machine learning technique where algorithms learn from labeled data. This data acts like a training manual, with clear examples of inputs and their corresponding desired outputs. By analyzing these pairs, the algorithm learns the relationship between them and can then predict outputs for new, unseen data. Imagine a student learning shapes from a teacher. Supervised learning works similarly, allowing machines to learn and make predictions based on labeled examples.
Unsupervised learning
In unsupervised machine learning, algorithms work with data that lacks predefined labels or categories. Unlike supervised learning where the data comes with clear instructions, unsupervised learning algorithms are tasked with finding hidden patterns and structures within the data itself. They achieve this by analyzing the data to group similar elements together, identify hidden categories, or find anomalies. This allows them to uncover interesting insights and prepare the data for further analysis or even generate entirely new data.
Reinforcement learning
In reinforcement learning, an agent interacts with an environment by taking actions and receiving rewards. Through trial and error, the agent learns which actions lead to the most reward, constantly refining its strategy. This method mimics how humans learn by experimentation and is useful for tasks where the best course of action isn’t explicitly defined.
Regression, classification
Regression and classification are two fundamental tasks in machine learning. Regression algorithms predict continuous values, like housing prices or weather forecasts. Classification algorithms, on the other hand, predict discrete categories, such as whether an email is spam or an image contains a cat. Essentially, regression predicts “what” while classification predicts “which”.
MLP
In machine learning, an MLP, or Multilayer Perceptron, is a fundamental type of artificial neural network. It consists of layers of interconnected nodes, inspired by the human brain. Unlike simpler models, MLPs have multiple “hidden layers” between the input and output layers. These layers allow MLPs to learn complex patterns in data that aren’t easily separated with straight lines. This makes them useful for tasks like image recognition, spam filtering, and even playing games.
Overfitting, Underfitting
Overfitting is a modeling error in machine learning when a model is excessively complex and performs well on training data but poorly on new, unseen data. It happens when the model learns the detail and noise in the training data to the extent that it negatively impacts the performance of the model on new data.
Data, Label
In machine learning, data is the raw material, like images, text, or numbers. It holds the information the model needs to learn from. Labels are added information that tells the model what the data represents. Think of data as ingredients and labels as instructions in a recipe – both are crucial for the model to learn and make predictions.
ReLU, Sigmoid
ReLU (Rectified Linear Unit) and sigmoid are activation functions used in machine learning’s artificial neurons. They determine how the neuron processes information. ReLU acts like a switch, only firing if the input is positive. This makes it faster to compute and avoids vanishing gradients in deep networks. Sigmoid squishes values between 0 and 1, but can suffer from vanishing gradients and may not be ideal for all tasks.
Backpropagation
Backpropagation is a machine learning algorithm used in neural networks to adjust the weights and biases in response to errors, effectively “learning” from the mistakes. It does so by propagating the error backwards through the network, hence the name, and then using gradient descent to iteratively fine-tune the network parameters until the model’s predictions are as accurate as possible.
Calculus
Calculus is a branch of mathematics that studies continuous change, primarily through the concepts of differentiation and integration. Differentiation measures the rate of change in a function, while integration accumulates the quantities produced by a function.
Linear Algebra
Linear algebra is the branch of mathematics focused on vectors, matrices, and linear equations. It provides the foundation for many machine learning algorithms, allowing us to represent data, perform transformations, and solve complex problems efficiently.
Vector math
Vector math extends regular arithmetic to objects with both magnitude (size) and direction. Imagine arrows representing forces or velocities. Vector addition considers both the length and direction of the arrows to find a resultant arrow. This allows us to analyze and manipulate quantities with direction, crucial in physics, engineering, and many machine learning applications.
Unicode
Unicode 1991 refers to the first version of the Unicode Standard published by the Unicode Consortium in 1991. It laid the foundation for character encoding and representation standards used in modern computing systems, facilitating the consistent handling of text across different platforms and languages.
Cloze Test
The Cloze Test is a linguistic assessment method used to evaluate language comprehension and proficiency. In this test, participants are presented with a passage of text in which certain words are removed and replaced with blanks. The participants must then fill in the blanks with the appropriate words based on the context of the remaining text. The Cloze Test is commonly used in language education and psycholinguistics to measure reading ability, vocabulary knowledge, and overall language understanding. It serves as a diagnostic tool to identify areas of strength and weakness in a learner’s linguistic abilities.
SGD
Stochastic Gradient Descent (SGD) is an iterative algorithm used in machine learning and deep learning to find the optimal parameters that minimize a function, often a loss function. Unlike the standard Gradient Descent that uses all data points to compute the gradient, SGD randomly selects a batch of data points per iteration, significantly speeding up the process and reducing computational load.
Momentum
Momentum is a technique used in optimization algorithms, such as Gradient Descent, to speed up learning and avoid local minima by adding a fraction of the direction of the previous step to a current step. This way, the algorithm accumulates the gradient of the past steps to determine the direction to go, somewhat similar to a ball rolling downhill.
AdaGrad
AdaGrad (Adaptive Gradient Algorithm) is an optimization algorithm in machine learning that adapts the learning rate to the parameters, performing smaller updates for parameters associated with frequently occurring features and larger updates for parameters associated with infrequently occurring features. It is particularly useful in scenarios where data is sparse and the learning rate needs to be adaptive.
RMSProp
RMSProp (Root Mean Square Propagation) is an adaptive learning rate optimization algorithm for neural networks, designed to attenuate the aggressively decreasing learning rate in conventional gradient descent methods. It adjusts the learning rate by dividing it by an exponentially decaying average of squared gradients, providing an individual learning rate for each parameter, effectively resolving the issue of diminishing learning rates in deep learning.
ADAM Optimizer
The ADAM (Adaptive Moment Estimation) Optimizer is a machine learning algorithm that calculates individual adaptive learning rates for different parameters, combining the advantages of two other extensions of stochastic gradient descent: AdaGrad and RMSProp. It’s particularly effective in settings where data and/or resources are sparse and computation is expensive.
Regularization
Regularization is a technique used in machine learning models to prevent overfitting by adding a penalty term to the loss function, which in turn reduces the complexity of the model. It helps to maintain a balance between bias and variance, ensuring that the model generalizes well on unseen data.
Dropout
Dropout is a regularization technique used in neural networks to prevent overfitting by randomly dropping out, or deactivating, a proportion of neurons during training. This forces the network to learn more robust features that are useful in conjunction with many different random subsets of the other neurons.
Layer Norm
Layer normalization is a technique in machine learning that stabilizes the training process of neural networks. It works by normalizing the activations (outputs) of each neuron within a hidden layer, independently across different training examples. This helps address a phenomenon called internal covariate shift, where the distribution of activations can change throughout training, hindering learning. Layer normalization improves gradient flow, allowing for faster training and better generalization performance of the model.
Batch Normalization
Batch normalization is a technique used during training in deep neural networks. It addresses a problem called internal covariate shift, where the distribution of data changes between layers as the network learns. Batch normalization normalizes the activations of each layer, making the training process faster and more stable. This allows the network to learn from a wider range of weight initializations and helps prevent overfitting.
Attention
Attention is a mechanism in machine learning models that allows them to focus on specific aspects of complex inputs, improving the accuracy of results. It’s often used in natural language processing to help models understand context, remember previous information, and produce more accurate translations or responses.
CNN
CNN, or Convolutional Neural Network, is a type of deep learning model excelling at tasks involving grids of data, like images. It extracts features through convolutional layers that slide over the input, identifying patterns and edges. Pooling layers then summarize this information. Fully-connected layers at the end use these features for classification or other tasks. CNNs are particularly effective in computer vision applications like image recognition and object detection.
AE
Autoencoders are unsupervised learners that compress data into a lower-dimensional space and then try to recreate the original data from that compressed version. This process helps them learn efficient representations of the data, useful for dimensionality reduction or anomaly detection.
GAN
Generative Adversarial Networks (GANs) are a type of machine learning system that uses two competing neural networks to create new data. One network, the generator, tries to produce realistic data based on a training set, while the other network, the discriminator, tries to identify if the data is real or generated. This competition pushes both networks to improve, resulting in the generator creating increasingly realistic new data, like images, music, or even text.
RNN
Recurrent Neural Networks (RNNs) are a type of deep learning model that excels at handling sequential data like text or speech. Unlike traditional neural networks, RNNs can remember past inputs thanks to a hidden state. This allows them to analyze sequences and make predictions based on the context, making them ideal for tasks like machine translation, speech recognition, and caption generation.
LSTM
LSTMs, or Long Short-Term Memory networks, are a special kind of Recurrent Neural Network (RNN) designed to overcome a weakness in RNNs. LSTMs have a built-in memory and control mechanisms that allow them to learn from long sequences of data. This makes them particularly good at tasks where understanding long-term context is important, like machine translation, speech recognition, and handwriting analysis.
BiRNN
BiRNN, short for Bidirectional Recurrent Neural Network, is a type of RNN that tackles sequences from both directions. Unlike a standard RNN that only looks at the past, a BiRNN considers both past and future elements in a sequence. This extra context allows BiRNNs to better understand the entire sequence, making them useful for tasks like sentiment analysis, speech recognition, and machine translation where surrounding information is crucial.
ResNet
Resnet, short for Residual Neural Network, is a type of Convolutional Neural Network (CNN) architecture specifically designed to overcome challenges in training very deep networks. Traditional CNNs suffer from vanishing gradients, where information gets lost as it passes through many layers. Resnet tackles this by introducing skip connections. These connections bypass some layers and add the original input directly to the output of those layers. This allows the network to learn more complex transformations while also preserving the ability to learn the identity function (simply outputting the input unchanged). This approach enables Resnets to achieve superior performance on various computer vision tasks compared to traditional CNNs, especially when dealing with very deep architectures.
VGG16
VGG16 is a deep learning model, specifically a Convolutional Neural Network (CNN), known for its accuracy in image recognition and classification. It analyzes images through a series of stacked layers, progressively extracting higher-level features. VGG16 is famous for its depth (16 layers) and is often used as a pre-trained model to jumpstart training on new image recognition tasks.
FaceNet
Facenet is a deep learning system for face recognition. It takes a person’s face image and creates a unique 128-dimensional code, like a fingerprint for their face. This code captures key facial features and distances between them. Faces from the same person will have similar codes, even under variations like lighting or pose. Facenet is a powerful tool for building face recognition applications.
VGGFace
VGGFace refers to a family of pre-trained models specifically designed for face recognition tasks in machine learning. Based on the VGG16 architecture, these models are trained on massive datasets of labeled faces. VGGFace excels at extracting facial features and can be used for various tasks like face detection, recognition, and verification. It’s often used as a starting point for fine-tuning on specific face recognition problems.
YOLO
YOLO, standing for “You Only Look Once,” is a machine learning algorithm for real-time object detection. Unlike some methods, YOLO uses a single neural network to efficiently analyze the entire image at once. It divides the image into a grid and predicts bounding boxes and class probabilities for objects within each grid cell. This makes YOLO super fast for applications like self-driving cars or video surveillance where real-time processing is crucial. However, it can be less accurate than some other object detectors.
CTC
CTC, or Connectionist Temporal Classification, is a technique used in machine learning specifically for sequence recognition tasks, often involving audio or text data. Unlike other methods, CTC doesn’t require perfect alignment between the input sequence (like speech) and the output (like text). It considers all possible alignments and picks the most likely one, making it robust for dealing with variations in speech speed or pronunciation. CTC is commonly used in speech recognition systems.
BPE
Byte Pair Encoding (BPE) is a data compression technique that replaces the most common pair of bytes in a dataset with a single byte not previously used. In natural language processing, BPE is used to split words into subwords to allow the model to handle rare and unseen words better.
Max Inner Product Search
Maximum Inner Product Search (MIPS) is a search algorithm used to find the vectors in a database that have the highest inner product with a query vector. It’s often employed in applications like recommendation systems and information retrieval.
Info Retreival
Information Retrieval (IR) is the process of obtaining relevant information from large repositories, such as databases or the internet, based on user queries. It encompasses a wide range of tasks including search engine development, document retrieval, and data mining. IR systems use algorithms to match user queries with indexed documents, ranking them by relevance using techniques like keyword matching, semantic analysis, and machine learning. Applications of IR extend to various domains, including web search engines, digital libraries, and enterprise search solutions. The field continually evolves, integrating advancements in natural language processing and artificial intelligence to improve the accuracy and efficiency of information retrieval processes.
Baseball
BASEBALL is an early automatic question-answering system developed to respond to queries about baseball games. Described in the paper “BASEBALL: An Automatic Question-Answerer,” this system was one of the pioneering efforts in natural language processing. It used a database of facts about baseball games and employed syntactic parsing to understand and answer user questions. BASEBALL demonstrated the feasibility of automated question answering by accurately retrieving information based on natural language queries, setting the groundwork for future advancements in AI-driven information retrieval and natural language understanding systems.
TREC Conf
The TREC Conference (Text REtrieval Conference) is an annual event organized by the National Institute of Standards and Technology (NIST) and the U.S. Department of Defense. It aims to support research within the information retrieval community by providing the infrastructure necessary for large-scale evaluation of text retrieval methodologies. Since its inception in 1992, TREC has facilitated advancements in search technologies through standardized benchmarking and shared tasks. Participants are given datasets and tasked with developing and evaluating systems to retrieve relevant information. The conference covers various tracks, including web search, question answering, and more specialized areas like legal and biomedical text retrieval. TREC’s collaborative environment has been instrumental in driving progress and innovation in the field of information retrieval.
Bigram/n-gram LM
A Bigram/N-gram Language Model (LM) is a type of statistical language model used in natural language processing that predicts the probability of a word given the previous ‘N-1’ words in a sentence. It’s called a ‘bigram’ when N=2, and ‘N-gram’ for N>2, representing sequences of words or letters to anticipate and better understand context.
Neural Probabilistic LM
Neural Probabilistic Language Model (NPLM) is a type of language model that leverages neural networks to predict the next word in a sequence based on the words that precede it. It uses the context of the sentence (previous words) to form a high-dimensional representation, which it then uses to compute the probability distribution of the next word.
Word Embedding
Word Embedding is a machine learning technique that maps words or phrases from a vocabulary into vectors of real numbers, allowing similar words to have similar numerical representations. This is crucial in natural language processing tasks, as it helps algorithms understand semantic and syntactic similarities between words.