Understanding AutoEncoder Part 1
:That Dimensionality Reduction
AutoEncoder are a neural network architecture that’s basically has two main components, which is encoder and decoder.The functions of encoder is to encode the data into a compressed representation that can be learned by the decoder so that the decoder can reconstruct the input. That’s basically the oversimplication of autoencoder.
But in truth, there’s actually more depth to it. So today, we would like to dive deep, I mean really, really deep on autoencoder. We will like to explore on that “compressed representation” and what it does really mean. We will explore the concepts, build a fully functional autoencoder for classification, and maybe perform some experiments to really understand on that “compressed representation”. So let’s begin.
Compressed Representation is Dimensionality Reduction
When talking about compressed representation, what the encoder really do is, taking the high dimensionality data, and reduce it so it becomes lower-dimensional data can be learned by decoder. Alright, two questions may arise from this.
What dimensions are we talking about, aren’t all data comes in two dimensions? (paper and computer screen)
Why the need to reduce the data dimensions? Why can’t Decoder learn straight from the data?
So, we will try to answer this questions one by one starting with dimensions
1. Dimensions in Data That dimensions actually refers to the variables that exists in the data. For examples, imagine that we have a data containing the registration forms of a student in schools. We have the name of the students, but at the same time we also have the age, gender, date of birth, address, height, weight, social status, grades and many more. We can assume that the name of the students will be the identified and the grades will be the dependent variable (that we want to solve) , but the rest will be the independent variables. All this variables represents the dimensions of the data.
For simpler understanding, we can assume that give a data with a set of features (say 10), that features will be the dimensions.
2. Why need to reduce dimensions? So, why do we need to reduce dimensions? The easiest answer to increase efficiency of computation. Apart from that, by performing dimensionality reductiom, we can ensure that our model select only the important features to be trained effectively. Manually, in machine learning, we did this via Feature Engineering. We perform SHAP, Correlaton analysis. But, there’s a far more efficient techniques which is the dimensionality reductions, which improved the automation and generalized better compated to feature engineering.
So, that’s what we will start with, with one of the basic techniques of dimensionality reduction, the Principal Component Analysis or PCA in short.
Principal Component Analysis
Principal Component Analysis is a mechine learning techniques that is fundamental in dimensionality reductions. It works on the principles of high variance in data. Consider a sample population in a city, say Kuala Lumpur, Malaysia. We have multiple dimensions in this data such as age, gender, religion, occupation, salary etc. Say, for example if we want to use this data to predict which one will buy a house in the city, we will need to select which features is the most important. We cannot use data such as gender to determine the likelihood of a person to buy a house in the city. Because a gender will only have two, which is male and female. That is low variance data. Meaning that the data is not that very far aways from the mean.
We will select features like age, occupation and salary to find out which of the population will buy the house in the city. These features generally have more variance. They are more widely disributed across the population. This variance is actually the key uses by PCA to determine which features are important, and which are not. How they might do this? So, let’s do some demonstration to find out.
First, let us generate the synthetic dataset:
#first, let us generate a synthetic dataset for our demonstration, we will use that hypothetical population dataset that we mentioned previously
import numpy as np
import pandas as pd
# Step 1: Generate a Synthetic Population Dataset
np.random.seed(42)
n_samples = 500 # Number of people in the dataset
# Simulated features
age = np.random.randint(18, 65, n_samples) # Age between 18 and 65
gender = np.random.choice([0, 1], n_samples) # 0 = Male, 1 = Female
religion = np.random.choice([0, 1, 2], n_samples) # 0 = Islam, 1 = Buddhism, 2 = Christianity
occupation = np.random.randint(1, 10, n_samples) # Encoded occupation categories
salary = np.random.normal(5000, 1500, n_samples) # Salary with mean 5000, std 1500
housing_status = np.random.choice([0, 1], n_samples) # 0 = Renting, 1 = Own house
#create that dataframe
data = pd.DataFrame({
'Age': age,
'Gender': gender,
'Religion': religion,
'Occupation': occupation,
'Salary': salary,
'Housing_Status': housing_status
})
#and display them (10 samples out of 500)
data.head(10)Our data will be displayed as below:
Alright, for now can we just ignore the logic of the data? I know it’s not logic looking at the numbers, but hey…it’s syntehtic.
Before we proceed, we must understand how PCA is done first. Below are the basic and simple steps of PCA
First, we must standardize the data ensuring standardized scale
Calculare the covariance matrix to capture how features vary with each other
Get the eigenvalues and the eigenvectors
Project the data based on the eigenvalues and eigenvectors
That’s a lot of mathematical terms, so we will look at them later.
For now, let’s just standardized our data first.
1. Data Standardization
PCA is very sensitive to unstardardized data. What this is means is that we need to change the data scale from it’s current min and max to scale of standard deviation of our data.. So,let’s do that first. Standardizing is done by substracting the mean of the data and divide that means with standard deviation of the data. So let’s do that.
# Convert the DataFrame to a NumPy array
X = data.values # shape: (500, 6)
# Separate features from the last column if you like
# But for demonstration, let's keep all 6 columns.
# (We can also decide to exclude 'Housing_Status' if we consider it a label.)
# Calculate mean and std for each feature (column)
mean_vec = np.mean(X, axis=0)
std_vec = np.std(X, axis=0, ddof=1) # ddof=1 for sample std
# Standardize
X_std = (X - mean_vec) / std_vec
print("Shape of standardized data:", X_std.shape)
print("Means (approx. 0):", np.mean(X_std, axis=0))
print("Stds (approx. 1):", np.std(X_std, axis=0, ddof=1))This will show our data as below, with slightly smaller numbers (since it has been scaled)
Now, we have to calculate the covariance matrix for our features.
2. Calculate Covariance Matrix
What covariance matrix is? It measures how variables change together. Since we have six features, we need to calculate how each features change with each other. This covariance is done in pairs, as per below mathematical formula
For a dataset with d features, we can compute the covariance for every pair of features and arrange these values in a dxd matrix:
Diagonal entries (e.g., Cov(X1,X2)) are variances of each feature.
So, let’s do that, let’s calculate the covariance matrix for all features with respect to each others.
n_features = X_std.shape[1]
n = X_std.shape[0]
# Covariance matrix: shape (3, 3)
cov_matrix = (X_std.T @ X_std) / (n - 1)
print("\nCovariance Matrix:\n", cov_matrix)This will printed our covariance matrix as below:
Ok, once we have calculated the covariance matrix for each features with respect to each other, we will use that covariance matrix to find the principal components through a method called eigen decomposition. What is that? Let’s dive in details
3. Finding Principal Components through Eigen-Decomposition
Eigen-Decomposition is the process breaking down the square matrix (in this case, the covariance matrix that we have calculated above) into a set of eigenvectors and eigenvalues. What is eigenvectors and eigenvalues? It is defined as below:
Eigenvectors is the characteristics direction of the transformation (transformation means matrix operations i.e multiplication)
-Eigenvalues is the characteristics scaling factors that tell us how much of those eigenvectors are expanded or shrunk
The eigenvalues will be sorted in descending order. This will determine which principal components will have the most to the least variance. From there, our eigenvectors will follow these order to make it easier for selection of the principal components. We can demonstrate this as below:
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)
idx = np.argsort(eigenvalues)[::-1] #sort the eigenvalues index
eigenvalues = eigenvalues[idx] #sort the eigenvalues itself
eigenvectors = eigenvectors[:, idx]
print("\nEigenvalues (descending):\n", eigenvalues)
print("\nEigenvectors (columns = principal components):\n", eigenvectors)
Once we determine our Principal components, we will select the top values/top k (based on the eigenvalues that we have sorted) and project all our data points on these Principal Components. If we see above, we sort of can guess which top we want to select. Since 3 of our eigenvalues have identical values (>1), we will select top 3.
By doing this, we can see find our principal components that has the most variance. At the same, we can also see how much variance ratio does our top 3 Principal components brings. Just run the code as below:
k = 3
top_k_eigenvectors = eigenvectors[:, :k]
X_pca = X_std @ top_k_eigenvectors
print("\nPCA-transformed data shape:", X_pca.shape)
print("First 5 rows of PCA-transformed data:\n", X_pca[:5])
# (Optional) Explained Variance
explained_variance_ratio = eigenvalues / np.sum(eigenvalues)
print("\nExplained Variance Ratio (all components):\n", explained_variance_ratio)
print("Cumulative explained variance (first 2 PCs):", np.cumsum(explained_variance_ratio[:3]))
Now that’s basically PCA. Our data has been transformed from 6 features to only 3. And it brought more than 54% of variance into our data. Which is not that good, since our data is synthetically generated, so I assume that might be the reason it’s variance is distibuted almost fairly. Now, let’s move on to autoencoder.
Let’s Build an AutoEncoder
We have understood how dimensionality reductions happen with the basics simplest form of dimensionality reduction, the PCA (Principal Components Analysis). Now, we would like to explore how this dimensionality reduction is done in an autoencoder, specifically the encoder part.
The answer to that question is pretty simple actually, what we did was we employ the weights as the transformation agent to select the best features among the dataset. What makes this better than PCA is because we use neural network as learnable parameters, so it can improve the accuracy of features selected. At the same time, it also has activation function which added non-linearity to the dimenionality redution. PCA can only apply linear dimensionality reduction,
So, let’s build an autoencoder with the purpose of classifying the handwritten digit from 0 to 9. Which is a bit of overkill, since we can do oit with simple Neural Network. But our goal is to learn about that latent space, the dimensionality reduction via autoencoder, so, that’s what we will do.
#first let's impoet all the necessary libraries
import torch
import torch.nn as nn
import torch.optim as optim
import matplotlib.pyplot as plt
from torchvision import datasets, transforms
from torch.utils.data import DataLoaderWe will now build the autoencoder class for MNIST Handwritten Digit recognition. Basically a classification tasks. We will ensure functions to return the output of encoder and decoder (before the softmax classification layer) apart from it’s classification. With this, we can call the encoder and decoder output for further analysis.
#now let's build the autoencoder for classification. we will return
#3 output from this class which is the
#encoder output, decoder output and the label classification
class AutoencoderClassifier(nn.Module):
def __init__(self, input_dim, latent_dim, num_classes):
super(AutoencoderClassifier, self).__init__()
# Encoder: Compress data into latent space
self.encoder = nn.Sequential(
nn.Linear(input_dim, 128),
nn.ReLU(),
nn.Linear(128, latent_dim) # Latent representation
)
# Decoder: Reconstruct data from latent space
self.decoder = nn.Sequential(
nn.Linear(latent_dim, 128),
nn.ReLU(),
nn.Linear(128, input_dim),
nn.Sigmoid() # Output is between 0 and 1
)
# Classifier: Predict digit labels from latent space
self.classifier = nn.Sequential(
nn.Linear(latent_dim, 64),
nn.ReLU(),
nn.Linear(64, num_classes), # Output layer (10 classes for digits 0-9)
nn.LogSoftmax(dim=1) # Log-Softmax for classification
)
def forward(self, x):
encoded = self.encoder(x) # Get latent representation
decoded = self.decoder(encoded) # Reconstruct input
classified = self.classifier(encoded) # Predict digit class
return encoded, decoded, classifiedLet’s download the MNIST datasets.
# Transform: Convert to tensor & flatten 28x28 images to 784-dimensional vectors
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Lambda(lambda x: x.view(-1)) # Flatten the image
])
# Load MNIST dataset
train_data = datasets.MNIST(root="./data", train=True, transform=transform, download=True)
test_data = datasets.MNIST(root="./data", train=False, transform=transform, download=True)
# Data loaders
train_loader = DataLoader(train_data, batch_size=64, shuffle=True)
test_loader = DataLoader(test_data, batch_size=64, shuffle=False)Let’s train our autoencoder and test it. We will train it on 10 epochs with learning rate of 0.001 using Adam as optimizer
# Model Parameters
input_dim = 28 * 28 # MNIST images are 28x28 pixels
latent_dim = 16 # Latent space size
num_classes = 10 # 10 digits (0-9)
# Initialize the model, loss functions, and optimizer
model = AutoencoderClassifier(input_dim, latent_dim, num_classes)
criterion_reconstruction = nn.MSELoss() # Reconstruction loss
criterion_classification = nn.NLLLoss() # Classification loss (Negative Log-Likelihood)
optimizer = optim.Adam(model.parameters(), lr=0.001)
# Training loop
num_epochs = 10
for epoch in range(num_epochs):
for images, labels in train_loader:
optimizer.zero_grad()
# Forward pass
encoded, decoded, classified = model(images)
# Compute losses
loss_reconstruction = criterion_reconstruction(decoded, images) # Autoencoder loss
loss_classification = criterion_classification(classified, labels) # Classification loss
# Total loss = reconstruction loss + classification loss
loss = loss_reconstruction + loss_classification
# Backpropagation
loss.backward()
optimizer.step()
print(f"Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}, Recon Loss: {loss_reconstruction.item():.4f}, Class Loss: {loss_classification.item():.4f}")
# Save the trained autoencoder model
torch.save(model.state_dict(), "model/autoencoder_mnist.pth")
print("Model saved successfully!")
Time to test it’s accuracy
# Function to compute accuracy
def compute_accuracy(model, data_loader):
correct = 0
total = 0
with torch.no_grad():
for images, labels in data_loader:
_, _, classified = model(images)
predictions = torch.argmax(classified, dim=1) # Get predicted labels
correct += (predictions == labels).sum().item()
total += labels.size(0)
return 100 * correct / total
# Evaluate on test data
accuracy = compute_accuracy(model, test_loader)
print(f"Test Accuracy: {accuracy:.2f}%")
97% accuracy! Wow. Yeah, but it’s as expected. We’re using a better model to perform simple classification tasks. So that’s why it works as good as expected. But our goal is not to train an autoencoder. Our goal is to understand how autoencoder works, and to do that, we need to perform some deep diving into the latent space.
Understanding the AutoEncoder Latent Space
We need to understand how the decoder part was able to correctly label all these data with just limited representation in the latent space. By understanding, we have a good basic grasp of understanding how most AI model now works. There are few ways for us to understand this latent space. Let’s list it down below:
Visualize the latent space
Sensitivity and Robustness Analysis
Eigenvalues Analysis
Visualize the Latent Space
First, let us visualize all the data in our latent space. But since we can’t viusalize more than 2 dimensions (our latent space dimensions is 16), let us flatten that out into a 2D plot. At the same time, we will cluseter all these data based on the MNIST digits labels. So, let’s do that.
# extract the latent space representations
all_latents = []
all_labels = []
with torch.no_grad():
for images, labels in test_loader:
encoded, _, _ = model(images)
all_latents.append(encoded.numpy())
all_labels.append(labels.numpy())
# Convert to NumPy
all_latents = np.vstack(all_latents)
all_labels = np.hstack(all_labels)
# Scatter plot of the latent space
plt.figure(figsize=(8, 6))
plt.scatter(all_latents[:, 0], all_latents[:, 1], c=all_labels, cmap="tab10", alpha=0.7)
plt.colorbar(label="Digit Label")
plt.xlabel("Latent Dimension 1")
plt.ylabel("Latent Dimension 2")
plt.title("Latent Space Representation of MNIST Digits")
plt.show()
There’s a lot to unpack based on the image above. First, there’s a lot of overlap between data points for several digit labels. This includes no.7, no 5 (which makes sense), no 8 and no 3 (which does closely resembles each other). This overlap shows that the model put’s these numbers close with each other because of their likeness. Something that we will test later when we perturbed (made changes) to the input image. Other numbers like 6 and 0 seems loosely interconnected and being recognized fairly well, so that’s something that we need to understand also. But this just shows where the data points is for each label across our latent space. To understand how these positions influenced by the input and how it influences the output, let us proceed with next analysis.
Sensitivity and Robustness Analysis
In the simplest term of understanding, the sensitivity and robustness analysis is done by applying perturbation (changes) to the input of our model and see how it impacts the output and the latent space of our model. In doing this, we can take our trained model, apply some gaussian noise to it, and call our model output. Since we already include our model to return the decoder output, and the encoder output(latent space), we can use that to visualize what each blocks of autoencoder sees, and it will help us understand what’s really happening when we change the input. At the same time, we will also calculate the latent difference. Meaning the position of of our data in the original image minus the position of our data in the noisy image.
Let’s do that below and see what will be the output
# Model Parameters
input_dim = 28 * 28 # MNIST images are 28x28 pixels
latent_dim = 16 # Latent space size
num_classes = 10 # 10 digits (0-9)
# Initialize the model, loss functions, and optimizer
model = AutoencoderClassifier(input_dim, latent_dim, num_classes)
model.load_state_dict(torch.load('model/autoencoder_mnist.pth'))
model.eval()
# Load MNIST test data
data_transform = transforms.Compose([transforms.ToTensor()])
test_dataset = datasets.MNIST(root="./data", train=False, transform=data_transform, download=True)
test_loader = DataLoader(test_dataset, batch_size=1, shuffle=True)
# Select one image for perturbation
image, label = next(iter(test_loader))
original_image = image.clone()
# Apply Gaussian noise perturbation
noise_std = 0.01 #we set the noise here
noisy_image = image + torch.randn_like(image) * noise_std
noisy_image = torch.clamp(noisy_image, 0, 1)
# Pass both images through the autoencoder
with torch.no_grad():
latent_orig, recon_orig, _ = model(original_image.view(1, -1)) # Extract only latent & reconstruction
latent_noisy, recon_noisy, _ = model(noisy_image.view(1, -1))
# Compute latent difference
latent_diff = torch.norm(latent_orig - latent_noisy, dim=1).item()
# Visualize results
fig, axes = plt.subplots(2, 3, figsize=(9, 6))
axes[0, 0].imshow(original_image.squeeze().numpy(), cmap='gray')
axes[0, 0].set_title("Original Image")
axes[0, 1].imshow(noisy_image.squeeze().numpy(), cmap='gray')
axes[0, 1].set_title("Noisy Image")
axes[0, 2].imshow(recon_orig.view(28, 28).numpy(), cmap='gray')
axes[0, 2].set_title("Reconstructed (Original)")
axes[1, 0].axis('off')
axes[1, 1].text(0.5, 0.5, f"Latent Difference: {latent_diff:.4f}", fontsize=12, ha='center')
axes[1, 1].axis('off')
axes[1, 2].imshow(recon_noisy.view(28, 28).numpy(), cmap='gray')
axes[1, 2].set_title("Reconstructed (Noisy)")
plt.tight_layout()
plt.show()This will show the plot as below where it display the both the original and noisy image and the reconstructed from both types of input.
In the example above, we see that when we introduce just a little bit of noise (0.01), there’s not much difference between the original image and the reconstructed image.However, when we start to increase the noise to 2, we can see that the reconstruction image are not identifiable anymore.
Notice also th latent difference becomes bigger showing that our data points in the latent space has been shifted.
Let us visualize this perturbation inside the latent space. Let’s ammend our latent space visualization code as below
import numpy as np
import torch
import torch.nn as nn
import matplotlib.pyplot as plt
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
from sklearn.decomposition import PCA
# Load trained autoencoder
model = AutoencoderClassifier(input_dim=28*28, latent_dim=16, num_classes=10)
model.load_state_dict(torch.load('model/autoencoder_mnist.pth'))
model.eval()
# Load MNIST test data
test_dataset = datasets.MNIST(root="./data", train=False, transform=transforms.ToTensor(), download=True)
test_loader = DataLoader(test_dataset, batch_size=256, shuffle=False)
# Store latent representations
latent_original = []
latent_perturbed = []
labels = []
# Define noise level
noise_std = 0.01 #adjust the noise level here
with torch.no_grad():
for images, label in test_loader:
images = images.view(images.size(0), -1) # Flatten images
# Get latent space for original images
latent_orig, _, _ = model(images)
latent_original.append(latent_orig.numpy())
# Apply perturbation (Gaussian noise)
noisy_images = images + torch.randn_like(images) * noise_std
noisy_images = torch.clamp(noisy_images, 0, 1) # Ensure valid pixel range
# Get latent space for perturbed images
latent_noisy, _, _ = model(noisy_images)
latent_perturbed.append(latent_noisy.numpy())
labels.append(label.numpy())
# Convert lists to numpy arrays
latent_original = np.vstack(latent_original)
latent_perturbed = np.vstack(latent_perturbed)
labels = np.hstack(labels)
# Apply PCA to reduce from 16D → 2D
pca = PCA(n_components=2)
latent_orig_2d = pca.fit_transform(latent_original)
latent_perturbed_2d = pca.transform(latent_perturbed) # Use same PCA transform
# Plot latent space before and after perturbation
plt.figure(figsize=(8,6))
# Original points (before perturbation)
plt.scatter(latent_orig_2d[:, 0], latent_orig_2d[:, 1], c=labels, cmap="tab10", alpha=0.3, label="Original")
# Perturbed points (after noise)
plt.scatter(latent_perturbed_2d[:, 0], latent_perturbed_2d[:, 1], c=labels, cmap="tab10", alpha=0.6, marker="x", label="Perturbed")
plt.colorbar(label="Digit Label")
plt.xlabel("Latent Dimension 1")
plt.ylabel("Latent Dimension 2")
plt.title("Latent Space Shift Before & After Perturbation")
plt.legend()
plt.show()
As you can see from the graph above, when we introduce just a little bit of noise, the data position in the space moves, but still, the movemement is negligible and we can still see where all data points ends in the visible cluster. But when we increase the noise to above 2, as below
all the data points move and spread around without any visible cluster can bee seen.
But, what does this means actually? What can we do with this this information. Well, if our latent difference shoot up so much when we introduce slight noise into it, that means our model does not generalize well. So it might be best if we improve our trainings, or maybe improve our architecture in overall. One way to improve it, is by further refining our latent space dimensions. By refining it, we can ensure that we select only the best features that really contributes to our output. And how might we do that? Let’s go to the next analysis
Eigenvalues Analysis
Remember that eigenvalues and eigenvectors that we do initially during PCA? We can use that eigenvalues to determine at what numbers of dimensions in latent space should our model being trained on? So, let’s do that. Let’s take our trained model, perform PCA on it, and map which PC has the highest variance to the lowest. Let’s implement the code below
import numpy as np
import torch
import matplotlib.pyplot as plt
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
from sklearn.decomposition import PCA
# Load trained autoencoder
model = AutoencoderClassifier(input_dim=28*28, latent_dim=16, num_classes=10)
model.load_state_dict(torch.load('model/autoencoder_mnist.pth'))
model.eval()
# Load MNIST test data
test_dataset = datasets.MNIST(root="./data", train=False, transform=transforms.ToTensor(), download=True)
test_loader = DataLoader(test_dataset, batch_size=256, shuffle=False)
# Collect latent representations
latent_vectors = []
with torch.no_grad():
for images, _ in test_loader:
images = images.view(images.size(0), -1) # Flatten images
latent, _, _ = model(images) # Extract latent representation
latent_vectors.append(latent.numpy())
# Convert to numpy array
latent_vectors = np.vstack(latent_vectors)
# Compute Covariance Matrix
cov_matrix = np.cov(latent_vectors, rowvar=False)
# Eigen Decomposition
eigenvalues, eigenvectors = np.linalg.eigh(cov_matrix)
# Sort eigenvalues in descending order
eigenvalues = np.flip(eigenvalues)
eigenvectors = np.flip(eigenvectors, axis=1)
# Plot eigenvalues
plt.figure(figsize=(7,5))
plt.plot(range(1, len(eigenvalues) + 1), eigenvalues, marker='o', linestyle='--')
plt.xlabel("Principal Component Index")
plt.ylabel("Eigenvalue (Variance Explained)")
plt.title("Eigenvalues of Latent Space")
plt.grid()
plt.show()
It seems like, our varriance fall below 10, when we go above PC 6. Hm, this MIGHT shows that our model can works with just 6 dimensional features! This is huge actually, since reducing our latent space dimensions, we can reduce our model parameter. So, let’s test that. Let’s see whether when we train our model with only 6 latent dimensions, we can preserve our model accuracy, and get a model that generalize better?
Let’s train a new model with only 6 latent dimensions. And save it with new model name.
# Model Parameters
input_dim = 28 * 28 # MNIST images are 28x28 pixels
latent_dim = 6 # Latent space size
num_classes = 10 # 10 digits (0-9)
# Initialize the model, loss functions, and optimizer
model8 = AutoencoderClassifier(input_dim, latent_dim, num_classes)
criterion_reconstruction = nn.MSELoss() # Reconstruction loss
criterion_classification = nn.NLLLoss() # Classification loss (Negative Log-Likelihood)
optimizer = optim.Adam(model8.parameters(), lr=0.001)
# Training loop
num_epochs = 10
for epoch in range(num_epochs):
for images, labels in train_loader:
optimizer.zero_grad()
# Forward pass
encoded, decoded, classified = model(images)
# Compute losses
loss_reconstruction = criterion_reconstruction(decoded, images) # Autoencoder loss
loss_classification = criterion_classification(classified, labels) # Classification loss
# Total loss = reconstruction loss + classification loss
loss = loss_reconstruction + loss_classification
# Backpropagation
loss.backward()
optimizer.step()
print(f"Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}, Recon Loss: {loss_reconstruction.item():.4f}, Class Loss: {loss_classification.item():.4f}")
# Save the trained autoencoder model
torch.save(model8.state_dict(), "model/autoencoder_mnist_2.pth")
print("Model saved successfully!")
Alright, it seems to work well. So let’s see whether it’s accuracy still stands. Run the code below
# Function to compute accuracy
def compute_accuracy(model, data_loader):
correct = 0
total = 0
with torch.no_grad():
for images, labels in data_loader:
# Reshape images to (batch_size, 784)
batch_size = images.size(0)
images = images.view(batch_size, -1) # Flatten the 28x28 images
# Forward pass
_, _, classified = model(images)
predictions = torch.argmax(classified, dim=1)
correct += (predictions == labels).sum().item()
total += labels.size(0)
return 100 * correct / total
# Evaluate on test data
accuracy = compute_accuracy(model8, test_loader)
print(f"Test Accuracy: {accuracy:.2f}%")
Owh no, the accuracy fell down. Why is that? To understand it, let’s perform that Eigenvalues analysis again.
import numpy as np
import torch
import matplotlib.pyplot as plt
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
from sklearn.decomposition import PCA
# Load trained autoencoder
model8 = AutoencoderClassifier(input_dim=28*28, latent_dim=6, num_classes=10)
model8.load_state_dict(torch.load('model/autoencoder_mnist_2.pth'))
model8.eval()
# Load MNIST test data
test_dataset = datasets.MNIST(root="./data", train=False, transform=transforms.ToTensor(), download=True)
test_loader = DataLoader(test_dataset, batch_size=256, shuffle=False)
# Collect latent representations
latent_vectors = []
with torch.no_grad():
for images, _ in test_loader:
images = images.view(images.size(0), -1) # Flatten images
latent, _, _ = model8(images) # Extract latent representation
latent_vectors.append(latent.numpy())
# Convert to numpy array
latent_vectors = np.vstack(latent_vectors)
# Compute Covariance Matrix
cov_matrix = np.cov(latent_vectors, rowvar=False)
# Eigen Decomposition
eigenvalues, eigenvectors = np.linalg.eigh(cov_matrix)
# Sort eigenvalues in descending order
eigenvalues = np.flip(eigenvalues)
eigenvectors = np.flip(eigenvectors, axis=1)
# Plot eigenvalues
plt.figure(figsize=(7,5))
plt.plot(range(1, len(eigenvalues) + 1), eigenvalues, marker='o', linestyle='--')
plt.xlabel("Principal Component Index")
plt.ylabel("Eigenvalue (Variance Explained)")
plt.title("Eigenvalues of Latent Space")
plt.grid()
plt.show()
This happens because PCA is a linear dimensionality reduction. Whereas autoencoder is not. Autoencoder has become a non-linear dimensionality reduction due to the presence of activation of function inside the network. That’s why we cannot assume a linear relationship to understand the latent space inside an autoencoder. So how might we do this? What method we can use to better understand the latent space of autoencoder? That is something we will explore in the next part.