Understanding AutoEncoder(Part 2): Navigating the Tesseract
Alright, let’s pick up where we left off.
We have understood how dimensionality reductions works through PCA. And try to go beyond it by doing dimensionality reduction via AutoEncoder. We have also visualize that latent space, the space where all our data has been projected in reduced dimensions space, and see where it all ends up. We also see what the impacts of tampering/pertubing (rip English) with our input data will have on our latent space, and our output data.
But last time, we also learn one important things. We now that both PCA and AutoEncoder although doing the same thing are actually different in nature. PCA apply linear transformation. While AutoEncoder is not, taking into the fact of that activation function that exists within the AutoEncoder. That is why, when we try to perform PCA on the latent dimensions that has been transformed by the AutoEncoder, we did not get one to one comparison. Latent dimensions that we select based on the eigenvalues, does not correspond to the performance of the model.
So, there must be a ways to fully understand the latent space of autoencoder. How do we pick which dimensions is useuful or not. This is very important for us to optimize our neural network. So, there’s actually two ways to determine which latent dimension is useful, and which is not. So we will explore both of them, today.
Traversing the Latent Space
In understanding the latent space, we need to be in the latent space. Imagine us as Cooper (from the movie interstellar) who is stuck in the Tesseract (that multiple dimensional reality). We’re floating in this space that we encountered for the first time.
But, when we try to make sense of that latent space, there’s multiple strings/bars around us. It represents all the space,time and dimensions that exists around us. We can’t possibly make sense of all of it. So what did Cooper do? He pick up. He focus on one. So that’s what traversing the latent space is. We need to pick up and explore what that dimensions really do.
So, we will be following up on our trained AutoEncoder which are trained on MNIST data. We will just play along with the input data, and the model that we have trained.
Pseudocode: Step by Step of Traversing the Latent Space
Pick up an input image, in this case the handwritten numnber.
Encode it into latent space
Pick one dimensions, z let’s say z[0]
Vary that one dimension value from -3 to 3 for example
Decode back the image
See the results
So let’s implement this in code:
import torch
import torch.nn as nn
import matplotlib.pyplot as plt
import numpy as np
from torchvision import datasets, transforms
# Load model
input_dim = 784
latent_dim = 16 # adjust based on your model
num_classes = 10
model = AutoencoderClassifier(input_dim, latent_dim, num_classes)
model.load_state_dict(torch.load("model/autoencoder_mnist.pth"))
model.eval()
# Load a single MNIST image
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Lambda(lambda x: x.view(-1)) # flatten to 784
])
mnist = datasets.MNIST(root="./data", train=False, transform=transform, download=True)
image, label = mnist[12] # change index to try different digits
image = image.unsqueeze(0)
# Latent traversal - only one dimension
dimension_to_traverse = 7 # pick which latent dimension to explore
with torch.no_grad():
z, _, _ = model(image) # encode
z = z.squeeze(0) # shape: [latent_dim]
steps = 10
values = np.linspace(-3, 3, steps)
fig, axes = plt.subplots(1, steps, figsize=(steps * 1.5, 2))
for j, val in enumerate(values):
z_new = z.clone()
z_new[dimension_to_traverse] = val # modify just one dimension
recon = model.decoder(z_new.unsqueeze(0)).view(28, 28)
axes[j].imshow(recon.numpy(), cmap="gray")
axes[j].set_title(f"{val:.1f}", fontsize=8)
axes[j].axis("off")
plt.suptitle(f"Latent Traversal: Dimension {dimension_to_traverse+1}")
plt.tight_layout()
plt.show()This will create output as below:
Alright, let’s first understand what we have done. When we introduce that variation in the data (-3 to 3), we move the latent dimensions across the axis by (-3,3). This movement will impact the output image that will be decoded since the latent space of our dimensions has been disturbed. From this adjustment, we can see which latent dimensions actually has impact to our results. As you can see above, if we pick dimesions 16 (the last dimensions), we did not see any major changes to our output image. But, if we pick dimensions 1, we can see that out output image has become blurred up to the point it’s not recognizable anymore.
So we can see that each latent dimensions have different impact to the image. Some controls the stroke of the image, some controls the rotation/orientation and some controls the classification/identification of the image. That’s the specific function of each of the dimensions. To understand which dimensions impact which context, let’s try to simulate all of the dimensions at once
import torch
import torch.nn as nn
import matplotlib.pyplot as plt
import numpy as np
from torchvision import datasets, transforms
# Load model structure
input_dim = 784
latent_dim = 16
num_classes = 10
model = AutoencoderClassifier(input_dim, latent_dim, num_classes)
model.load_state_dict(torch.load("model/autoencoder_mnist.pth"))
model.eval()
# Load MNIST sample
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Lambda(lambda x: x.view(-1)) # Flatten
])
mnist = datasets.MNIST(root="./data", train=False, transform=transform, download=True)
image, label = mnist[12]
image = image.unsqueeze(0)
# Latent traversal
with torch.no_grad():
z, _, _ = model(image)
z = z.squeeze(0)
steps = 10
values = np.linspace(-3, 3, steps)
fig, axes = plt.subplots(latent_dim, steps, figsize=(steps, latent_dim))
for i in range(latent_dim):
for j, val in enumerate(values):
z_new = z.clone()
z_new[i] = val
recon = model.decoder(z_new.unsqueeze(0)).view(28, 28).numpy()
axes[i, j].imshow(recon, cmap="gray")
axes[i, j].axis("off")
if j == 0:
axes[i, j].set_ylabel(f"z[{i}]", fontsize=8, rotation=0, labelpad=20, va="center")
plt.subplots_adjust(top=0.93) # leave space for suptitle
plt.suptitle("Latent Traversal for MNIST Autoencoder", fontsize=14)
plt.tight_layout()
plt.show()This will create a large chart as below:
So, based on what we see above, we can classify each dimensions fuctions as below
Dimension 1 — Have no major impact
Dimension 2 — Control over shape refinement or model confidence
Dimension 3 — Have no major impact
Dimension 4 — Identity control, it morph from slanted “5” to “9”
Dimension 5 — Control over shape refinement or model confidence
Dimension 6 — Stroke and rotation control
Dimension 7 — Control over shape refinement or model confidence
Dimension 8 — Identity control, it morph from “2” to “9”
Dimension 9 — Stroke and rotation control
Dimension 10 — Have no major impact
Dimension 11 — Identity control. It morph from “9” to “4”
Dimension 12 — Have no major impact
Dimension 13 — Stroke and rotation control
Dimension 14 — Have no major impact
Dimension 15 — Have no major impact
Dimension 16 — Have no major impact
But this is just based on my analysis. An untrained eye, who done this for the first time. You guys might have different views, so if you guys have different views, you might put it in the comment. For now, that’s what I see. But, I want to bring your attention to another problem.
We know that, based on my haphazard analysis, there are 7 dimensions that have no major impact to our output. So, we technically we can remove all this 7 dimensions right? Before we do that, I think it will be wise if we check with our input data first.
Let’s run the code back, but with different input image this time
import torch
import torch.nn as nn
import matplotlib.pyplot as plt
import numpy as np
from torchvision import datasets, transforms
# Load model structure
input_dim = 784
latent_dim = 16
num_classes = 10
model = AutoencoderClassifier(input_dim, latent_dim, num_classes)
model.load_state_dict(torch.load("model/autoencoder_mnist.pth"))
model.eval()
# Load MNIST sample
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Lambda(lambda x: x.view(-1)) # Flatten
])
mnist = datasets.MNIST(root="./data", train=False, transform=transform, download=True)
image, label = mnist[19]
image = image.unsqueeze(0)
# Latent traversal
with torch.no_grad():
z, _, _ = model(image)
z = z.squeeze(0)
steps = 10
values = np.linspace(-3, 3, steps)
fig, axes = plt.subplots(latent_dim, steps, figsize=(steps, latent_dim))
for i in range(latent_dim):
for j, val in enumerate(values):
z_new = z.clone()
z_new[i] = val
recon = model.decoder(z_new.unsqueeze(0)).view(28, 28).numpy()
axes[i, j].imshow(recon, cmap="gray")
axes[i, j].axis("off")
if j == 0:
axes[i, j].set_ylabel(f"z[{i}]", fontsize=8, rotation=0, labelpad=20, va="center")
plt.subplots_adjust(top=0.93) # leave space for suptitle
plt.suptitle("Latent Traversal for MNIST Autoencoder", fontsize=14)
plt.tight_layout()
plt.show()
Getting a headache yet? Yeah, that’s what I feel when I first performed the experiment. We see that the dimensions now behaving differently for different data input. So, we can’t really remove the dimensions, because each dimensions treat each image input differently. Why is this happening?
This actually happen because of the dimensions entanglement. Now we’re getting into the realm of quantum (at least in the mindset framing, not actual quantum). So that dimensions entanglement and disentanglement is something that we will discuss next.
Understanding the Entangled Dimensions
Dimensions that have mixed property (sometime adjust stroke, sometimes do nothin) is what we called an entangled dimensions. At one sense, it’s good that we have an entangled dimensions that seems multipurpose capability. But, for generality, an entangled dimensions cannot be depended to explore new data. This will cause future issues if we want to expand our trained model outside of their training realm.
What we want actually is a disentangled dimensions. A dimensions that is consistently fixed for one purpose. We’re going back to that Cooper methapore. We know that in the movie, Cooper started playing with the strings in the Tesseract, sending message to her daughter in the past. He tried the strings/dimensions one by one and figured out which strings corresponds to which. But this can only happens because the strings has consistent function. What if the strings behave differently for different input? That will be a major headache for Cooper, and he will definitely stuck there indefinitely. And that’s a rather grimm ending.
That is the same with autoencoders. As it is important to have a fully disentangled dimensions in its latent space. So the model will be fully reliable in the future.
That’s why, in 2018 there’s a paper released by Eastwood & Williams (a winning paper actually) in 2018, titled “A FRAMEWORK FOR THE QUANTITATIVE EVALUATION OF DISENTANGLED REPRESENTATIONS”.
The full paper can be read here:
https://openreview.net/pdf?id=By-7dz-AZ
They proposed a quantitate evaluation method called DCI Metrics. So, that’s what we will do next
The DCI Metrics
DCI metrics stands for
D — Disentanglement (does each latent variable affect only one factor?)
C — Completeness (is each factor represented by only one (or few) variables?)
I — Informativeness (can latent variables predict the factors at all?)
So, in doing this, what we need to do is take the latent space dimensions/variables and use it to train a classical machine learning classifier. Each space will be coupled with the input label data and we will used this to determine how well the latent space is represented. So, let’s do that.
#first, let's take out the latent space vectors and it;s label and save it as a training data
import torch
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import numpy as np
# Set params
batch_size = 128
input_dim = 784
# Load model
model = AutoencoderClassifier(input_dim=input_dim, latent_dim=16, num_classes=10)
model.load_state_dict(torch.load("model/autoencoder_mnist.pth"))
model.eval()
# Load test data
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Lambda(lambda x: x.view(-1)) # flatten to 784
])
test_set = datasets.MNIST(root='./data', train=False, download=True, transform=transform)
test_loader = DataLoader(test_set, batch_size=batch_size, shuffle=False)
# Collect latent representations and labels
X_latent = []
y_labels = []
with torch.no_grad():
for images, labels in test_loader:
z, _, _ = model(images)
X_latent.append(z.cpu())
y_labels.append(labels.cpu())
X_latent = torch.cat(X_latent).numpy()
y_labels = torch.cat(y_labels).numpy()#next, let's use the gradient boosting classfiier to train our latent dimensions and labels
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import accuracy_score
clf = GradientBoostingClassifier()
clf.fit(X_latent, y_labels)
y_pred = clf.predict(X_latent)
accuracy = accuracy_score(y_labels, y_pred)
feature_importance = clf.feature_importances_ # shape: [latent_dim]Alright, whart we did above was taken our latent dimensions and the input labels and turn it into x and y data. This data will then be fed to train a Gradient Boosting Classifier machine learning algorithm. Why gradient boosting?
Well, for one, classical machine learning algorithms still keep each feature/dimensions as separate. They do not entangle the dimensions as what being done here in deep learning. Secondly, gradient boosting is non-linear, so it can capture the non-linearity in the latent dimensions.
Alright, what we will do next is use that feature importance and accuracy to map it out inside our DCI metrices. The DCI metrices is quantitatively defined by:
Informativeness
2. Disentanglement
3. Completeness
The formula above is for generalized case. We only want to use this for 1 single classification, so our implementation will be simplified as below
I — just the accuracy
D — the square of p. Where p is actually the normalization of importances with error correction. (epsilon, 1e-8)
C — just select the maximum p.
So let’s implement this as below
def compute_dci_metrics(importances):
eps = 1e-8
p = importances / (np.sum(importances) + eps)
disentanglement = np.sum(p ** 2)
completeness = np.max(p)
return disentanglement, completeness#then with that information, we can map out our DCI metrics
dis, comp = compute_dci_metrics(feature_importance)
print(f" DCI Metrics")
print(f" - Informativeness (accuracy): {accuracy:.4f}")
print(f" - Disentanglement: {dis:.4f}")
print(f" - Completeness: {comp:.4f}")This will output our result as below:
Ah, as expected. It seems like although our accuracy is high, our disentanglement and completeness are very-very low. This shows that our dimensions does not disentangle with each other. And a disentangled latent space is bad for generalized task.
So, let’s ask a big question. Why does this happen actually?
This happens because autoencoder is not objectively designed to get a disetangled dimensions. What it wants actually is to reconstruct the input with as low error as possible. So, it doesn’t have the motivation to get a fully disentangled latent space. The issue lies with how the architecture is designed actually. There’s no other ways to improve the disentanglement of our latent space without adjusting our architecture. And that is what we will explore later on. See you next time.