Computer Vision with Transformers: Beyond CNNs

Vision Transformers (ViT) replaced Convolutional Neural Networks (CNNs) in many cutting-edge architectures by treating image patches like words in a sentence.

Why Transformers for Vision?

CNNs use local filters, meaning they have a very localized "receptive field." A pixel in the corner doesn't easily talk to a pixel in the opposite corner until deep in the network. Transformers use Self-Attention, allowing every patch of the image to attend to every other patch immediately in the first layer, establishing deep global context.

1. Patch Imbedding

Instead of feeding raw pixels into a convolution, we split the image into a grid of 16x16 patches. We flatten these patches, project them via a linear layer (creating the embedding), and add Positional Encodings (so the model knows where the patch sits).

import torch
import torch.nn as nn

class PatchEmbedding(nn.Module):
    def __init__(self, in_channels=3, patch_size=16, emb_size=768):
        super().__init__()
        self.patch_size = patch_size
        self.proj = nn.Conv2d(in_channels, emb_size, kernel_size=patch_size, stride=patch_size)

    def forward(self, x):
        # x is (Batch, Channels, Height, Width)
        x = self.proj(x)  # (Batch, EmbSize, H/16, W/16)
        x = x.flatten(2)  # (Batch, EmbSize, NumPatches)
        x = x.transpose(1, 2) # (Batch, NumPatches, EmbSize)
        return x

2. The Transformer Encoder

The patches are fed through standard Transformer encoder blocks containing Multi-Head Self Attention (MHSA) and Multi-Layer Perceptrons (MLPs).

3. The Class Token

A special <CLS> token is prepended to the sequence of patches. This token gathers information from all the other patches via self-attention. At the end of the transformer stack, we pass only the <CLS> token to a standard linear layer to make our final image classification prediction!

CNNs vs ViTs in Production

• Small Datasets: CNNs still win. They have a strong "inductive bias" (they inherently understand edges and textures).
• Large Datasets: ViTs dominate. Given massive amounts of data, the Transformer figures out visual structures globally and scales far better than deep CNNs.