The dual-encoder architecture used by CLIP is composed of a text encoder and an image encoder. Here is how it works:
Collection of data: The model learns from the data, which is a wide dataset with millions of images and, most importantly, textual descriptions.
Text Encoder: The text encoder encodes the textual descriptions in high-dimensional vectors.
Image Encoder: The image encoder does a similar job but with images, turning them into high-dimensional vectors.
This is through a contrastive loss function used while training, to make the vector spaces of matched texts and images close and the ones of unmatched pairs far apart. After being pre-trained, the model can execute all tasks with no tasks or specific training inscribed in the models’ generalization ability on pre-trained datasets.
This generalization entails contrasting a given piece of extracted text from an image with another image.
