Introduction

Would it be wonderful if we create any images that come to our mind without any effort? Like getting whatever images we want and even whatever we wish. Yes, it's possible now. What if I tell you some interesting Artificial Intelligence model can do it for you when you need images of your needs. The AI model is called Image GPT (Generative Pre-training from Pixels) is one form of GPT (Generative Pre-trained Transformer). We will discuss briefly about Image GPT in detail.

GPT-4 

As many have come to know about the importance of GPT-4 in recent times. In short, GPT-4 (Generative Pre-trained Transformer 4) is a multimodal large language model created by OpenAI and the fourth in its GPT series. It is a type of deep learning model used to generate human-like text. It’s common uses include           

1. Translating text to other languages

2. Generating code

3. Generating blog post, stories, conversations and other content types

4. Summarising text

5. Answering questions

GPT Transformer Architecture

There are other GPT models such as GPT-2, GPT-3, GPT3.5. But, GPT-4 is advanced (making factual or reasoning errors, 40% higher than GPT 3.5) and GPT-4 is significantly larger and more powerful than GPT-3, with 170 trillion parameters compared to GPT-3’s 175 billion parameters. 

What is Image GPT?

Image GPT, also called iGPT (Generative Pre-training from Pixels) developed by OpenAI. It is a machine learning model that combines the capabilities of GPT language models with computer vision to generate realistic images. Similar to GPT, iGPT is trained on large datasets of images using unsupervised learning methods, allowing it to learn patterns and relationships in visual data. It then uses this knowledge to generate new images, often with impressive results.GPT is shown to be effective in tasks such as image completion, image classification and image generation. 

     Model-generated Completions of human-provided half-images    

The model is based on the Transformer architecture, which was originally developed for natural language processing (NLP) tasks. It was originally introduced in a research paper titled "Generative Pre-training from Pixels” by OpenAI in 2021.  

Facts about iGPT

We can see some of the interesting facts about iGPT (Image GPT),

1. iGPT has achieved state-of-the-art results in several computer vision tasks, such as image classification, image completion and image generation. 

2. The pre-training process for iGPT is computationally expensive, as it requires massive amounts of data and computational resources. The largest iGPT model has over 1.6 billion parameters.

3. Once pre-trained, iGPT can be fine-tuned on specific image tasks, such as image classification or generation. Fine-tuning involves further training the model on a smaller, task-specific dataset to improve its performance on that task. 

4. iGPT has also been used in various creative applications, such as generating realistic faces, animals, and landscapes.

5. It is also used in generating personalised avatars, and synthesising photorealistic images of 3D objects.

Image Classification Problem

Unsupervised and self-supervised learning or learning without human-labeled data is a longstanding challenge of machine learning. But it has been an incredible success in language transformer models like BERT, GPT-2, RoBERTa, T5 and other variants. But the same models have not been successful in producing strong features for image classification. Some of the challenges are;

1. Ambiguity: Images can be complex and ambiguous which makes it difficult to identify meaningful patterns or structures in them. So, it’s a problem for unsupervised learning techniques to disentangle different factors

2. Variability: Since images can vary widely in terms of viewpoint, lighting, background, occlusion, and other factors. Unsupervised techniques may not be able to capture the full range of variation in the data, leading to poor generalisation performance on new images

3. Scale: Image datasets can be very large, with billions of images, making it difficult to process and analyze them using unsupervised learning techniques. Additionally, large datasets require significant computational resources, which can be a bottleneck for training unsupervised models

4. Evaluation: Evaluating the unsupervised features are more challenging since there is no clear metric to check

From Language GPT to Image GPT

Word prediction (like GPT-2 and BERT) have been extremely successful using unsupervised learning in language. Downstream language tasks appear in text, like question-answers, and passages-summaries. But, pixel sequences do not clearly contain labels for the image they belong to. But there is still a reason why GPT-2 on images might work. An idea known as “Analysis by Synthesis” suggests a model to know about object categories. A large transformer is trained on next level prediction which learns to generate diverse samples with clearly recognizable objects. This idea led to usage of  many early generative models and more recently, BigBiGAN is an example which encouraged stronger classification performance. Using this, GPT-2 achieved top-level classification performance in many settings, providing further evidence for analysis by synthesis.

Experimental results

The model performance can be assessed by two methods, which involves downstream classification tasks in both methods. The first method is referred to as a linear probe. It uses the trained model to extract features from the images in the downstream dataset and then it fits a logistic regression to the labels. The second method fine tunes the entire model on the downstream dataset where fine-tuning in iGPT refers to the process of further training the pre-trained GPT model on a smaller, task-specific dataset to improve its performance on that task. 

Both next pixel prediction and image classification are not relevant. In the above graph, it's clear that the feature quality is sharply increasing, then a mildly decreasing function of depth. This suggests that a transformer generative model operates in two phases. In the first phase, a contextualized image feature has been built using gathered information at each position. In the second position, this contextualized feature is used to solve the conditional next pixel prediction task. These two stages are similar to another unsupervised neural net. Next results established the link between generative performance and feature quality.

After conducting enough experiments using linear probes on the ImageNet and also using features, below is the result of accuracy by using different methods and different parameters.

Here, masking of pixels has been done instead of training the model to predict the next pixel and training the model to predict them from the unmasked ones. Its been found that through linear probe performance on BERT models is significantly worse, they excel during fine-tuning.

In recent days, limited amounts of human-labelled data are allowed in the framework of semi-supervised learning. It has been a development to use semi-supervised learning in iGPT which gives path to create images and also relies on clever techniques such as consistency regularisation, data augmentation or pseudo-labeling.

Limitations

iGPT has shown impressive results in image generation and other image related applications. But, it still has some limitations. 

1. Limited ability to generate high-quality images for some applications.

2. It does not have a deep understanding of 3D space. 

3. iGPT is a large and complex model that requires significant computational resources to train and to deploy. 

4. Training iGPT on large datasets can take several weeks or even months using specialised hardware like GPUs. 

5. It is a black-box model. It is difficult to understand how it generates images or specific features. 

6. Most self-supervised results use convolutional-based encoders which consumes inputs at high resolution

7. iGPT generates images based on a single learned distribution. It cannot generate multimodal images.

8. It may have difficulty generating images of rare or unusual objects that are not well-represented in the training data.

Conclusion

Image GPT or iGPT is a powerful generative model that uses unsupervised learning to learn a distribution over images. It is based on the same architecture as the popular language model GPT. and can generate high-quality images that are diverse and realistic. iGPT has shown impressive results on a wide range of tasks, including image completion, texture synthesis, and image manipulation. However, iGPT has some limitations, such as limited spatial reasoning and interpretability, and it can be computationally expensive to train and deploy. Despite these limitations, iGPT represents an important step forward in the development of generative models for images, and it has the potential to enable new applications in areas such as art, design, and entertainment. Further research is needed to address the remaining challenges and limitations of iGPT, and to explore its full potential for generating and manipulating images.