Multi-Label Recognition of Non-Rigid Characters in Video via Transfer Learning Using Deep Learning Architectures

Image classification and object detection have been active research areas for decades, evolved from manually designed feature extraction methods to deep learning based approaches. While Deep Learning has revolutionized these fields, it also has its drawbacks. Deep Learning methods require a huge amount of labeled training data, which is time consuming and costly. Transfer Learning addresses this problem, by utilizing pre-trained weights from models that have already learned general features, such as shapes, textures and edges and fine-tuning them on smaller domain-specific datasets. This thesis investigates the effects of transfer learning on frame-wise multi-label character recognition of "The Muppet Show", a TV series featuring a cast of puppet characters. The task is to determine which of the six main characters, namely Kermit The Frog, Miss Piggy, Fozzie Bear, Rowlf The Dog, Statler & Waldorf and The Swedish Chef, appear in each frame, where multiple characters can be present simultaneously. First, a selection of backbone architectures is compared to identify the best model. This model then undergoes an ablation study to further optimize performance and finally a qualitative generalization analysis on unseen episodes is conducted using Gradient-weighted Class-Activation Mapping (Grad-CAM). The results of this thesis show that transfer learning provides the single largest performance gain, increasing the average macro-mAP from 42.40% to 74.75% with less training time required than training from scratch. CLIP-ViT-B/16 proved to be the best-performing backbone, achieving the highest macro-mAP, as well as the lowest performance loss on Out-Of-Distribution episodes. Unfreezing two encoder layers and applying data augmentation improved the average macro-mAP to 89.16%. The Grad-CAM analysis further reveals that the model successfully learned character-specific discriminative features for most characters, but also shows that setting-bound characters such as Statler & Waldorf and the Swedish Chef are sometimes detected through their background environment. Overall, this work demonstrates that transfer learning enables effective character recognition on a limited Muppet Show dataset, but truly robust generalization remains constrained by the diversity of the training material.