Multi-Modal Sarcasm Detection Based on Cue Learning
Research project on detecting sarcasm using multi-modal approach combining text and images
Overview
Developed a multi-modal approach for sarcasm detection by integrating text and image data using CLIP architecture combined with Large Language Models. Achieved significant improvements in sarcasm detection accuracy compared to uni-modal approaches, addressing the challenge of understanding complex sarcastic expressions in social media content.
Problem & Objective
The rapid growth of internet and social media has generated massive volumes of data requiring sentiment analysis for public opinion monitoring and decision-making. Sarcasm, where intended meaning often contradicts literal meaning, poses a major challenge causing errors in sentiment analysis. Traditional uni-modal approaches using only text are insufficient to capture complex nuances of sarcasm, while existing multi-modal models often require large datasets and perform poorly on small samples.
My Contribution
- Developed multi-modal sarcasm detection model combining CLIP and LLM architectures
- Integrated LLM for text pre-processing to enhance semantic understanding
- Leveraged CLIP's multimodal capabilities (text + image) for effective feature fusion
- Achieved 83.56% accuracy and 79.48% F1-Score on 1024 test samples
- Compared multiple architectures including CLIP-only and ViT+BERT combinations
- Demonstrated significant performance advantage over uni-modal approaches
Features
- Multi-modal input processing (text and images)
- CLIP-based feature extraction for text-image alignment
- LLM integration for advanced text preprocessing
- Cue learning approach for sarcasm detection
- Effective performance on small-sample scenarios
- Comparative analysis of multiple model architectures
- High accuracy (83.56%) and F1-Score (79.48%)
Challenges
Handling the complexity of sarcasm where literal and intended meanings often contradict each other. Balancing model performance with dataset size constraints, as traditional multi-modal models require large training datasets. Successfully integrated multiple modalities while maintaining efficiency and achieving strong performance even with limited samples.
Outcome
Achieved superior performance with CLIP+LLM model reaching 83.56% accuracy and 79.48% F1-Score on 1024 test shots. The model consistently outperformed alternatives including CLIP-only and ViT+BERT combinations across all metrics, demonstrating the effectiveness of integrating LLM preprocessing with CLIP's multimodal architecture. This research contributes to more accurate sentiment analysis and public opinion assessment in social media contexts.
Impact
Demonstrated that integrating CLIP with LLM for multi-modal sarcasm detection significantly outperforms traditional approaches. The research shows that combining visual and textual cues through CLIP's architecture, enhanced by LLM's text processing capabilities, creates a powerful synergy for understanding sarcastic content. This approach addresses the critical challenge of accurate sentiment analysis in social media monitoring.
Tech Stack
Machine Learning · Deep Learning · NLP · Computer Vision · CLIP · LLM · Research