The widespread presence of social media in everyday life has led to a surge in online interactions. While it offers numerous benefits, social media is frequently misused as a platform to propagate hate and negativity, which can have detrimental effects on users’ mental well-being. Although several tools exist to detect harmful content, low-resource languages like Bengali remain significantly under-resourced in such efforts. Given that contemporary social media content often spans multiple modalities like text and images, a multimodal approach offers a more robust solution for automatic content moderation. This study presents a multimodal hateful meme detection model tailored for Bengali, which simultaneously processes textual and visual information. The Res-AFFNet (Residual-Adapter Feature-level Fusion Network) architecture employs MuRIL (Multilingual representations for Indian languages) and ViT (Vision Transformer) to process text and image, respectively. Each modality’s embeddings are passed through lightweight adapter units, and a residual fusion (80% original +20% adapted) is applied. The adapter outputs are fused and passed through a linear projection and classifier to predict whether the input is ‘hate’ or ‘not-hate’. The multimodal approach outperforms unimodal approaches in terms of accuracy. Trained and evaluated on the MUTE dataset, the model achieves an improvement of 1.27% over the existing baseline. Additionally, ablation studies highlight the effectiveness of the multimodal framework compared to unimodal and reduced-component variants. Multimodal Bengali hate speech detection can be used for automatic content moderation on social networking websites and help support low-resource communities. One future scope of this study is that this framework could be extended to other low-resource languages. Fine-grained classification of hate speech can also be incorporated in this framework by expanding the dataset annotations.