Multimodal Book Genre Prediction

A multimodal system that predicts book genres from cover images by combining complementary visual encoders with text representations.

The architecture pairs ViT and VisionMamba image encoders, which capture global and local visual structure respectively, with text embeddings derived from vision-language-model-generated descriptions of each cover. Rather than relying on OCR to read the limited text printed on a cover, the VLM descriptions supply richer thematic and narrative cues that the raw tokens miss.

Evaluated on 57,000 covers spanning 30 genres, the method reaches 63.31% Top-1 and 83.03% Top-3 accuracy. The project provided experience with transformer and state-space visual backbones, multimodal feature fusion, and evaluation on a substantial dataset.