PrideShield

OrientationStart here

This page is the complete technical account of the project, written to be read straight through. Chapters 1–8 run in the order the work actually happened: problem → data → preprocessing → modeling → evaluation → what was learned → what would change.

Chapter 1The problem

Why a text classifier is structurally unable to solve this, and what the absence of a budget forced the design to become.

The multimodal gap

Automated content moderation is mature on plain text and weak on images that contain text. Anti-LGBTQ+ harassment in meme form sits exactly in that gap. A meme has two channels — the picture and the overlaid caption — and the hostile meaning is frequently carried by neither channel alone but by the relationship between them: an innocuous stock photo paired with an innocuous phrase, where the pairing is the attack.

A text-only classifier sees only the caption, and a caption stripped of its image is often unremarkable. This is not a tuning problem; the signal is genuinely absent from the input. Any system that reads only one channel will systematically under-detect this class of content.

The zero-budget constraint

The project had no funding: no paid APIs, no annotation service, no GPU allocation beyond free-tier Colab. That single constraint shaped the architecture more than any modeling preference did, and reading the rest of this page without it makes several decisions look arbitrary.

The constraint was not purely limiting. Frozen embeddings are computed once and cached, which made the pipeline fast and every experiment cheap. The real cost landed on dataset scale, and that is what bounds every result on this page.

Chapter 2Building the dataset

There was no corpus to download, so the first several months of the project were data engineering, not machine learning.

Meta's Hateful Memes Challenge (2020) was the only substantial public benchmark at the time. It covers hate broadly rather than anti-LGBTQ+ content specifically, and its examples are purpose-built — human annotators constructed "benign confounders" over licensed stock imagery — so it does not reflect memes as they actually circulate. Text-only hate-speech corpora exist in quantity but cannot support a multimodal task at all. The corpus therefore had to be built end to end.

flowchart LR
    A["`**4 sources**
Reddit · 4chan
DDG / Bing / Google`"]
    B["`**1,696**
after pHash
deduplication`"]
    C["`**1,667**
dual-labeled
2 annotators`"]
    D["`**1,320**
final dataset
80.9% / 19.1%`"]
    A --> B --> C --> D
    D --> E["924 train"]
    D --> F["198 validation"]
    D --> G["198 test"]
    style D fill:#5b7f9e,color:#fff,stroke:#3f6180
    style A fill:__NEUTRAL_FILL__,stroke:__NEUTRAL_STROKE__,color:__NEUTRAL_TEXT__
    style B fill:__NEUTRAL_FILL__,stroke:__NEUTRAL_STROKE__,color:__NEUTRAL_TEXT__
    style C fill:__NEUTRAL_FILL__,stroke:__NEUTRAL_STROKE__,color:__NEUTRAL_TEXT__
    

Collection: six methods, three failed

The failures explain the eventual architecture better than the successes do.

Deduplication was a correctness requirement

The measured post-deduplication corpus is 1,696 images, 476 MB. The pre-dedup raw count was not preserved in the surviving artifacts, so no deduplication rate is claimed here.

Annotation and label reconciliation

1,667 memes were labeled independently by two annotators using three codes: hateful (1), not hateful (0), and a third code for items that could not be judged (−1). Independent dual labeling on a subjective task is not redundancy — the disagreement rate is itself a measurement of how hard the task is.

Dataset statistics

All figures measured from the surviving artifacts, not estimated.

OCR ran over 1,405 distinct memes and produced 19,938 text regions — an average of 14.2 detected regions per image. That number is a concrete measure of how fragmented meme text is compared with ordinary document OCR, where a page is a handful of blocks. Mean detection confidence was 0.711 (median 0.793) with a long tail down to zero: the quantitative case for the correction stack in Chapter 3.

OCR coverage is 1,405 of the 1,696 deduplicated images. The 291-image gap is not explained by the surviving artifacts — most likely unreadable or non-meme files removed before extraction — and is recorded here rather than smoothed over.

The final table keeps Extracted Text (raw OCR), corrected_text, levenshtein_distance and normalized_levenshtein side by side, so the correction stage is auditable per row rather than being an opaque transformation.

Splits

A single fixed 70/15/15 split, generated once and reused across all three encoders.

Chapter 3Preprocessing

Turning scraped images into a usable multimodal training table — where the hard part was reading the text off the images.

flowchart TD
    R["Raw scraped images"]
    R --> D["`**Deduplication**
perceptual hash (pHash)`"]
    D --> O["`**OCR extraction**
EasyOCR`"]
    O --> T["`**T5 correction**
structural repair`"]
    T --> L["`**Levenshtein repair**
token-level spellcheck`"]
    L --> M["`**Label reconciliation**
keep only agreeing items`"]
    M --> F["`Training table
image + corrected caption + label`"]
    style F fill:#5b7f9e,color:#fff,stroke:#3f6180
    style O fill:#a8813f,color:#fff,stroke:#83642f
    style T fill:#a8813f,color:#fff,stroke:#83642f
    style L fill:#a8813f,color:#fff,stroke:#83642f
    style R fill:__NEUTRAL_FILL__,stroke:__NEUTRAL_STROKE__,color:__NEUTRAL_TEXT__
    style D fill:__NEUTRAL_FILL__,stroke:__NEUTRAL_STROKE__,color:__NEUTRAL_TEXT__
    style M fill:__NEUTRAL_FILL__,stroke:__NEUTRAL_STROKE__,color:__NEUTRAL_TEXT__
    

OCR extraction

OCR — optical character recognition — is the step that reads text out of an image and returns it as characters. Meme text is close to the worst case for it: stylized fonts, low contrast against busy backgrounds, spread across multiple panels with no reading order, and frequently rendered as part of the picture rather than overlaid on it. EasyOCR handled the common cases and failed on the rest, producing dropped characters, merged words, spurious characters hallucinated from background texture, and scrambled ordering.

The correction stack

Two stages were layered on the raw EasyOCR output, in this order and for this reason:

1. T5 sequence-to-sequence correction. T5 is a text-to-text transformer; here it rewrites a damaged string into a plausible clean one. It handles structural damage — merged words, lost spacing, broken segmentation — which spans token boundaries and therefore cannot be fixed by a spellchecker operating one word at a time.
2. Levenshtein / spellchecker repair. Levenshtein distance counts the single-character insertions, deletions and substitutions needed to turn one string into another. This stage maps residual character-level errors onto their nearest dictionary neighbours, and doubles as a partial defense against the deliberate misspellings used to evade keyword filters.

Measured effect: mean edit distance of 8.5 characters between raw and corrected text (median 6, max 229). No ablation of the correction stack was run, so its contribution to final accuracy is documented but not isolated — see Chapter 8.

Chapter 4Modeling

How a meme becomes two numeric vectors, and what is trained on top of them.

How CLIP works, and why it fits

flowchart TB
    subgraph PRE["CLIP pretraining · contrastive objective"]
      direction LR
      IM["Image"] --> VIT["`Vision encoder
ViT or ResNet`"]
      TX["Caption"] --> TXE["`Text encoder
Transformer`"]
      VIT --> EMB["`**Shared embedding space**
matching pairs pulled together,
mismatched pushed apart`"]
      TXE --> EMB
    end

    subgraph USE["How PrideShield uses it · encoders frozen"]
      direction LR
      MI["Meme image"] --> FV["`Vision encoder
frozen`"]
      MC["OCR caption"] --> FT["`Text encoder
frozen · 77 tokens`"]
      FV --> V1["`image vector
512 / 768 / 1024-d`"]
      FT --> V2["`text vector
same width`"]
    end

    PRE -.->|"pretrained weights"| USE

    style EMB fill:#5b7f9e,color:#fff,stroke:#3f6180
    style VIT fill:__NEUTRAL_FILL__,stroke:__NEUTRAL_STROKE__,color:__NEUTRAL_TEXT__
    style TXE fill:__NEUTRAL_FILL__,stroke:__NEUTRAL_STROKE__,color:__NEUTRAL_TEXT__
    style IM fill:__NEUTRAL_FILL__,stroke:__NEUTRAL_STROKE__,color:__NEUTRAL_TEXT__
    style TX fill:__NEUTRAL_FILL__,stroke:__NEUTRAL_STROKE__,color:__NEUTRAL_TEXT__
    style FV fill:#a8813f,color:#fff,stroke:#83642f
    style FT fill:#a8813f,color:#fff,stroke:#83642f
    

Architecture and fusion

Free-tier Colab could not fine-tune a 304M-parameter vision transformer, which fixed the architecture before any modeling preference entered: frozen CLIP encoders as feature extractors, plus a small trainable MLP head. The upside is that embeddings are computed once and cached to disk, after which every experiment trains in seconds — which is what made a three-encoder comparison and a 30-trial hyperparameter search affordable at all.

flowchart LR
    I["Meme image"] --> VE["`CLIP vision encoder
frozen`"]
    C["OCR caption"] --> TE["`CLIP text encoder
frozen · 77-token limit`"]
    VE --> N1["L2 normalise"]
    TE --> N2["L2 normalise"]
    N1 --> CAT["`concatenate
2 × embed dim`"]
    N2 --> CAT
    CAT --> MLP["`**MLP head** (trained)
512 → 256 → 128 → 2
LeakyReLU · dropout 0.4`"]
    MLP --> OUT["hateful / not hateful"]
    style VE fill:__NEUTRAL_FILL__,stroke:__NEUTRAL_STROKE__,color:__NEUTRAL_TEXT__
    style TE fill:__NEUTRAL_FILL__,stroke:__NEUTRAL_STROKE__,color:__NEUTRAL_TEXT__
    style I fill:__NEUTRAL_FILL__,stroke:__NEUTRAL_STROKE__,color:__NEUTRAL_TEXT__
    style C fill:__NEUTRAL_FILL__,stroke:__NEUTRAL_STROKE__,color:__NEUTRAL_TEXT__
    style MLP fill:#5b7f9e,color:#fff,stroke:#3f6180
    style OUT fill:#54876a,color:#fff,stroke:#3c6650
    

L2 normalisation before concatenation scales each modality's vector to unit length, so whichever encoder happens to produce larger-magnitude outputs cannot dominate the fused representation purely by scale. The fused vector is then twice the encoder's embedding width — 1,024 for ViT-B/32, 1,536 for ViT-L/14@336px, 2,048 for RN50x64.

# src/models.py — the trained component, in full
fuse = L2norm(image_emb) ‖ L2norm(text_emb)     # → 2 × embed_dim

Linear(2·embed_dim, 512) → LeakyReLU → Dropout(0.4)
    → Linear(512, 256)   → LeakyReLU → Dropout(0.4)
    → Linear(256, 128)   → LeakyReLU → Dropout(0.4)
    → Linear(128, 2)

The three encoders

CLIP ships in several sizes and two architectural families. Three were benchmarked under identical downstream conditions — same head, same search budget, same splits — so that differences are attributable to the encoder rather than to the setup around it.

Parameter counts are the CLIP vision tower only, computed directly with open_clip rather than quoted from memory; full model sizes including the text tower are 151.3M, 427.9M and 623.3M respectively. Note that ViT-L/14@336px varies patch size, input resolution and parameter count together, so the comparison identifies which encoder is better but does not isolate why.

Cached embeddings

Each encoder's embeddings were computed once and written to disk, which is what made repeated experiments cheap:

Class imbalance

At roughly 80/20, a model that always predicts "hateful" already scores about 80% — so the training signal has to be corrected or the model will learn the shortcut. Two standard strategies were implemented:

• Focal Loss, written from scratch rather than imported. It multiplies the standard cross-entropy by (1 − p_t)γ, which shrinks the contribution of examples the model already gets right and concentrates gradient on the hard ones.
• Class-weighted cross-entropy, which simply weights each class inversely to its frequency.

Hyperparameter search

A 30-trial Optuna study per encoder searched hidden-layer widths, dropout, activation, learning rate, weight decay and loss function, selecting on validation performance. The winning configuration for ViT-L/14@336px:

Chapter 5Evaluation

The protocol first, then the numbers — because the numbers only mean something given the protocol.

Protocol

• Test set: 198 examples, 165 hateful / 33 not hateful, one fixed split, never used for model selection.
• Selection: hyperparameters chosen on the 198-example validation split; the test split was scored once per encoder.
• Reference floor: the majority-class predictor scores 83.33% on this test split.
• Text-only baseline: a transformer classifier trained on FRENK-LGBT, a public text corpus — not on this project's meme captions.
• Not done: k-fold cross-validation, multi-seed runs, confidence intervals.

Results

Best configuration — CLIP ViT-L/14@336px → searched MLP head:

How to read these numbers

ROC-AUC is the most robust single number here at 0.9030. It measures how well the model ranks a random hateful example above a random benign one across all thresholds, so unlike accuracy it is insensitive to the class skew that makes the accuracy figures flattering.

Precision, recall and F1 are positive-class figures — computed with average="binary" on the hateful class, which is the easy class at 83% prevalence. Macro-averaged figures, which weight the small benign class equally, would be materially lower and were not recorded.

Chapter 6Findings

What the experiments actually established — including the parts that did not work.

More parameters did not mean better representations

RN50x64 has the most parameters (420.4M in the vision tower) and the widest embedding (1,024-d) of the three — and finished last. It lost by ~5 points to ViT-B/32, which has roughly a fifth of its parameters and half its embedding width, and by ~9 points to ViT-L/14@336px, which is ~28% smaller.

Capacity was not the binding factor; representation quality was. A plausible reading is that the task requires relating overlaid text semantics to image semantics across the whole frame, and ViT's global self-attention captures that relational structure in a way convolutional inductive bias, with its local receptive fields, does not. Stated as an interpretation, not a proven mechanism — CLIP's ViT and ResNet variants also differ in training recipe and data seen, so architecture is not the only variable in play.

Rejected variants

The fusion ceiling

Concatenating two frozen embeddings is early fusion in its crudest form. The MLP receives two independent vectors glued end to end and must infer their relationship from scratch, with no mechanism for the modalities to attend to one another. For memes this is a real handicap, and precisely at the point that defines the task: hostility lives in the interaction between image and caption — a benign image plus a benign phrase producing a hateful whole — and concatenation gives the model no direct way to represent that interaction. Cross-attention was the obvious alternative and was out of reach on free-tier compute.

Chapter 7Future work

This is 2024-era tooling. What has changed since, and how the project would be rebuilt today.

Frozen CLIP plus a small head was a defensible design under a zero-dollar budget in 2024. It is not what anyone should build now — and the parts of the field that moved fastest are precisely the parts this project struggled with: multimodal fusion, OCR on stylized text, and labeling throughput.

Where the state of the art is

The relevant capability is not better image classification — it is that these models read the text inside the image and reason about its relationship to the imagery in a single pass. That is exactly the operation concatenated CLIP embeddings cannot perform, and it dissolves the fusion ceiling described above rather than engineering around it. Separately, LoRA / QLoRA now make it feasible to fine-tune multi-billion-parameter VLMs on a single consumer GPU, so the compute wall that forced frozen encoders is largely gone.

The rebuild, in order

1. Establish the zero-shot baseline first. Measure a modern VLM zero-shot on the existing 1,320 examples, reporting macro metrics and AUPRC. This may end the project — if zero-shot matches 93.94%, the trained pipeline has no justification. This inverts the original workflow: collect → train → evaluate becomes evaluate-what-exists → collect-only-where-it-fails.
2. Break the labeling bottleneck. Dataset size bounded nearly every result here, and labeling throughput bounded dataset size. VLM pre-labeling with human adjudication of low-confidence cases could plausibly cover an order of magnitude more data for the same effort. Keep contested items as soft labels or a hard-case benchmark instead of dropping them, and deliberately collect benign content to reach a realistic class balance.
3. Replace the fusion mechanism. Fine-tune a VLM directly (LoRA/QLoRA) so the model attends jointly over image and caption natively — and drop the OCR correction stack entirely, since modern VLMs read stylized meme text far better than 2024-era OCR.
4. Fix the evaluation protocol. Stratified splits; a like-for-like text-only ablation on the same corpus; k-fold CV with confidence intervals; macro metrics and AUPRC as headline figures; multi-seed runs; confusion matrices retained.
5. Evaluate on public benchmarks too. The space is far better served now than in 2024 — HarMeme/Harm-P, MAMI, HatReD (hateful-meme reasoning and explanation) and HOMO-MEX for Spanish-language LGBTQ+ hate. A custom corpus should be the domain-specific supplement, not the whole evaluation.
6. Evaluate what actually gates deployment. False-positive rate on reclaimed in-group language — the dominant failure mode, and the one where errors do the most harm. Adversarial robustness to character substitution, leetspeak, crops and re-encoding. Subgroup breakdowns. Calibration, so uncertain cases can be routed to a human.
7. Explanation, not just classification. A VLM can be prompted to explain why a meme is hateful, producing an auditable rationale. For moderation — where decisions get appealed and must be justified — that is far more useful than a scalar score.
8. Infrastructure from day one. Experiment tracking, config-driven sweeps, seeded runs tied to versioned dataset snapshots.

Chapter 8Limitations

What these results do not support. This chapter is load-bearing.

ReferenceGlossary

Every term used above, in one place.

Embedding

A fixed-length list of numbers representing a piece of content, arranged so semantically similar content lands nearby in that space.

CLIP

Contrastive Language–Image Pretraining. A vision encoder and a text encoder trained jointly on image–caption pairs, producing a shared space where image and text vectors are directly comparable.

Contrastive objective

A training signal that pulls matching pairs together in embedding space and pushes mismatched pairs apart.

Frozen encoder

A pretrained model used purely as a feature extractor — its weights are not updated during training. Cheap, and it lets embeddings be computed once and cached.

ViT (Vision Transformer)

An image model that splits the image into fixed-size patches and applies self-attention across all of them, giving every patch a global view of the image.

ResNet / convolutional inductive bias

A convolutional architecture that builds up features from local receptive fields. Efficient and strong on texture and local pattern, but relational structure across the whole frame has to be assembled through depth.

Early fusion / concatenation

Combining two modalities by gluing their vectors end to end before the classifier. Simple, and unable to model interaction between the modalities directly.

Cross-attention

A fusion mechanism that lets each modality attend to the other, so the model can represent their interaction explicitly. The alternative to concatenation, and more expensive.

MLP head

A small stack of fully connected layers trained on top of frozen features to produce the final classification.

L2 normalisation

Rescaling a vector to unit length, so magnitude differences between modalities do not let one dominate the fused representation.

Dropout

Randomly zeroing a fraction of activations during training to reduce overfitting.

Focal Loss

A loss that multiplies cross-entropy by (1 − p_t)γ, down-weighting examples the model already classifies confidently so gradient concentrates on hard ones.

Class-weighted cross-entropy

Standard cross-entropy with each class weighted inversely to its frequency, so the minority class is not ignored.

Majority-class floor

The accuracy obtained by always predicting the most common class. On the test split here it is 83.33%, and it is the only sensible reference point for the accuracy figures.

ROC-AUC

The probability the model ranks a random positive above a random negative. Threshold-independent and far less flattered by class imbalance than accuracy.

AUPRC

Area under the precision–recall curve. More informative than ROC-AUC when the positive class is rare — not recorded for this project.

Macro vs. binary averaging

Binary averaging reports metrics for the positive class only; macro averaging weights every class equally. The figures here are binary, which is the more flattering of the two on this data.

Optuna

A hyperparameter-search library. Used here for a 30-trial study per encoder over layer widths, dropout, activation, learning rate, weight decay and loss function.

OCR

Optical character recognition — reading text out of an image and returning it as characters. EasyOCR was used here.

Perceptual hash (pHash)

A hash of an image's visual content rather than its bytes, so re-encoded, cropped or watermarked copies produce similar hashes and can be detected as duplicates.

Levenshtein distance

The number of single-character insertions, deletions and substitutions needed to turn one string into another.

T5

A text-to-text transformer; used here to rewrite structurally damaged OCR output into plausible clean text.

VLM (vision-language model)

A model that takes an image and a text prompt together and reasons over both jointly — the class of model that supersedes the embed-then-classify design used here.

LoRA / QLoRA

Parameter-efficient fine-tuning: train small low-rank adapters over a (quantized) frozen base model instead of updating all its weights.