Data/Masking Variations

[pszemraj]

thread for ideas on data/masking. We already know that fixed MLM mask %, masking random tokens the whole time isn't the best way to pretrain.

• mixing in poor quality data via some schedule/curricula. masked model should see what trash looks like
• "intelligent masker" sort of like generator in electra, but not fake words, where to mask. Draws on Ernie's work in v1-v3. Should not completely replace random masking, and probably follow some phases/schedule/curricula as well. Does not need to be trained alongside model (potentially) instead leveraging gliner-bi-edge-v2.0 or similar

[pszemraj]

re: poor quality data/CoLA: see Language Models Improve When Pretraining Data Matches Target Tasks

[pszemraj]

MeCo inspired metadata conditioning

Metadata Conditioning for Pre-training Efficiency https://arxiv.org/abs/2501.01956

Core Adaptations

1. Asymmetric masking schedule

• 70% runs: metadata visible, mask only content
• 20% runs: content visible, mask only metadata
• 10% runs: standard MLM (mask both)
• Intuition: Explicitly teach bidirectional source↔content mapping before full MLM task

2. Metadata prediction auxiliary loss

• Add task: predict masked URL from content
• Loss: L_total = L_MLM + 0.1 * L_metadata
• Intuition: Make grouping explicit rather than implicit; forces learning "what source produces this content?"

3. Source-adaptive masking rates

• Wikipedia: 10% masking (dense factual text)
• Code repos: 20% masking (high redundancy)
• Social media: 25% masking (noisy, benefits from reconstruction)
• Intuition: Information density varies by source; MLM-specific opportunity (impossible in causal LMs)

4. Extended cooldown (20% vs 10%)

• MLM tasks are metadata-free at fine-tuning
• Test if longer cooldown is necessary
• Intuition: Downstream forgetting is faster for discriminative tasks

more stuff (click to expand)

Key Comparisons

5. Position ablation: prepend vs append vs surround

• Prepend: [URL] [content]
• Append: [content] [URL]
• Surround: [URL-start] [content] [URL-end]
• Intuition: Bidirectional attention might make position less critical than in causal LMs

6. Cross-document attention: help or hurt?

• Test with/without cross-doc attention (opposite hypothesis from causal LMs)
• Intuition: Might actually help if masked tokens attend to similar-source content across docs

7. Efficiency gain measurement

• Match baseline GLUE scores with X% less data
• Prediction: 10-20% gain (vs 33% for causal LMs)
• Intuition: Bidirectional attention partially compensates for missing metadata

Critical Validation

8. Zero-shot vs fine-tuned performance

• Compare metadata effects before/after fine-tuning
• Test if conditional inference survives classifier head training
• Intuition: Fine-tuning likely erases metadata conditioning; method most useful for few-shot scenarios

9. Metadata maskable vs never-masked

• A: Metadata always visible (like MeCo)
• B: Metadata treated like normal tokens (can be masked)
• Intuition: Tests whether consistent conditioning signal or bidirectional prediction task works better

---

Priority order: 1 → 2 → 7 → 4 → 3 (cover core method, novel MLM opportunity, validate gains, fix cooldown, optimize per-source)

[pszemraj]

SkyLadder Implementation for NeoBERT

Better and Faster Pretraining via Context Window Scheduling

it's just dynamic attention masking that fragments sequences during training = faster, potentially better

code https://github.com/sail-sg/SkyLadder