[Discussion] Notion To MD v4: Media Handler Technical Details

[souvikinator]

Media Handler Technical Documentation

Overview

The Media Handler module serves as a crucial component in the notion-to-md pipeline, processing and managing all media-related content after the Block Fetcher completes its work. With the introduction of the builder pattern in v4, the configuration and setup have become more intuitive and flexible while maintaining the core functionality.

High Level Overview

flowchart LR
    A[Receive Block] --> B[Detect Media]
    B --> C{Media Type Check}
    C -->|Notion Asset| D1[Apply Strategy] 
    C -->|External URL| D2[Check Preservation]
    D1 --> E[Process Media]
    D2 -->|Preserve| F1[Keep Original]
    D2 -->|Transform| F2[Apply Strategy]
    F2 --> E
    E --> G[Update Manifest]
    G --> H[Return Updated Block]

Loading

Core Responsibilities

1. Media Detection: Scanning blocks for media content (images, videos, files)
2. Strategy Application: Handling media based on configured strategy (download/upload/direct)
3. Manifest Management: Tracking media files and their relationships to blocks
4. URL Transformation: Converting Notion's temporary URLs to permanent ones
5. Cleanup: Managing orphaned media files

Configuration Using Builder Pattern

The new builder pattern provides a more intuitive way to configure media handling. Here's how the configuration system is structured:

Base Configuration

const n2m = new NotionToMarkdownBuilder(notionClient)
    .setContentPath('./content/posts')  // Base directory for markdown output
    .build();

Media Strategy Configuration

The builder pattern introduces three distinct methods for configuring media handling strategies:

1. Direct Strategy (useNotionUrls):

const n2m = new NotionToMarkdownBuilder(notionClient)
    .useNotionUrls()
    .build();

2. Download Strategy (downloadMediaTo):

const n2m = new NotionToMarkdownBuilder(notionClient)
    .downloadMediaTo({
        outputPath: './public/images',
        preserveExternalUrls: false,
        transformPath: (path) => `/images/${path.basename}`
    })
    .build();

3. Upload Strategy (uploadMediaUsing):

const n2m = new NotionToMarkdownBuilder(notionClient)
    .uploadMediaUsing({
        uploadHandler: async (media) => {
            // Upload implementation
            return url;
        },
        cleanupHandler: async (mediaInfo) => {
            // Cleanup implementation
        },
        preserveExternalUrls: true,
        transformPath: (url) => url.replace(
            'my-blog-assets.s3.amazonaws.com',
            'cdn.myblog.com'
        )
    })
    .build();

Strategy Requirements

Each strategy has specific requirements that must be met:

1. Download strategy requires:

   • outputPath: Destination directory for media files
   • Optional transformPath: Function to customize media URLs
   • Optional preserveExternalUrls: Boolean to control external URL handling

2. Upload strategy requires:

   • uploadHandler: Async function to handle media uploads
   • Optional cleanupHandler: Function to manage file cleanup
   • Optional transformPath: Function to modify URLs post-upload
   • Optional preserveExternalUrls: Boolean to control external URL handling

3. Direct strategy (useNotionUrls):

   • No additional configuration required
   • Simplest to implement but least permanent solution

Low Level Design

flowchart TD
    Start[Receive Block] --> DetectMedia{Contains Media?}
    DetectMedia -->|No| ReturnUnchanged[Return Block]
    DetectMedia -->|Yes| LoadManifest[Load Media Manifest]
    
    LoadManifest --> CheckType{Check Media Type}
    CheckType -->|Notion| StrategyCheck{Check Strategy}
    CheckType -->|External| PreserveCheck{Preserve URLs?}
    
    PreserveCheck -->|Yes| KeepURL[Keep Original URL]
    PreserveCheck -->|No| StrategyCheck
    
    StrategyCheck -->|Download| DownloadProcess[Download Media]
    StrategyCheck -->|Upload| UploadProcess[Upload to Storage]
    StrategyCheck -->|Direct| UseNotionURL[Use Notion URL]
    
    DownloadProcess --> UpdateManifest[Update Manifest]
    UploadProcess --> UpdateManifest
    UseNotionURL --> UpdateManifest
    KeepURL --> UpdateManifest
    
    UpdateManifest --> CleanupCheck{Need Cleanup?}
    CleanupCheck -->|Yes| RunCleanup[Remove Unused Files]
    CleanupCheck -->|No| UpdateBlock[Update Block URLs]
    RunCleanup --> UpdateBlock
    
    UpdateBlock --> End[Return Updated Block]

Loading

Media Manifest System

The Media Handler maintains a manifest file for each page to track media assets and their relationships to blocks. The manifest helps with:

1. Detecting changes to media content
2. Managing orphaned files
3. Optimizing re-downloads
4. Maintaining consistency between builds

Manifest Structure

{
    "block_456": {
      "lastEdited": "2024-01-02T15:00:00Z",
      "mediaPath": "/static/images/page-123/image2.jpg",
      "mediaType": "notion",
      "strategy": "download"
    }
}

Implementation Examples

1. Static Site Generator Implementation

const n2m = new NotionToMarkdownBuilder(notionClient)
    .setContentPath('./content/posts')
    .downloadMediaTo({
        outputPath: './public/images',
        preserveExternalUrls: false,
        transformPath: (path) => `/images/${path.basename}`
    })
    .build();

// This will:
// 1. Download media to ./public/images
// 2. Update markdown with relative paths
// 3. Track media in manifest
await n2m.pageToMarkdown('notion-page-id', 'my-post.md');

2. Cloud Storage Implementation

const n2m = new NotionToMarkdownBuilder(notionClient)
    .setContentPath('./content/posts')
    .uploadMediaUsing({
        uploadHandler: async (media) => {
            const url = await uploadToS3({
                bucket: 'my-blog-assets',
                key: `images/${media.filename}`,
                body: media.data,
                contentType: media.mimeType
            });
            return url;
        },
        cleanupHandler: async (mediaInfo) => {
            await deleteFromS3({
                bucket: 'my-blog-assets',
                key: `images/${mediaInfo.filename}`
            });
        },
        preserveExternalUrls: true
    })
    .build();

3. Preview/Temporary Content Implementation

const n2m = new NotionToMarkdownBuilder(notionClient)
    .setContentPath('./content/posts')
    .useNotionUrls()
    .build();