[Discussion] Notion To MD v4: Block Fetcher Technical Details

[souvikinator]

Block Fetcher Technical Doc

Overview

graph TD
    A[Initialize BlockFetcher] --> B[Configure Settings]
    B --> C[Queue Management]
    C --> D[Rate Limited Fetching]
    D --> E[Block Organization]
    E --> F[Return Structured Data]

Loading

The module acts as an fetching layer that handles the complexity of retrieving nested Notion content while balancing performance and API constraints. It's designed to solve several challenges:

1. Concurrent Block Fetching
   Instead of fetching blocks sequentially or recursively (which was a performance bottleneck in previous versions)
   a. it implements a batch processing system. All elements of the batch are processed concurrently.
   b. The code uses a configurable batch size (default: 3) to fetch multiple blocks concurrently while staying within API limits.

2. Rate Limiting Management
   The component implements a configurable rate limiting system:
   a. Tracks requests per second (default max: 3)
   b. Implements cooldown periods when limits are reached

3. State Management
   It maintains several internal data structures:
   a. queue: Tracks blocks that need processing
   b. results: Stores fetched blocks
   c. pageProps: Stores page properties
   d. processedBlocks: Prevents duplicate processing

4. Block Organization
   The component handles the complexity of Notion's nested structure:
   a. Maintains parent-child relationships
   b. Preserves block ordering
   c. Supports child page content (optional)
   d. Handles page properties (optional)

Configuration Options

The BlockFetcher accepts the following configuration parameters:

1. includeChildPageContent (boolean): Controls whether content from child pages should be fetched
2. fetchPageProperties (boolean): Determines if page properties should be retrieved
3. maxRequestsPerSecond (number): Sets the rate limit for API requests (default 3 as per Notion API)
4. batchSize (number): Defines how many blocks to process in parallel

Detailed Implementation Flow

flowchart TD
    Init[Initialize BlockFetcher] --> Config[Load Configuration]
    Config --> Queue[Initialize Queue with Root ID]
    Queue --> Props{Fetch Page Props?}
    Props -->|Yes| FetchProps[Get Page Properties]
    Props -->|No| Process
    FetchProps --> Process[Process Queue]
    
    subgraph QueueProcessing[Main Processing Loop]
        Process --> Empty{Queue Empty?}
        Empty -->|No| Batch[Take Batch of Blocks]
        Batch --> RateLimit[Apply Rate Limiting]
        RateLimit --> Fetch[Fetch Block Children]
        Fetch --> Store[Store Results]
        Store --> HasChildren{Has Children?}
        HasChildren -->|Yes| TypeCheck{Block Type}
        TypeCheck -->|Regular| AddToQueue[Add to Queue]
        TypeCheck -->|Child Page| ChildConfig{Include Child Pages?}
        ChildConfig -->|Yes| AddToQueue
        ChildConfig -->|No| Process
        HasChildren -->|No| Process
        AddToQueue --> Process
    end
    
    Empty -->|Yes| Organize[Build Block Hierarchy]
    Organize --> Output[Return Data Structure]

Loading

Interface

interface BlockFetcherConfig {
  includeChildPageContent?: boolean;
  fetchPageProperties?: boolean;
  maxRequestsPerSecond?: number;
  batchSize?: number;
}

interface FetcherOutput {
  properties: PageObjectProperties;
  blocks: ListBlockChildrenResponseResults;
}

class BlockFetcher {
  constructor(client: Client, config?: BlockFetcherConfig);
  async getBlocks(pageId: string): Promise<FetcherOutput>;
  private async rateLimitRequest<T>(request: () => Promise<T>): Promise<T>;
  private async fetchBlockChildren(blockId: string): Promise<ListBlockChildrenResponseResults>;
  private organizeBlocks(rootId: string, processed?: Set<string>): ListBlockChildrenResponseResults;
}