Architecture & Design

Fresh

Design Principles

MD-LD follows strict engineering principles for reliability and performance:

• Zero dependencies — Pure JavaScript, ~85KB unminified (20KB gzipped)
• Streaming-first — Single-pass parsing, O(n) complexity
• Character-based tokenization — 20-28% faster than regex-based approaches
• Memory-efficient — ~640 bytes per quad retained after GC
• Standards-compliant — RDF/JS data model, W3C CURIE 1.0
• Dual-layer origin — Every parse emits both a semantic quad graph and a walkable textual topology graph simultaneously
• Explicit semantics — No guessing, inference, or heuristics
• Deterministic — Same input always produces same output

Processing Pipeline

Document processing follows this pipeline:

graph LR
    A[MD-LD Text] --> B[Tokenization]
    B --> C[Prefix Resolution]
    C --> D[Semantic Processing]
    D --> E[Quad Emission]
    E --> F[Origin Tracking]
    F --> G[Final Resolution]
    G --> H[quads + origin + primary]

Stages in detail:

1. Line-by-line scanning — Sequential token creation with character ranges
2. Context resolution — Prefix and vocabulary expansion
3. Subject tracking — Current subject management
4. Annotation processing — Semantic block evaluation
5. Quad emission — RDF triple generation with polarity handling
6. Origin tracking — Block and span chain construction
7. Final resolution — Hard invariant enforcement, primary metadata extraction

Core Parser Structures

Token Processing Pipeline

The parser uses character-based tokenizers for optimal performance:

const PROCESSORS = [
    { type: 'fence',      test: line => detectFence(line.trim()),      process: handleFence },
    { type: 'prefix',     test: line => detectPrefix(line),            process: handlePrefix },
    { type: 'standalone', test: line => detectStandaloneSubject(line), process: handleStandaloneSubject },
    { type: 'heading',    test: line => detectHeading(line),           process: handleHeading },
    { type: 'list',       test: line => detectList(line),              process: handleList },
    { type: 'blockquote', test: line => detectBlockquote(line),        process: handleBlockquote },
    { type: 'para',       test: line => line.trim(),                   process: handlePara }
];

Character-Based Tokenization

Character-based tokenizers replace regex patterns for performance:

• detectFence() — Code block fence detection
• detectPrefix() — Namespace prefix declarations
• detectHeading() — Markdown heading detection
• detectList() — List item detection
• detectBlockquote() — Blockquote detection
• detectStandaloneSubject() — Subject annotation detection

Parser State Object

const state = {
    ctx: {},                    // Prefix context for IRI expansion
    df: DataFactory,            // RDF/JS DataFactory instance
    quads: [],                  // Final resolved graph state
    quadBuffer: new Map(),      // Current parsing buffer
    removeSet: new Set(),       // External retractions
    origin: {
        quadIndex: new Map(),   // Lean quad provenance
        blocks: new Map(),      // Semantic anchors
        spans: new Map(),       // Textual topology
        documentStructure: []
    },
    currentSubject: null,       // Current subject context
    primarySubject: null,       // First non-fragment subject declaration
    primaryType: null,          // First rdf:type declaration
    primaryLabel: null,         // First rdfs:label literal
    primaryComment: null,       // First rdfs:comment literal
    statements: [],             // Elevated statements array
    statementCandidates: new Map(), // Incomplete rdf:Statement patterns
    lastBlockEnd: 0,            // Byte position after last block
    lastBlockId: null,          // ID of previous block
    lastSpanId: null            // ID of previous span
};

Origin: Blocks and Spans

The parser output includes a complete document chain at no extra cost:

[Block] --(Span)-- [Block] --(Span)-- [Block]

• Blocks (origin.blocks) — semantic anchors: tokens that produced RDF quads, with prevSpanId/nextSpanId links
• Spans (origin.spans) — textual observations: raw byte ranges between blocks, with bidirectional block and span links

Spans store no text — content is always recovered via sourceText.slice(span.range[0], span.range[1]). This unlocks context-aware UI, autocomplete neighborhood retrieval, and cross-document topology without any parser-level interpretation.

Performance Architecture

Memory Management

• Streaming parsing — Full document never in memory
• Lazy evaluation — Context computed on demand
• Efficient indexing — O(1) subject/object/predicate lookup
• Garbage collection — Minimal retained state (~640 bytes per quad)

Character-Based Optimization

Character-based tokenizers provide:

• 20-28% faster parsing than regex-based approaches
• Predictable performance — O(n) complexity
• Memory efficiency — No regex engine overhead
• Maintainability — Clear character logic

Scaling Architecture

Real-time Applications (60fps):

• Maximum: 4,527 quads per frame
• Use case: Interactive knowledge graphs
• Pattern: Incremental updates only

Batch Processing (1-second):

• Maximum: 225,059 quads per second
• Use case: Background reindexing, imports
• Pattern: Worker thread, chunked processing

Enterprise Scale:

Size	Quads	Approach
Small	under 4K	Real-time updates
Medium	4K to 225K	Batch reindexing
Large	over 225K	Streaming architecture

Retraction System

Retractions are first-class citizens in MD-LD. The system enforces a hard invariant: quads ∩ remove = empty set — a triple is never simultaneously present and retracted.

function processRetraction(quad, quadBuffer, removeSet, quads) {
    const quadKey = quadIndexKey(quad.subject, quad.predicate, quad.object);

    if (quadBuffer.has(quadKey)) {
        // Intra-document cancel: remove from current state
        quadBuffer.delete(quadKey);
        removeFromQuadsArray(quads, quad);
    } else {
        // External retract: target prior document state
        removeSet.add(quad);
    }
}

Elevated Statements Detection

The parser detects rdf:Statement patterns in a single pass. When a node is typed as rdf:Statement and has rdf:subject, rdf:predicate, and rdf:object properties, the parser automatically extracts the elevated SPO triple and adds it to result.statements.

Primary Metadata Extraction

The parser tracks four primary metadata fields during parsing to provide immediate document identity without post-processing:

Field	Source	Purpose
primarySubject	First non-fragment {=subject}	Canonical document identity
primaryType	First .Class declaration	Document category
primaryLabel	First {label} literal	Human-readable name
primaryComment	First {comment} literal	Human-readable description

These are exposed in the parse result as:

• result.primarySubject — the string IRI
• result.primary — object with { subject, type, label, comment }

Standards Compliance

RDF/JS Data Model

MD-LD implements the RDF/JS data model specification:

• NamedNode — IRIs and CURIEs
• Literal — Typed and language-tagged values
• Quad — RDF triples with graph context
• Dataset — Quad collections with indexing

W3C Standards

• RDF 1.1 — Core RDF concepts
• RDFS — Schema vocabulary
• PROV-O — Provenance ontology
• SHACL — Constraint validation
• W3C CURIE 1.0 — Compact URI syntax

Single-Pass Design

The parser processes documents in a single forward pass:

• No backtracking or look-ahead beyond current line
• Streaming-friendly for large documents
• Predictable memory usage
• Linear time complexity

Error Handling

• Graceful degradation — Continue parsing on errors
• Context preservation — Maintain parser state
• Detailed reporting — Line/column information
• Recovery strategies — Skip invalid content

Testing Strategy

pnpm test

Comprehensive test suite covering:

• Syntax parsing and tokenization
• Context management and prefix folding
• Polarity system and retractions
• Elevated statements detection
• Primary metadata extraction
• Round-trip parse/generate cycles
• Origin tracking and provenance