Custom Languages

A language definition maps source ranges to the library's semantic token classes.

Minimal definition

import {
  createHighlighter,
  defineLanguage,
  type TokenRange,
} from '@tanstack/highlight/core'

const taskList = defineLanguage({
  name: 'task-list',
  aliases: ['tasks'],
  tokenize(code) {
    const ranges: Array<TokenRange> = []
    const pattern = /\b(?:TODO|DONE|BLOCKED)\b/g
    let match: RegExpExecArray | null

    while ((match = pattern.exec(code))) {
      ranges.push({
        start: match.index,
        end: match.index + match[0].length,
        className: match[0] === 'DONE' ? 'inserted' : 'keyword',
      })
    }

    return ranges
  },
})

const highlighter = createHighlighter({
  languages: [taskList],
})

defineLanguage() is an identity helper that preserves the literal name type.

Range contract

Each TokenRange contains:

type TokenRange = {
  start: number
  end: number
  className: HighlightTokenClass
}

Ranges use zero-based, end-exclusive UTF-16 offsets. Return non-overlapping ranges. The core clamps and sorts ranges by source position, then drops invalid or overlapping entries before creating tokens. A broad range that starts first can therefore suppress a more specific range inside it.

Prefer leaving text unclassified over assigning an inaccurate class. Gaps are preserved as plain text automatically.

Delegation

The tokenizer receives a TokenizerContext:

type TokenizerContext = {
  hasLanguage(lang: string): boolean
  tokenize(code: string, lang: string): Array<TokenRange>
}

An outer definition can delegate a substring and offset the returned ranges into its own source coordinates. Check hasLanguage() first so the dependency remains optional.

Avoid making a custom definition import another shipped definition directly. Registry delegation preserves selective imports and lets the application decide dependencies.