Blog with Preprocessors

When creating this website, I had a few DX requirements for the articles:

• preserve all the tooling Svelte offers
• write tables in Markdown
• style dark code blocks with my personal VS Code theme
• add new features (like Math) as needed
• transform everything on the server
• write the code as if it were on the client
• not have to wait for compatibility when Svelte 5 is released

Evaluating the Options

0.1: Unified

Unified is an ecosystem of packages that all accept a standardized abstract syntax tree (AST). The uniformity means you can convert your language into an AST, run a pipeline of transformations, and then transform it into some other language. Originally, I thought of using the unified ecosystem to convert Markdown to an AST, use rehypePrettyCode in the middle for styling, and then convert it to HTML. This is a great solution for technical and non-technical users alike because of how easy Markdown is to write.

Something like this would work:

// https://github.com/syntax-tree/unist#list-of-utilities unified AST utilities
// https://github.com/syntax-tree/mdast#list-of-utilities markdown AST utilities
// https://github.com/syntax-tree/hast#list-of-utilities html AST utilities
// https://github.com/remarkjs/remark/blob/main/doc/plugins.md remark: markdown AST plugins
// https://github.com/rehypejs/rehype/blob/main/doc/plugins.md rehype: html AST plugins

import matter from 'gray-matter';
import { rehypeAddCopyBtnToCodeTitle, remarkReadingTime, type RemarkReadingTimeData } from './plugins.js';
import fs from 'fs/promises';
import path from 'path';
import rehypeCodeTitles from 'rehype-code-titles';
import { default as rehypePrettyCode, type Theme } from 'rehype-pretty-code';
import rehypeSlug from 'rehype-slug';
import rehypeStringify from 'rehype-stringify';
import remarkGfm from 'remark-gfm';
import remarkParse from 'remark-parse';
import remarkRehype from 'remark-rehype';
import remarkSmartypants from 'remark-smartypants';
import remarkTableOfContents from 'remark-toc';
import { unified } from 'unified';
import { darker } from './code-themes/darker.js';
import type { Parsed } from './types.js';

type PluginData = RemarkReadingTimeData;

/** @throws Error */
export const mdToHTML = async <T extends object>(markdown: string) => {
	const split = matter(markdown);
	const rawMd = split.content;
	const frontMatter = split.data as T;

	const result = await unified()
		.use(remarkParse) // Convert Markdown string to Markdown AST
		.use(remarkGfm) // Use GitHub flavored Markdown
		.use(remarkSmartypants) // Convert ASCII to Unicode punctuation: “ ” – — …
		.use([[remarkTableOfContents, { tight: true }]]) // Generate TOC list from headings (tight removes <p> from <li> when nested)
		.use(remarkReadingTime) // Add reading time to result.data
		.use(remarkRehype) // Convert Markdown AST to HTML AST
		.use(rehypeSlug) // Add IDs to headings
		.use(rehypeCodeTitles) // Add titles to code blocks
		.use(rehypePrettyCode, { theme: { light: 'rose-pine-dawn', dark: darker as unknown as Theme } }) // Add code syntax, line/word highlighting, line numbers
		.use(rehypeAddCopyBtnToCodeTitle) // Add copy button to code blocks
		.use(rehypeStringify) // Convert HTML AST to HTML string
		.process(rawMd);

	const pluginData = result.data as PluginData;
	const readingTime = Math.ceil(pluginData.readingTime.minutes);

	return {
		rawHTML: result.value as string,
		data: { ...frontMatter, readingTime },
	};
};

/** @throws Error if the file cannot be read or parsed */
export async function parsePath<T extends object>(mdPath: { slug: string; path: string }): Promise<Parsed<T>> {
	const rawMdContent = await fs.readFile(mdPath.path, 'utf-8').catch((_err) => {
		throw new Error(`Unable to readFile ${mdPath.path}`);
	});
	return await mdToHTML<T>(rawMdContent).catch((_err) => {
		throw new Error(`Unable to parse ${mdPath.slug}`);
	});
}

/** @throws fs.readdir Error */
export async function parseDir<T extends object>(inDir: string): Promise<Parsed<T>[]> {
	const mdPaths = (await fs.readdir(inDir, { withFileTypes: true })).map((dirent) => ({
		slug: dirent.name,
		path: path.join(dirent.path, dirent.name, `${dirent.name}.md`),
	}));

	const res = await Promise.all(
		mdPaths.map(async (p) => {
			try {
				return await parsePath<T>(p);
			} catch (err) {
				console.error(`\n${(err as Error).message}`);
				return null;
			}
		}),
	);
	return res.filter((r) => r !== null) as Parsed<T>[];
}

The largest drawback is that we'd be writing Markdown and forgoing the Svelte ecosystem entirely. Furthermore, even if that were acceptable, this makes inlining Svelte components cumbersome. Would we write multiple .md files, import/process them separately, and embed them into .svelte components? Would we write Markdown, and inject Svelte components in-between?

0.2: MDsveX

MDsveX – a Markdown preprocessor for Svelte – solves the embedded Svelte problem. Preprocessors transform the input files before passing them to the Svelte compiler, and this particular preprocessor enables writing Markdown and Svelte in the same file.

The biggest concern at the time of my decision was that it hadn't been updated in 7 months with over 150+ open issues. It didn't inspire confidence that it would support Svelte 5 upon its release, which was a large consideration in which method to use. Ideally, the technique we use would be resistant to changes in Svelte. The second reason was that MDsveX is a separate language, and because of that, it wouldn't just work with other Svelte tooling.

I really only want Markdown for three things: tables, code blocks, and math. The general idea of a preprocessor, however, is very appealing.

0.3: Preprocessors

We can easily add a new feature to Svelte by writing a simple preprocessor wrapping a dedicated package for each new functionality. There's Marked for Markdown, Shiki for code highlighting, and KaTeX for math.

<!--  -->

<!-- shiki-start
const foo = "bar";
shiki-end -->

<!-- md-start
# Markdown
md-end -->

<!--\[ V={4 \over 3}\pi r^{3} \] -->

Writing the Preprocessors

Markdown

We'll start with the Markdown preprocessor because it does nothing but wrap Marked. Writing a preprocessor that wraps an existing package is dead simple. We just have to remove the code between our delimiters, process it, and put it back in. There are two obvious choices for how to pull out the code. We can either loop over the raw string content with indexOf or we can walk a Svelte AST tree. Here are two examples of how we can implement the preprocessor.

import { walk } from 'estree-walker';
import MagicString from 'magic-string';
import { marked as hostedMarked } from 'marked';
import { parse, type PreprocessorGroup } from 'svelte/compiler';
import type { Logger } from './logger.js';

const delimiter = { start: 'md-start', end: 'md-end' };
const delimLoc = { start: delimiter.start.length + 1, end: -delimiter.end.length - 1 };

export function processMarkdown({
	include,
	logger,
	marked = hostedMarked,
}: {
	include?: (filename: string) => boolean;
	logger?: Logger;
	marked?: typeof hostedMarked;
} = {}) {
	return {
		name: 'md',
		markup({ content, filename }) {
			if (!filename) return;
			if (include && !include(filename)) return;

			try {
				const s = new MagicString(content);
				const ast = parse(content, { filename, modern: true });
				let count = 0;

				walk(ast.fragment, {
					enter(node: (typeof ast.fragment.nodes)[number]) {
						if (node.type !== 'Comment') return;
						const trimmed = node.data.trim();
						if (!trimmed.startsWith(delimiter.start)) return;
						if (!trimmed.endsWith(delimiter.end)) return;
						s.remove(node.start, node.end);
						s.appendLeft(node.start, marked(trimmed.slice(delimLoc.start, delimLoc.end), { async: false }) as string);
						count++;
					},
				});

				if (count) logger?.info?.({ count }, filename);
				return { code: s.toString() };
			} catch (err) {
				if (err instanceof Error) logger?.error?.(err, filename);
				else logger?.error?.(Error('Failed to render Markdown.'), filename);
			}
		},
	} satisfies PreprocessorGroup;
}

import { marked as hostedMarked } from 'marked';
import type { Logger } from './logger.js';
import type { PreprocessorGroup } from 'svelte/compiler';

const delimiter = { start: 'md-start', end: 'md-end' };

/**
 * @throws {Error} If the comment block is incomplete.
 */
const replaceComments = (content: string, replacer: (comment: string) => null | string): string => {
	let resultContent = content;

	let startIdx = resultContent.indexOf('<!--');

	while (startIdx !== -1) {
		const endIdx = resultContent.indexOf('-->', startIdx + 4);
		if (endIdx === -1) throw new Error(`Incomplete comment block at start ${startIdx}. Aborting.`);

		const extractedContent = resultContent.substring(startIdx + 4, endIdx);

		const replaced = replacer(extractedContent);
		if (replaced === null) {
			startIdx = resultContent.indexOf('<!--', endIdx + 3);
			continue;
		}
		const before = resultContent.substring(0, startIdx);
		const after = resultContent.substring(endIdx + 3);

		resultContent = before + replaced + after;
		startIdx = resultContent.indexOf('<!--', startIdx + replaced.length);
	}

	return resultContent;
};

export function processMarkdown({
	include,
	logger,
	marked = hostedMarked,
}: {
	include?: (filename: string) => boolean;
	logger?: Logger;
	marked?: typeof hostedMarked;
} = {}) {
	return {
		name: 'md',
		markup({ content, filename }) {
			if (!filename) return;
			if (include && !include(filename)) return;

			try {
				let count = 0;

				const code = replaceComments(content, (comment) => {
					let trimmed = comment.trim();
					if (!trimmed.startsWith(delimiter.start)) return null;
					count++;
					if (!trimmed.endsWith(delimiter.end)) throw new Error(`Incomplete md (count: ${count}). Aborting.`);
					trimmed = trimmed.substring(delimiter.start.length + 1, trimmed.length - delimiter.end.length);
					return marked(trimmed, { async: false }) as string;
				});

				if (count) logger?.info?.({ count }, filename);
				return { code };
			} catch (err) {
				if (err instanceof Error) logger?.error?.(err, filename);
				else logger?.error?.(Error('Failed to render Markdown.'), filename);
			}
		},
	} satisfies PreprocessorGroup;
}

export type Logger = {
	error?: (e: Error, filename: string) => void;
	info?: (a: { count: number }, filename: string) => void;
};

export const createMdLogger = (formatFilename: (filename: string) => void = (filename: string) => filename): Logger => {
	return {
		error: (e, filename) => console.error(`[PREPROCESS] | Mark | Error | ${formatFilename(filename)} | ${e.message}`),
		info: (detail, filename) =>
			console.info(`[PREPROCESS] | Mark | Info | ${formatFilename(filename)} | count: ${detail.count}`),
	};
};

And we can now register it with Svelte.

import { processMarkdown, createMdLogger } from '@samplekit/preprocess-markdown';
import adapter from '@sveltejs/adapter-auto';
import { vitePreprocess } from '@sveltejs/vite-plugin-svelte';
import { opts } from './src/lib/shiki/index.js';

const preprocessorRoot = `${import.meta.dirname}/src/routes/`;
const formatFilename = (/** @type {string} */ filename) => filename.replace(preprocessorRoot, '');
const include = (/** @type {string} */ filename) => filename.startsWith(preprocessorRoot);

/** @type {import('@sveltejs/kit').Config} */
const config = {
	preprocess: [
		processMarkdown({
			include,
			logger: createMdLogger(formatFilename),
		}),
		vitePreprocess(),
	],
	kit: {
		adapter: adapter(),
	},
};

export default config;

Hopefully it's clear how powerful this simple idea is. We could write some complicated logic to parse a file and determine where a table might start or end, but that would necessarily require us to work in a different language without the Svelte ecosystem. In my opinion sacrificing Svelte tooling to use these preprocessors would be the wrong tradeoff.

Math

We'll do the same thing for Markdown, but in this version, we'll want two delimiters. One for inline math like this $V={4 over 3}pi r^{3}$ and one for display math like this:

It's very similar to the AST approach above.

import { walk } from 'estree-walker';
import katex from 'katex';
import MagicString from 'magic-string';
import { parse, type PreprocessorGroup } from 'svelte/compiler';
import type { Logger } from './logger.js';

const display = { start: String.raw`\[`, end: String.raw`\]` };
const inline = { start: String.raw`\(`, end: String.raw`\)` };
const delimLoc = { start: 3, end: -3 };

export function processKatex({
	include,
	logger,
	renderToString = katex.renderToString,
}: {
	include?: (filename: string) => boolean;
	logger?: Logger;
	renderToString?: (tex: string, options?: katex.KatexOptions) => string;
} = {}) {
	return {
		name: 'katex',
		markup({ content, filename }) {
			if (!filename) return;
			if (include && !include(filename)) return;
			const s = new MagicString(content);
			const ast = parse(content, { filename, modern: true });
			let count = 0;

			walk(ast.fragment, {
				enter(node: (typeof ast.fragment.nodes)[number]) {
					if (node.type !== 'Comment') return;
					const trimmed = node.data.trim();

					let displayMode: boolean | undefined = undefined;
					if (trimmed.startsWith(display.start) && trimmed.endsWith(display.end)) displayMode = true;
					else if (trimmed.startsWith(inline.start) && trimmed.endsWith(inline.end)) displayMode = false;
					if (displayMode === undefined) return;

					s.remove(node.start, node.end);

					let parsed;
					try {
						const rawInput = String.raw`${trimmed.slice(delimLoc.start, delimLoc.end)}`;
						parsed = renderToString(rawInput, {
							displayMode,
							throwOnError: true,
						});
					} catch (err) {
						logger?.error?.(err instanceof Error ? err : Error('Failed to render KaTeX.'), filename);
						return;
					}

					const content = displayMode
						? `<div class="overflow-x-auto">{@html \`${parsed}\`}</div>`
						: `{@html \`${parsed}\`}`;
					s.appendLeft(node.start, content);
					count++;
				},
			});

			if (count) logger?.info?.({ count }, filename);

			return { code: s.toString() };
		},
	} satisfies PreprocessorGroup;
}

Annoyingly, KaTeX logs directly to the console. To prevent this, let's wrap the call to KaTeX in a trap function.

import { walk } from 'estree-walker';
import katex from 'katex';
import MagicString from 'magic-string';
import { parse, type PreprocessorGroup } from 'svelte/compiler';
import type { Logger } from './logger.js';

const display = { start: String.raw`\[`, end: String.raw`\]` };
const inline = { start: String.raw`\(`, end: String.raw`\)` };
const delimLoc = { start: 3, end: -3 };

// https://github.com/KaTeX/KaTeX/issues/3720
const catchStdErr = ({ tmpWrite, trappedFn }: { trappedFn: () => void; tmpWrite: (str: string) => boolean }) => {
	const write = process.stdout.write;
	try {
		process.stderr.write = tmpWrite;
		trappedFn();
	} finally {
		process.stderr.write = write;
	}
};

export function processKatex({
	include,
	logger,
	renderToString = katex.renderToString,
}: {
	include?: (filename: string) => boolean;
	logger?: Logger;
	renderToString?: (tex: string, options?: katex.KatexOptions) => string;
} = {}) {
	return {
		name: 'katex',
		markup({ content, filename }) {
			if (!filename) return;
			if (include && !include(filename)) return;
			const s = new MagicString(content);
			const ast = parse(content, { filename, modern: true });
			let count = 0;

			walk(ast.fragment, {
				enter(node: (typeof ast.fragment.nodes)[number]) {
					if (node.type !== 'Comment') return;
					const trimmed = node.data.trim();

					let displayMode: boolean | undefined = undefined;
					if (trimmed.startsWith(display.start) && trimmed.endsWith(display.end)) displayMode = true;
					else if (trimmed.startsWith(inline.start) && trimmed.endsWith(inline.end)) displayMode = false;
					if (displayMode === undefined) return;

					s.remove(node.start, node.end);

					let parsed;
					try {
						const rawInput = String.raw`${trimmed.slice(delimLoc.start, delimLoc.end)}`;
						const warns: Error[] = [];
						catchStdErr({
							trappedFn: () => {
								parsed = renderToString(rawInput, {
									displayMode,
									throwOnError: true,
								});
							},
							tmpWrite: (str) => {
								if (!str.startsWith('No character metrics for ')) warns.push(Error(str));
								return true;
							},
						});
						if (logger?.warn) {
							warns.forEach((err) => logger.warn?.(err, filename));
						}
					} catch (err) {
						logger?.error?.(err instanceof Error ? err : Error('Failed to render KaTeX.'), filename);
						return;
					}

					const content = displayMode
						? `<div class="overflow-x-auto">{@html \`${parsed}\`}</div>`
						: `{@html \`${parsed}\`}`;
					s.appendLeft(node.start, content);
					count++;
				},
			});

			if (count) logger?.info?.({ count }, filename);

			return { code: s.toString() };
		},
	} satisfies PreprocessorGroup;
}

Looks good. But what about reactivity? It's quite possible you'll want to use handlebar substitution inside the equations. If we simply tried to use it as is, KaTeX would choke on the syntax. We'll have to make a special TeX like macro for Svelte. \s seems as good as any. It should take the Svelte content out, replace it with unique single character placeholders, process the TeX, and then put the Svelte content back in.

// the nuts and bolts of it
const { svelteFreeString, extractedSvelteContent } = replaceSvelteAndStore(rawInput);
const mathString = katex.renderToString(svelteFreeString)
const parsed = restoreSvelte(mathString, extractedSvelteContent);

We'll use some Unicode characters as our storage placeholder.

import { walk } from 'estree-walker';
import katex from 'katex';
import MagicString from 'magic-string';
import { parse, type PreprocessorGroup } from 'svelte/compiler';
import type { Logger } from './logger.js';

const display = { start: String.raw`\[`, end: String.raw`\]` };
const inline = { start: String.raw`\(`, end: String.raw`\)` };
const delimLoc = { start: 3, end: -3 };

// https://github.com/KaTeX/KaTeX/issues/3720
const catchStdErr = ({ tmpWrite, trappedFn }: { trappedFn: () => void; tmpWrite: (str: string) => boolean }) => {
	const write = process.stdout.write;
	try {
		process.stderr.write = tmpWrite;
		trappedFn();
	} finally {
		process.stderr.write = write;
	}
};

const unicodeInsertionPlaceholders = [
	'␇',
	'␈',
	'␉',
	'␊',
	'␋',
	'␌',
	'␍',
	'␎',
	'␏',
	'␐',
	'␑',
	'␒',
	'␓',
	'␔',
	'␕',
	'␖',
	'␗',
	'␘',
	'␙',
	'␚',
	'␛',
	'␜',
	'␝',
	'␞',
	'␟',
	'␠',
];

function replaceSvelteAndStore(input: string): { svelteFreeString: string; extractedSvelteContent: string[] } {
	const extractedSvelteContent: string[] = [];
	let index = 0;
	const svelteFreeString = input.replace(/\\s\{([^}]*)\}/g, (_match, p1) => {
		if (index >= unicodeInsertionPlaceholders.length) throw new Error('Too many variable substitutions.');
		extractedSvelteContent.push(p1);
		const unicodePlaceholder = unicodeInsertionPlaceholders[index];
		index++;
		return `{` + unicodePlaceholder + `}`;
	});

	return { svelteFreeString, extractedSvelteContent };
}

function restoreSvelte(mathString: string, extractedSvelteContent: string[]): string {
	if (!extractedSvelteContent.length) return mathString;

	const unicodeMap = new Map();
	extractedSvelteContent.forEach((content, i) => {
		unicodeMap.set(unicodeInsertionPlaceholders[i], content);
	});

	const unicodePlaceholderRegex = new RegExp(`(${unicodeInsertionPlaceholders.join('|')})`, 'g');

	return mathString.replaceAll(unicodePlaceholderRegex, (placeholder) => {
		const svelteContent = unicodeMap.get(placeholder);
		return `\${${svelteContent}}`;
	});
}

export function processKatex({
	include,
	logger,
	renderToString = katex.renderToString,
}: {
	include?: (filename: string) => boolean;
	logger?: Logger;
	renderToString?: (tex: string, options?: katex.KatexOptions) => string;
} = {}) {
	return {
		name: 'katex',
		markup({ content, filename }) {
			if (!filename) return;
			if (include && !include(filename)) return;
			const s = new MagicString(content);
			const ast = parse(content, { filename, modern: true });
			let count = 0;

			walk(ast.fragment, {
				enter(node: (typeof ast.fragment.nodes)[number]) {
					if (node.type !== 'Comment') return;
					const trimmed = node.data.trim();

					let displayMode: boolean | undefined = undefined;
					if (trimmed.startsWith(display.start) && trimmed.endsWith(display.end)) displayMode = true;
					else if (trimmed.startsWith(inline.start) && trimmed.endsWith(inline.end)) displayMode = false;
					if (displayMode === undefined) return;

					s.remove(node.start, node.end);

					let parsed;
					try {
						const rawInput = String.raw`${trimmed.slice(delimLoc.start, delimLoc.end)}`;
						const { svelteFreeString, extractedSvelteContent } = replaceSvelteAndStore(rawInput);
						const warns: Error[] = [];
						catchStdErr({
							trappedFn: () => {
								const mathString = renderToString(svelteFreeString, {
									displayMode,
									throwOnError: true,
									strict: (errorCode: string, errorMsg: string) => {
										if (errorCode === 'unknownSymbol' && errorMsg.startsWith('Unrecognized Unicode character'))
											return 'ignore';
									},
								});
								parsed = restoreSvelte(mathString, extractedSvelteContent);
							},
							tmpWrite: (str) => {
								if (!str.startsWith('No character metrics for ')) warns.push(Error(str));
								return true;
							},
						});
						if (logger?.warn) {
							warns.forEach((err) => logger.warn?.(err, filename));
						}
					} catch (err) {
						logger?.error?.(err instanceof Error ? err : Error('Failed to render KaTeX.'), filename);
						return;
					}
					const content = displayMode
						? `<div class="overflow-x-auto">{@html \`${parsed}\`}</div>`
						: `{@html \`${parsed}\`}`;
					s.appendLeft(node.start, content);
					count++;
				},
			});

			if (count) logger?.info?.({ count }, filename);

			return { code: s.toString() };
		},
	} satisfies PreprocessorGroup;
}