Back
Source code being split into tokens

Making a Language: The Lexer

Building the lexer for Spark — turning source code into tokens with line/column tracking, comments, and escape sequences.

May 18, 2026#typescript#compilers

In the previous part, we laid out Spark's syntax and type system. Now we build the first stage of the pipeline: the lexer.

The lexer reads source code character by character and groups them into tokens. Think of it like looking at a sentence and identifying each word and punctuation mark without caring about what the sentence means. It produces a flat list that the parser will later give structure to.

Keywords

lexer.ts
1const KEYWORDS: Record<string, TokenType> = {
2  val: TokenType.Let,
3  when: TokenType.If,
4  else: TokenType.Else,
5  func: TokenType.Fn,
6  return: TokenType.Return,
7  yes: TokenType.True,
8  no: TokenType.False,
9};

A simple lookup table maps Spark's keyword strings to their token types. When the lexer reads an identifier, it checks this map. If the word is a keyword, it emits the corresponding type instead of a generic Identifier token. This is how val becomes a Let token while x stays an Identifier.

Position Tracking

lexer.ts
1function pos(): { line: number; col: number } {
2  return { line, col };
3}
4
5function advance(n = 1) {
6  for (let j = 0; j < n; j++) {
7    if (source[i] === "\n") {
8      line++;
9      col = 1;
10    } else {
11      col++;
12    }
13    i++;
14  }
15}

Every token needs to know where it came from. When something goes wrong during parsing or evaluation, we point to the exact line and column. The advance function moves the index forward one character at a time, incrementing the line counter and resetting the column whenever it hits a newline.

Whitespace and Comments

lexer.ts
1if (ch === " " || ch === "\t" || ch === "\r") {
2  advance();
3  continue;
4}
5
6if (ch === "\n") {
7  advance();
8  continue;
9}

Whitespace and newlines are discarded immediately. They have no meaning in Spark. The lexer just skips them and moves on.

lexer.ts
1if (ch === "/" && source[i + 1] === "/") {
2  while (i < source.length && source[i] !== "\n") advance();
3  continue;
4}
5
6if (ch === "/" && source[i + 1] === "*") {
7  advance(2);
8  while (i < source.length && !(source[i] === "*" && source[i + 1] === "/"))
9    advance();
10  advance(2);
11  continue;
12}

Two kinds of comments: single-line // which skips until a newline, and block /* */ which skips everything until the closing delimiter. Block comments handle multiple lines correctly because advance updates the line counter automatically.

Numbers

lexer.ts
1if (isDigit(ch)) {
2  let num = "";
3  while (i < source.length && (isDigit(source[i]) || source[i] === ".")) {
4    num += source[i];
5    advance();
6  }
7  tokens.push({ type: TokenType.Number, value: num, ...start });
8  continue;
9}

Number literals consume consecutive digits and decimal points. The raw string is stored as the token's value. The parser will handle converting it to an actual numeric type.

Strings

lexer.ts
1if (ch === '"') {
2  advance();
3  let str = "";
4  while (i < source.length && source[i] !== '"') {
5    if (source[i] === "\\" && i + 1 < source.length) {
6      const next = source[i + 1];
7      if (next === "n") str += "\n";
8      else if (next === "t") str += "\t";
9      else if (next === '"') str += '"';
10      else if (next === "\\") str += "\\";
11      else str += next;
12      advance(2);
13    } else {
14      str += source[i];
15      advance();
16    }
17  }
18  advance(); // skip closing "
19  tokens.push({ type: TokenType.String, value: str, ...start });
20  continue;
21}

String literals accumulate characters between double quotes. Escape sequences (\n, \t, \", \\) are recognised and replaced with their actual character values. The closing quote is consumed but not added to the string value.

Identifiers and Keywords

lexer.ts
1if (isAlpha(ch) || ch === "_") {
2  let ident = "";
3  while (
4    i < source.length &&
5    (isAlphaNumeric(source[i]) || source[i] === "_")
6  ) {
7    ident += source[i];
8    advance();
9  }
10  const type = KEYWORDS[ident];
11  tokens.push({
12    type: type ?? TokenType.Identifier,
13    value: ident,
14    ...start,
15  });
16  continue;
17}

Identifiers start with a letter or underscore and consume alphanumeric characters. After reading the full word, the lexer checks the keywords map. If the word is a keyword like when or func, the token gets that keyword's type. Otherwise it's a regular Identifier.

Operators

lexer.ts
1if (ch === "=" && source[i + 1] === "=") {
2  tokens.push({ type: TokenType.EqEq, value: "==", ...start });
3  advance(2);
4  continue;
5}
6if (ch === "!" && source[i + 1] === "=") {
7  tokens.push({ type: TokenType.BangEq, value: "!=", ...start });
8  advance(2);
9  continue;
10}
11if (ch === "<" && source[i + 1] === "=") {
12  tokens.push({ type: TokenType.LtEq, value: "<=", ...start });
13  advance(2);
14  continue;
15}
16if (ch === ">" && source[i + 1] === "=") {
17  tokens.push({ type: TokenType.GtEq, value: ">=", ...start });
18  advance(2);
19  continue;
20}
21if (ch === "=" && source[i + 1] === ">") {
22  tokens.push({ type: TokenType.Arrow, value: "=>", ...start });
23  advance(2);
24  continue;
25}
26if (ch === "." && source[i + 1] === ".") {
27  tokens.push({ type: TokenType.DotDot, value: "..", ...start });
28  advance(2);
29  continue;
30}

Two-character operators are checked before single-character ones. This matters because = alone is an assignment, but == is equality. The lexer peeks at the next character and if it matches a known pair, consumes both at once with advance(2).

lexer.ts
1const singleMap: Record<string, TokenType> = {
2  "=": TokenType.Eq,
3  "+": TokenType.Plus,
4  "-": TokenType.Minus,
5  "*": TokenType.Star,
6  "/": TokenType.Slash,
7  "<": TokenType.Lt,
8  ">": TokenType.Gt,
9  "(": TokenType.LParen,
10  ")": TokenType.RParen,
11  "{": TokenType.LBrace,
12  "}": TokenType.RBrace,
13  "[": TokenType.LBracket,
14  "]": TokenType.RBracket,
15  ",": TokenType.Comma,
16};
17
18const mapped = singleMap[ch];
19if (mapped !== undefined) {
20  tokens.push({ type: mapped, value: ch, ...start });
21  advance();
22  continue;
23}

Single-character operators and delimiters use another lookup map. Brackets, parentheses, arithmetic operators, and comma are all handled here. If a character doesn't match anything, it's silently skipped.

Helper Functions

lexer.ts
1function isDigit(ch: string): boolean {
2  return ch >= "0" && ch <= "9";
3}
4
5function isAlpha(ch: string): boolean {
6  return (ch >= "a" && ch <= "z") || (ch >= "A" && ch <= "Z");
7}
8
9function isAlphaNumeric(ch: string): boolean {
10  return isDigit(ch) || isAlpha(ch);
11}

Three small helpers that test character ranges. No regex, no external dependencies, just ASCII range checks. isAlphaNumeric is used by the identifier reader, isDigit by the number reader.

Example

Running tokenize("val x = 10") produces:

1[
2  { type: Let,        value: "val", line: 1, col: 1 },
3  { type: Identifier, value: "x",   line: 1, col: 5 },
4  { type: Eq,         value: "=",   line: 1, col: 7 },
5  { type: Number,     value: "10",  line: 1, col: 9 },
6  { type: EOF,        value: "",    line: 1, col: 11 },
7]

The lexer has no idea what a variable is. It just recognizes patterns and passes along a flat list of tokens. The parser turns that list into meaning.

In the next part, we'll build the parser, the component that turns this flat token list into a structured tree.