核心概念 · Unicode 17Core concepts · Unicode 17

文本、Bidi 与 IMEText, Bidi & IME

从字节、字素簇、双向段落到输入法合成：zenit 把文本当作系统能力，而不是一串按码点排开的 glyph。From bytes to grapheme clusters, bidirectional paragraphs and IME composition: zenit treats text as a system capability, not a row of glyphs laid out per code point.

预计阅读 12 分钟 · 含真实窗口录像12 min read · includes real window recordings

一条文本，四层职责One line, four layers

存储、编辑边界、方向解析和平台塑形各有唯一的 authority，最后统一成可命中、可选择的视觉行。任何一层都不替另一层做决定。Storage, edit boundaries, direction resolution and platform shaping each have exactly one authority, and together produce visual lines you can hit-test and select. No layer makes decisions on another’s behalf.

TEXT STACK

前两层在 src/text_core，双向算法在 src/i18n，塑形在 macOS 上交给 CoreText。The first two layers live in src/text_core, the bidi algorithm in src/i18n, and shaping on macOS is CoreText’s job.

层Layer负责Owns位置Where

UTF-8 文档坐标UTF-8 document coordinates稳定保存 byte offset，所有编辑入口做边界归一化Stable byte offsets, normalized to boundaries at every edit entrysrc/text_core

UAX #29 字素簇UAX #29 graphemes光标、删除、选区不会切开 ZWJ emoji、组合附加符或旗帜Caret, delete and selection never split ZWJ emoji, combining marks or flagssrc/text_core/grapheme.zig

UAX #9 BidiUAX #9 bidi段落方向、isolate、embedding、括号对与逻辑 / 视觉映射Paragraph direction, isolates, embeddings, bracket pairs, logical ↔ visual mappingsrc/i18n/bidi.zig

CoreText 塑形CoreText shaping字体 fallback、连字、glyph run、视觉放置与 caret xFont fallback, ligatures, glyph runs, visual placement and caret xsrc/render/text_renderer.zig

字素不是码点Graphemes, not code points

一个家庭 emoji 是 7 个 Unicode scalar（4 个人加 3 个 ZWJ），é 可以是 e 加一个组合重音，一面旗帜是两个区域指示符。用户眼里它们各是一个字符。zenit 用扩展字素簇作为光标移动、删除、选区和容量截断的原子。A family emoji is seven Unicode scalars (four people and three ZWJs), é can be e plus a combining acute, and a flag is two regional indicators. To the user each is one character. zenit uses extended grapheme clusters as the atom for caret movement, deletion, selection and capacity truncation.

UAX #29

cursor_pos 是 UTF-8 字节偏移；它只会停在簇边界上。单行 Input 的 2048 字节容量截断也不会留下半个字素。cursor_pos is a UTF-8 byte offset and only ever lands on a cluster boundary. Even truncation at the single-line Input’s 2048-byte capacity never leaves half a grapheme.

彩色 emoji 走 BGRA 图集页与着色器的彩色分支；灰度对照组保持灰色，混排行与普通文本同一管线。Color emoji go through a BGRA atlas page and the shader’s color branch; the grayscale control stays gray, and mixed runs share the normal text pipeline.

Alt+← / → 整词跳过希腊文与西里尔文单词，双击选中整词；家庭 emoji 始终是一个簇。Alt+← / → jumps whole Greek and Cyrillic words, double-click selects a word, and the family emoji stays one cluster.

双向文本Bidirectional text

文本在内存里按逻辑顺序存放，屏幕上按视觉顺序显示。UAX #9 先为每个字符解析嵌入层级，再按规则 L2 从最高层级开始逐级反转。下例是 LTR 段落中的 abc אבג 123：希伯来文得到层级 1，紧随其后的数字得到层级 2。Text is stored in logical order and shown in visual order. UAX #9 first resolves an embedding level for every character, then rule L2 reverses runs from the highest level down. Below, abc אבג 123 sits in an LTR paragraph: the Hebrew resolves to level 1 and the digits that follow it to level 2.

UAX #9

数字自身仍从左到右读，但整个 RTL 片段（含数字）作为一块从右到左排列。The digits still read left to right, but the whole RTL stretch — digits included — is laid out right to left as a block.

UAX #9 解析器是 zenit 自研的可移植实现，位于 src/i18n/bidi.zig，属性表由固定的 Unicode 17.0.0 数据生成。macOS 上最终的 glyph 塑形与 RTL 连字仍整段交给 CoreText，文本渲染器不会把 RTL 片段逐码点拆开。The UAX #9 resolver is zenit’s own portable implementation in src/i18n/bidi.zig, with property tables generated from pinned Unicode 17.0.0 data. On macOS, final glyph shaping and RTL joining are still handed to CoreText a whole run at a time — the text renderer never splits an RTL run per code point.

zenit Storybook showing mixed Arabic, Hebrew and Latin text — 真实 Storybook：阿拉伯文、希伯来文与 latin + arabic + latin 混排由同一条文本管线布局；虚拟鼠标停在 RTL story 上。The real Storybook: Arabic, Hebrew and latin + arabic + latin mixes laid out by one text pipeline; the virtual cursor rests on the RTL story.

RTL 与混排段落中的光标移动和选区。Caret movement and selection in RTL and mixed-direction paragraphs.

Unicode 17 一致性门禁Unicode 17 conformance

运行时属性表由仓库固定的 Unicode 17.0.0 数据生成；官方测试文件原样放在 vendor/unicode/17.0.0，并在 SHA256SUMS 中用 SHA-256 固定。这里的「支持」不是挑几个 emoji 写单元测试，而是跑完整的官方用例。Runtime property tables are generated from Unicode 17.0.0 data pinned in the repo; the official test files are vendored unmodified under vendor/unicode/17.0.0 and pinned by SHA-256 in SHA256SUMS. “Supported” here doesn’t mean a handful of emoji unit tests — it means running the complete official suites.

490,846BidiTest 类型序列BidiTest type sequences

770,241段落方向判定paragraph-direction evaluations

91,707BidiCharacterTest 序列BidiCharacterTest sequences

766GraphemeBreakTest 用例GraphemeBreakTest cases

Unicode conformance gates

python3 tools/generate_grapheme_data.py --check
python3 tools/generate_bidi_data.py --check
(cd vendor/unicode/17.0.0 && shasum -a 256 -c SHA256SUMS)
zig build test-text-core
zig build test-bidi-conformance

--check 确认生成的 Zig 表与数据一致；test-text-core 跑完 GraphemeBreakTest 全部用例；test-bidi-conformance 跑两个完整的官方 bidi 文件（到 UAX #9 规则 L2），它也包含在 zig build test-headless 中。--check confirms the generated Zig tables match the data; test-text-core runs every GraphemeBreakTest case; test-bidi-conformance runs both complete official bidi files (through UAX #9 rule L2) and is also part of zig build test-headless.

IME 是一等公民IME as a first-class citizen

IME 不是把最终汉字伪装成一次键盘输入。平台事件保留阶段语义：ime_preedit 携带合成文本和合成区内的光标偏移，ime_commit 携带最终文本。合成期间 Input 与 Textarea 渲染带下划线的 marked text，但 canonical buffer 只在提交时改变。没有单独的「取消」事件：空的 preedit 就表示取消合成。IME isn’t the final characters disguised as a keystroke. Platform events keep the phases: ime_preedit carries the composing text plus a caret offset inside it, and ime_commit carries the final text. While composing, Input and Textarea render underlined marked text, but the canonical buffer changes only on commit. There is no separate “discard” event — an empty preedit cancels the composition.

IME LIFECYCLE

preedit 与 commit 是两种事件，不是同一个字符串的中间快照。提交后短暂处于 commit_pending_end，用来吞掉平台随后可能重复送来的同一段文本，然后回到 idle。Preedit and commit are two kinds of event, not snapshots of one string. After a commit the input briefly sits in commit_pending_end to swallow a duplicate of the same text the platform may send next, then returns to idle.

一等公民体现在What first-class means具体契约Concrete contract

独立事件Distinct eventsime_preedit / ime_commit，不丢阶段信息；空 preedit 取消合成ime_preedit / ime_commit keep the phase; an empty preedit cancels

重新转换Reconversion两种事件都可带 replacement range（UTF-8 字节区间），把已提交文本拉回合成态或就地替换Both events may carry a replacement range (UTF-8 byte span) to pull committed text back into composition or replace it in place

视觉合成Visual compositionmarked range 下划线与高亮、合成光标、候选窗位置跟随视觉行Marked-range underline and highlight, composing caret, and candidate-window position follow the visual line

编辑安全Edit safety光标、删除、选区和容量截断都落在扩展字素边界Caret, delete, selection and capacity truncation all land on extended grapheme boundaries

可测试TestableHarness 可分别注入 preedit / commit，并回读 ime_phase、ime_preedit_len、buffer、cursor_pos、anchorThe harness injects preedit / commit separately and reads back ime_phase, ime_preedit_len, buffer, cursor_pos, anchor

真机门禁Real-system gatescripts/verify_ime.sh 用系统拼音 / 日文输入源走完整的 NSTextInputClient 路径scripts/verify_ime.sh drives the full NSTextInputClient path with the system Pinyin / Japanese input sources

Input showing underlined nihongo preedit text — Preedit：可见带下划线的 marked text 与 I-beam，canonical buffer 尚未改变。Preedit: underlined marked text and the I-beam are visible; the canonical buffer hasn’t changed.

Input showing committed 日本語 — Commit：日本語进入 canonical buffer，preedit 归零。Commit: 日本語 enters the canonical buffer and the preedit is cleared.

测试输入法Testing IME

E2E 客户端把两个阶段作为独立的 RPC 暴露出来，inputState 按 test id 回读输入框的真实状态。The E2E client exposes both phases as separate RPCs, and inputState reads an input’s real state back by test id.

ime.test.ts

import { imeCommit, imePreedit, inputState } from "./client";

// The input must already have focus (e.g. click it first).

await imePreedit("nihongo");
let s = await inputState("story.input.name");
// s.ime_phase === "composing", s.ime_preedit_len === 7, s.buffer === ""

await imeCommit("日本語");
s = await inputState("story.input.name");
// s.buffer === "日本語", s.ime_preedit_len === 0

await imePreedit("");   // empty preedit cancels an active composition

Harness 在真实 Retina 窗口里聚焦 Input，分别注入 preedit 与 commit，再切到 RTL 和 Checkbox；I-beam、手型与点击脉冲都由虚拟鼠标画进 drawable。The harness focuses an Input in a real Retina window, injects preedit and commit separately, then moves on to RTL and Checkbox; the I-beam, hand cursor and click pulse are drawn into the drawable by the virtual cursor.

TIP

测试注入与真输入法各司其职。Injection and real IMEs test different things. Harness 注入让阶段、buffer 与像素断言可重复；候选窗、输入源切换和 NSTextInputClient 集成仍由 scripts/verify_ime.sh 的真系统路径验收（需要 GUI 会话、辅助功能权限，运行期间约 8 秒不要碰键盘）。Harness injection makes phase, buffer and pixel assertions repeatable; the candidate window, input-source switching and NSTextInputClient integration are still verified through the real system path in scripts/verify_ime.sh (needs a GUI session and Accessibility permission; hands off the keyboard for about 8 seconds).

Harness 的完整用法见 E2E 自动化。See E2E harness for the full harness workflow.

← PREV

响应式系统Reactivity

样式与主题Styling & themes