chunking¶
Split a large document into smaller pages or JSON data files for static site generators and web components.
opm.chunking ¶
Document chunking
Splits large TEI documents into smaller HTML pages with metadata and fragments for static site generation.
ChunkProcessor ¶
ChunkProcessor(
module_path: Path,
xml_root: _Element,
config: ChunkingConfig,
project_root: Path,
project_config: ProjectConfig | None = None,
webcomponents: bool = False,
xpath_env: XPathEnvironment | None = None,
source_dir: Path | None = None,
documents: Collection[str] | None = None,
document: str | None = None,
anchors: Mapping[str, str] | None = None,
merge_manifest: bool = False,
footnote_order: str | Callable | None = None,
)
Splits one document into chunks and writes them in one output format.
Most callers want opm.project.Project.chunk or
chunk_document, which build one of these. Use it directly to
select chunks (select_chunks) or read their metadata without
writing anything.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
module_path
|
Path
|
The compiled transform module for chunk content. |
required |
xml_root
|
_Element
|
Root element of the parsed document. |
required |
config
|
ChunkingConfig
|
The |
required |
project_root
|
Path
|
The directory |
required |
project_config
|
ProjectConfig | None
|
The project settings (parameters, template context). |
None
|
webcomponents
|
bool
|
Enable web-component mode. |
False
|
xpath_env
|
XPathEnvironment | None
|
The XPath environment to evaluate in; see
|
None
|
source_dir
|
Path | None
|
Directory of the source file, for copying images. |
None
|
documents
|
Collection[str] | None
|
Names of every document in the run, for the templates. |
None
|
document
|
str | None
|
Name of this document's source file. |
None
|
Source code in src/opm/chunking.py
113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 | |
odd_name
property
¶
ODD name to advertise to pb-view.
Read from the ODD_NAME baked into the generated module, i.e. the ODD
it was actually compiled from, so the index and CSS reference the right
ODD. pb-view sends odd=<name>.odd and loads css/<name>.css.
select_chunks ¶
Find chunk elements.
If config.selector is set to a dotted Python path it is imported
and called as selector(root, config). Otherwise the xpath
expression from the chunking config is evaluated.
Selectors that rebuild a region as a detached tree record copy → source
in opm.runtime.source_map while they build, which is what lets
$get() in an ODD step back to the stored document. The map holds
both trees alive, so it is reset here — once per document, before the
selector runs.
Source code in src/opm/chunking.py
shared_root ¶
Return the directory holding output shared across documents.
Each document gets its own subdirectory, so stylesheets and assets
belong one level up, beside the collection index — one copy for the
whole edition. Without link_doc the output directory is itself the
root.
Source code in src/opm/chunking.py
url_prefix ¶
Return the relative path from a chunk page back to shared_root.
write_shared_files ¶
Write stylesheets and copy assets into shared_root.
Chunking always produces several pages sharing one stylesheet, so the
stylesheets are always written as files — the same thing
--format pb-view has always done. Templates receive
odd_css_url pointing at it, alongside the odd_css string, which
stays available so a template that inlines it keeps working. Stylesheets
among config.assets are listed in asset_styles, in declared
order. index_url points back at the collection index written at
shared_root (see build_index) — empty when the caller sets
no link_doc, as there is then no such page to link to.
With html, the stylesheet of the packaged page layout is written to
css/opm-layout.css too, and layout_css_url points at it —
whether or not the page template extends opm/layout.html.j2. A
project copy at opm/opm-layout.css under templates_dir (the page
template's directory) replaces the packaged one.
Safe to call once per document — the writes are idempotent.
Source code in src/opm/chunking.py
copy_referenced_images ¶
Copy the local images html references into output_dir.
An img/@src is written relative to the source document, and the
chunk pages sit flat in the output directory, so each image goes to the
same relative path there. Files are looked up the way EPUB output does:
next to the source document, then in a sibling images/ directory.
Remote and root-relative URLs are left alone, as are paths that would
land outside the output directory and images that cannot be found.
Source code in src/opm/chunking.py
entry_file ¶
Return the filename of this document's first chunk (its entry point).
That is the name file_pattern gives the first chunk once chunks are
selected, and 001.html before then.
Source code in src/opm/chunking.py
entry_href ¶
Return the link to this document's entry point, relative to the output root.
config.link_doc names the per-document subdirectory, giving
quickstart.xml/001.html; without it there is no prefix.
Source code in src/opm/chunking.py
generate_chunk_metadata ¶
Generate metadata for a chunk.
Source code in src/opm/chunking.py
build_anchor_index ¶
Map source xml:id (or plain @id) values to the chunk file that owns them.
JATS and other no-namespace vocabularies identify elements with @id,
so both are indexed; xml:id wins where an element carries both.
Source code in src/opm/chunking.py
process_fragment ¶
process_fragment(
fragment: FragmentConfig,
context_node: _Element | None = None,
_cache: dict[tuple, str] | None = None,
chunk_index: int | None = None,
) -> str
Process a fragment transform.
The xpath is evaluated with the chunk as context node, so a
chunk-relative expression (../../text[@type='translation']/div)
picks that chunk's counterpart. An absolute expression that yields one
element per chunk (//body/div[@xml:lang='en']) is instead aligned by
position — the parallel-column case — using chunk_index. Both formats
pass the index, so --format pb-view and the HTML output resolve
fragments identically.
Source code in src/opm/chunking.py
650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 | |
process_chunk ¶
Process a single chunk with its fragments.
Source code in src/opm/chunking.py
process_global_fragments ¶
Process global fragments once.
Source code in src/opm/chunking.py
render_chunk_template ¶
render_chunk_template(
chunk_result: ChunkResult,
global_fragments: dict[str, str],
template_path: Path | None = None,
) -> str
Render a chunk with its template context.
Source code in src/opm/chunking.py
generate_manifest ¶
Generate JSON manifest for static site builders.
Source code in src/opm/chunking.py
process_all ¶
process_all(
template_path: Path | None = None,
on_progress: Callable[[int, int], None] | None = None,
output_format: str = "html",
) -> None
Process all chunks and save files.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
template_path
|
Path | None
|
Optional Jinja2 template path (ignored for |
None
|
on_progress
|
Callable[[int, int], None] | None
|
Optional callback |
None
|
output_format
|
str
|
|
'html'
|
Source code in src/opm/chunking.py
918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 | |
export_pb_view ¶
export_pb_view(
doc_path: str | None = None,
on_progress: Callable[[int, int], None] | None = None,
) -> None
Export chunks as data consumable by the pb-view web component.
Lays the output out the way pb-view resolves it in static mode, where
the data location is derived from the static root and the document
path (${static}/${path}/...):
<output_dir>/<doc_path>/index.json— lookup table mapping pb-view's computed parameter keys to part files<output_dir>/<doc_path>/<xml:id>.json— one part per chunk, mirroring the response of TEI Publisher's/api/parts/<doc>/jsonendpoint<output_dir>/<doc_path>/<name>.json— one part per global fragment (e.g.toc.json), keyed by the fragment xpath anduser.*params; a sibling<name>.htmlcarries the same content as well-formed XML, for consumers that store it in an XML database (see_wellformed_fragment_xml)<output_dir>/<doc_path>/<name>-<xml:id>.json— per-chunk fragments<output_dir>/css/<odd>.css— stylesheet, shared by every document under the same static root
doc_path should match the path of the consuming pb-document.
When omitted the data is written directly into <output_dir> (single
document at the static root).
Chunks carrying an xml:id are addressed by it; chunks without one get
a stable synthetic id. pb-view navigates via nextId/previousId
(xml:ids) when present; next/previous are kept truthy for
navigate() but root is left null so subscribed dynamic views do
not call the parts API with root=<xml:id>.
on_progress is an optional callback (current, total) invoked after
each chunk is written.
Source code in src/opm/chunking.py
1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 | |
IndexEntry
dataclass
¶
chunk_document ¶
chunk_document(
module_path: Path | None,
xml_path: Path,
config: ChunkingConfig,
project_root: Path,
template_path: Path | None = None,
on_progress: Callable[[int, int], None] | None = None,
project_config: ProjectConfig | None = None,
webcomponents: bool = False,
xpath_extensions: tuple[str, ...] | None = None,
output_format: str = "html",
doc_path: str | None = None,
documents: Collection[str] | None = None,
anchors: Mapping[str, str] | None = None,
merge_manifest: bool = False,
footnote_order: str | Callable | None = None,
) -> dict[str, str]
Chunk a document using the specified configuration.
The page template receives document (config.link_doc, or
xml_path's name if that is unset) and documents, the names of every
document in the run — file names such as serafin01.xml, or whatever
[chunking] document_name assigned. For a directory run, pass the same
documents set for each file; by default it is just that one name.
ODDs in config (the main one and the fragments') that have no compiled
module yet are compiled here. opm.project.Project.chunk handles
a whole directory the way opm chunk does.
footnote_order names the callable ordering each chunk's footnotes before
they are placed, overriding [transform] footnote_order; see
inject_cached_footnotes.
Returns the run's anchor map, xml:id → chunk file. Pass it as
anchors to a later run over the same output directory, and #id
links there into this run's pages resolve as if both were one run. Pass
merge_manifest for that later run, so manifest.json lists both.
Source code in src/opm/chunking.py
1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 | |
collect_index_entries ¶
collect_index_entries(output_dir: Path) -> list[IndexEntry]
Collect one IndexEntry per chunked document under output_dir.
Reads the manifest.json each document run writes, so this works on any
existing output directory without re-chunking. Directories without a
readable manifest are skipped.
Source code in src/opm/chunking.py
build_index ¶
build_index(
output_dir: Path,
*,
template_path: Path | None = None,
title: str | None = None,
odd_css: str | None = None,
module_path: Path | None = None,
project_config: ProjectConfig | None = None,
project_root: Path | None = None,
chunking_config: ChunkingConfig | None = None,
webcomponents: bool = False,
) -> Path | None
Render <output_dir>/index.html listing every chunked document.
http.server serves index.html in preference to a directory listing,
so writing this file is all that is needed for opm serve to show a real
landing page.
project_config also supplies the template context, so the index and
the chunk pages read the same [context] values. Pass webcomponents to
match the run's effective mode, so the index page is given
webcomponents_url exactly when the chunk pages are.
Pass module_path (and optionally project_config) to have the ODD's
generated CSS and the project stylesheet resolved the same way chunk pages
resolve them, so an index template can style a browse record's tei-*
classes exactly as the document pages do. An explicit odd_css wins.
Returns the path written, or None when output_dir holds no chunked documents.
Source code in src/opm/chunking.py
1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 | |
build_index_json ¶
Write <output_dir>/index.json listing every chunked document.
The JSON counterpart of build_index. A directory run splits its
documents into one subdirectory each, and nothing at the root says what they
are or what order they belong in — a static site generator would have to
rediscover that by scanning. This writes it once, from the same
collect_index_entries the HTML index is built from, so both agree.
Returns the path written, or None when output_dir holds no chunked documents.
Source code in src/opm/chunking.py
Chunk selectors¶
The functions you can name under [chunking] selector.
opm.navigation ¶
Built-in chunk-selection algorithms for TEI, DocBook and JATS documents. Those determine how a large XML document is split into smaller sections to be viewed on one page.
Functions here can be referenced in opm.toml via the
chunking.selector key, e.g.:
[chunking]
selector = "opm.navigation.tei_div_chunks"
depth = 2
# Or page-break milestones (Shakespeare plays, facsimiles, …):
# selector = "opm.navigation.tei_pb_chunks"
Each selector callable has the signature:
load_selector ¶
tei_div_chunks ¶
tei_div_chunks(
root: _Element, config: ChunkingConfig
) -> list[_Element]
Return chunk elements by walking TEI div divisions up to config.depth.
Mirrors nav:next-page / nav:fill in TEI Publisher's
navigation-tei.xql (pb-pagination excluded):
- Walk every
divundertext/bodyin document order whose nesting level (count(ancestor-or-self::div)) is at most depth. - A div with no child
divwithin the depth limit is a leaf chunk and is included as-is. - A div whose child divs are separate chunks is skipped, unless it has
content before its first child div (a heading, an opening section).
That content becomes a chunk of its own, placed before the children: a
detached copy of the div holding only the intro, recorded in
opm.runtime.source_mapso$get()still reaches the stored document. Whitespace, comments and empty milestones such aspbalone do not count as intro.
With depth = 1 the result is identical to //text/body/div.
Source code in src/opm/navigation.py
tei_pb_chunks ¶
tei_pb_chunks(
root: _Element, config: ChunkingConfig
) -> list[_Element]
Return one reconstructed page tree per TEI pb milestone.
Ports nav:get-content for element(tei:pb) and
nav:milestone-chunk from navigation-tei.xql (view="page"):
- Each
pbin document order starts a page. - Content runs from that
pbup to (but not including) the nextpb. - The page is carved from the deepest common ancestor with the next
pb, or — on the last page — from the nearestdiv/text(or the wholetextwhen the document has only onepb). - The returned element is a synthetic tree preserving intervening structure
(
sp,p, nesteddiv, …), not the barepb.
Chunk roots are stamped with xml:id from the pb when present,
otherwise page-{@n} or page-{index}, so pb-view / file naming have a
stable handle even though Folio pb elements often lack xml:id.
Source code in src/opm/navigation.py
80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 | |
dbk_section_chunks ¶
dbk_section_chunks(
root: _Element, config: ChunkingConfig
) -> list[_Element]
Return chunk elements by walking DocBook section elements up to config.depth.
Mirrors nav:next-page / nav:fill in navigation-dbk.xql:
- Sections with no child sections within the depth limit are leaf chunks and are included as-is.
- Sections whose child sections are themselves separate chunks are skipped
— UNLESS they have content (elements or text) before their first child
section. In that case a shallow copy of the parent section containing
only that pre-subsection content is inserted as an extra chunk immediately
before the child-section chunks. This mirrors the
nav:filllogic (without the fill-size threshold).
Source code in src/opm/navigation.py
jats_sec_chunks ¶
jats_sec_chunks(
root: _Element, config: ChunkingConfig
) -> list[_Element]
Return chunk elements for a JATS article: front, then sec, then back.
sec nests exactly like DocBook section, so the same depth walk and
intro-carving applies (see dbk_section_chunks). A journal article
keeps its scholarly apparatus outside body, so the front matter (title,
contributors, abstract) becomes the landing chunk and back (reference
list, appendices) the closing one — otherwise chunking would drop both.
JATS has no namespace, so the element names are unprefixed.