Skip to content

Project

opm.Project provides a high-level API into opm. It loads an opm.toml and does what the commands do, taking the same settings from the config: transform, chunk, index and coverage. ODDs compile on demand into the user cache, as they do for the CLI.

from pathlib import Path

from opm import Project, collect_xpath_errors
from opm.indexing import write_jsonl
from opm.typst_compile import compile_pdf

project = Project.load('edition/opm.toml')

# One document in several formats. The ODD for each mode comes from
# [transform.<mode>] odd, then [transform] odd, then the packaged ODD.
html = project.transform('edition/data/letter.xml')
docx = project.transform('edition/data/letter.xml', mode='docx')       # bytes
toc = project.transform('edition/data/letter.xml', parameters={'mode': 'toc'})

# Typst output compiled to PDF: running typst is up to you.
typst = project.transform('edition/data/letter.xml', mode='typst')
pdf = compile_pdf(typst, root=Path('edition/data'))                    # needs typst

# Chunk a directory into JSON for a static site generator.
run = project.chunk('edition/data/letters', format='json', overwrite=True)
print(run.output_dir, [doc.name for doc in run.documents])

# Search-index records, and the ODD's coverage of the corpus.
write_jsonl(project.index('edition/data'), Path('index.jsonl'))
report = project.coverage('edition/data')

# See the XPath expressions that failed at run time.
with collect_xpath_errors() as log:
    project.transform('edition/data/letter.xml')
for failure in log.ordered_failures():
    print(f'{failure.count}× {failure.expression}: {failure.message}')

In a web service

A Project keeps the modules it has loaded and the register documents it has parsed, so load it once and reuse it. One instance can be shared between threads. It is a snapshot: after changing an ODD or a register, load a new one.

from lxml import etree
from opm import Project

project = Project.load('/srv/edition/opm.toml')

def render(path: str, view: str = 'div') -> str:
    return project.transform(f'/srv/edition/data/{path}', parameters={'view': view})

def render_fragment(root: etree._Element, xml_id: str) -> str:
    # An element parsed from a file still resolves doc() against that file.
    return project.transform(root, xpath=f'id("{xml_id}")')

Paths

Paths in opm.toml are relative to the file's directory. Relative paths passed to the methods are relative to the current directory, as usual in Python. The chunk output_dir is relative to Project.root, which is the config file's directory unless Project.load(..., root=...) says otherwise.

Settings in code

A project doesn't need a config file:

from pathlib import Path

from opm import Project, ProjectConfig

project = Project(ProjectConfig(parameters={'lang': 'en'}), root='build')
print_project = project.with_config(document_css=Path('print.css'))

The functions in opm.transform, opm.chunking and opm.indexing are the layer below Project, for callers that need finer control.

opm.project

The supported way to drive opm from Python: Project.

A specific project configuration and associated methods. Each method does what the command of the same name does, taking the same settings from the config:

from opm import Project

project = Project.load()                       # ./opm.toml
html = project.transform('data/doc.xml')       # -t web
docx = project.transform('data/doc.xml', mode='docx')
run = project.chunk('data/letters', format='json', overwrite=True)
records = project.index('data')

The lower-level functions in opm.transform, opm.chunking and opm.indexing stay available for callers that need finer control.

ChunkRun dataclass

ChunkRun(
    output_dir: Path,
    documents: tuple[Path, ...],
    format: str,
    modules: tuple[ResolvedTransform, ...],
    index_file: Path | None = None,
)

What Project.chunk wrote.

output_dir instance-attribute

output_dir: Path

Root of the output, holding one subdirectory per document.

documents instance-attribute

documents: tuple[Path, ...]

The XML files that were chunked, in order.

format instance-attribute

format: str

html, json or pb-view.

modules instance-attribute

modules: tuple[ResolvedTransform, ...]

The transform modules used: the main one first, then the fragment ones.

index_file class-attribute instance-attribute

index_file: Path | None = None

The index.html an HTML run writes at the output root.

Project

Project(
    config: ProjectConfig | None = None,
    *,
    root: Path | str | None = None,
)

An opm project represents a specific configuration and its associated methods.

Each method takes whatever its arguments leave open from the config, as the command of the same name does. ODDs compile on demand into the user cache, as they do for the CLI.

A project is a snapshot. It keeps the modules it has loaded and the register documents it has parsed, so repeated calls are cheap, for example in a web service. After editing an ODD or a register, make a new Project. One instance can be shared between threads.

An XPath expression that fails at run time counts as false or empty, as it does in the CLI. To see which ones failed, wrap the calls in collect_xpath_errors:

from opm import Project, collect_xpath_errors

project = Project.load('edition/opm.toml')
with collect_xpath_errors() as log:
    html = project.transform('edition/data/doc.xml')
for failure in log.ordered_failures():
    print(failure.expression, failure.message)

Relative paths given to the methods are relative to the current directory, as usual in Python. Paths inside opm.toml are relative to the file's directory, and the chunk output_dir to root.

Parameters:

Name Type Description Default
config ProjectConfig | None

The project settings. Defaults to an empty config, which uses the packaged ODD.

None
root Path | str | None

The project directory: chunk output goes below it, and styles/default-styles.css there replaces the packaged base rules. Defaults to the current directory.

None
Source code in src/opm/project.py
def __init__(
    self,
    config: ProjectConfig | None = None,
    *,
    root: Path | str | None = None,
) -> None:
    self.config: ProjectConfig = config if config is not None else ProjectConfig()
    """The settings from ``opm.toml``."""
    self.root: Path = Path(root).absolute() if root is not None else Path.cwd()
    """The project directory."""
    # Project modules (XPath extensions, chunk selectors) import from here.
    self.config.extend_sys_path()
    self._lock = threading.Lock()
    self._resolved: dict[tuple[str, Path | None, bool], ResolvedTransform] = {}
    self._modules: dict[Path, ModuleType] = {}
    self._registers: tuple[dict[str, Any], dict[str, list]] | None = None

config instance-attribute

config: ProjectConfig = (
    config if config is not None else ProjectConfig()
)

The settings from opm.toml.

root instance-attribute

root: Path = absolute() if root is not None else cwd()

The project directory.

load classmethod

load(
    path: Path | str | None = None,
    *,
    root: Path | str | None = None,
) -> Project

Load the project with the configuration given in path.

Without path, opm.toml in the current directory is read if it exists; if not, the project has default settings. root defaults to the config file's directory.

Raises:

Type Description
FileNotFoundError

path was given but is not a file.

Source code in src/opm/project.py
@classmethod
def load(cls, path: Path | str | None = None, *, root: Path | str | None = None) -> Project:
    """Load the project with the configuration given in *path*.

    Without *path*, ``opm.toml`` in the current directory is read if it
    exists; if not, the project has default settings. *root* defaults to
    the config file's directory.

    Raises:
        FileNotFoundError: *path* was given but is not a file.
    """
    if path is not None and not Path(path).is_file():
        raise FileNotFoundError(f'Config file not found: {path}')
    config_path = Path(path) if path is not None else Path(CONFIG_FILENAME)
    config = load_project_config(config_path)
    return cls(config, root=root if root is not None else config_path.absolute().parent)

with_config

with_config(**changes: Any) -> Project

A new project with some ProjectConfig fields replaced.

For example, project.with_config(document_css=Path('print.css')).

Source code in src/opm/project.py
def with_config(self, **changes: Any) -> Project:
    """A new project with some [`ProjectConfig`][opm.config.ProjectConfig] fields replaced.

    For example, ``project.with_config(document_css=Path('print.css'))``.
    """
    return Project(replace(self.config, **changes), root=self.root)

compile

compile(
    mode: str | None = None, odd: Path | str | None = None
) -> ResolvedTransform

Compile the ODD for output mode, or find it in the cache.

The ODD is odd, else [transform.<mode>] odd, else [transform] odd, else the packaged teipublisher.odd.

Parameters:

Name Type Description Default
mode str | None

An output mode such as web, markdown, docx, typst or json (see opm.output_modes). Defaults to web.

None
odd Path | str | None

The ODD to use instead of the configured one.

None

Raises:

Type Description
ValueError

mode is not an output mode.

Source code in src/opm/project.py
def compile(self, mode: str | None = None, odd: Path | str | None = None) -> ResolvedTransform:
    """Compile the ODD for output *mode*, or find it in the cache.

    The ODD is *odd*, else ``[transform.<mode>] odd``, else
    ``[transform] odd``, else the packaged ``teipublisher.odd``.

    Args:
        mode: An output mode such as ``web``, ``markdown``, ``docx``,
            ``typst`` or ``json`` (see [`opm.output_modes`][opm.output_modes]).
            Defaults to ``web``.
        odd: The ODD to use instead of the configured one.

    Raises:
        ValueError: *mode* is not an output mode.
    """
    name = output_mode(mode).name
    chosen = Path(odd) if odd is not None else self.config.odd_for_type(name)
    return self._resolve(name, chosen)

module

module(
    mode: str | None = None, odd: Path | str | None = None
) -> ModuleType

The loaded transform module for mode; see compile.

Source code in src/opm/project.py
def module(self, mode: str | None = None, odd: Path | str | None = None) -> ModuleType:
    """The loaded transform module for *mode*; see [`compile`][opm.project.Project.compile]."""
    return self._load(self.compile(mode, odd).module_path)

transform

transform(
    source: Source,
    *,
    mode: str | None = None,
    odd: Path | str | None = None,
    xpath: str | None = None,
    parameters: dict[str, str] | None = None,
    xpath_extensions: Sequence[str] | None = None,
    webcomponents: bool | None = None,
    template: Path | None = None,
) -> str | bytes

Transform source, as opm transform does.

Parameters:

Name Type Description Default
source Source

An XML file, or a tree or element parsed with lxml. When an element comes from a parsed file, doc() resolves against that file and $parameters?input_path names it.

required
mode str | None

The output mode (web by default); see compile.

None
odd Path | str | None

The ODD to use instead of the configured one.

None
xpath str | None

XPath 3.1 expression selecting the element to transform. Unprefixed names use the document's default namespace.

None
parameters dict[str, str] | None

XPath $parameters, over [transform.parameters].

None
xpath_extensions Sequence[str] | None

Extension modules to use instead of the configured ones.

None
webcomponents bool | None

Enable web-component mode. None uses the config.

None
template Path | None

The document template to use instead of the configured one.

None

Returns:

Type Description
str | bytes

str for text output (HTML, Markdown, Typst, JSON) and bytes

str | bytes

for DOCX and EPUB.

Source code in src/opm/project.py
def transform(
    self,
    source: Source,
    *,
    mode: str | None = None,
    odd: Path | str | None = None,
    xpath: str | None = None,
    parameters: dict[str, str] | None = None,
    xpath_extensions: Sequence[str] | None = None,
    webcomponents: bool | None = None,
    template: Path | None = None,
) -> str | bytes:
    """Transform *source*, as ``opm transform`` does.

    Args:
        source: An XML file, or a tree or element parsed with lxml. When
            an element comes from a parsed file, ``doc()`` resolves
            against that file and ``$parameters?input_path`` names it.
        mode: The output mode (``web`` by default); see [`compile`][opm.project.Project.compile].
        odd: The ODD to use instead of the configured one.
        xpath: XPath 3.1 expression selecting the element to transform.
            Unprefixed names use the document's default namespace.
        parameters: XPath ``$parameters``, over ``[transform.parameters]``.
        xpath_extensions: Extension modules to use instead of the
            configured ones.
        webcomponents: Enable web-component mode. ``None`` uses the config.
        template: The document template to use instead of the configured one.

    Returns:
        ``str`` for text output (HTML, Markdown, Typst, JSON) and ``bytes``
        for DOCX and EPUB.
    """
    return transform_with_config(
        self.module(mode, odd),
        _source_root(source),
        _source_path(source),
        self.config,
        xpath=xpath,
        parameters=parameters,
        xpath_extensions=xpath_extensions,
        webcomponents=webcomponents,
        template=template,
        documents=self._documents(),
    )

chunk_output_dir

chunk_output_dir(
    output_dir: Path | str | None = None,
) -> Path

Where chunk writes: output_dir, else [chunking] output_dir, below root.

Source code in src/opm/project.py
def chunk_output_dir(self, output_dir: Path | str | None = None) -> Path:
    """Where [`chunk`][opm.project.Project.chunk] writes: *output_dir*, else ``[chunking] output_dir``, below [`root`][opm.project.Project.root]."""
    return self.root / (output_dir if output_dir is not None else self._chunking().output_dir)

chunk_modules

chunk_modules(
    odd: Path | str | None = None,
) -> tuple[ResolvedTransform, ...]

Compile the modules a chunk run uses: the main one, then one per fragment ODD.

The main ODD is odd, else [chunking] odd, else the packaged one. Fragments without an ODD of their own use the main module.

Raises:

Type Description
ValueError

The config has no [chunking] section.

Source code in src/opm/project.py
def chunk_modules(self, odd: Path | str | None = None) -> tuple[ResolvedTransform, ...]:
    """Compile the modules a chunk run uses: the main one, then one per fragment ODD.

    The main ODD is *odd*, else ``[chunking] odd``, else the packaged one.
    Fragments without an ODD of their own use the main module.

    Raises:
        ValueError: The config has no ``[chunking]`` section.
    """
    chunking = self._chunking()
    main_odd = Path(odd) if odd is not None else chunking.odd
    main = self._resolve('web', main_odd, packaged_default=main_odd is None)
    fragments = tuple(
        self._resolve(fragment.mode, fragment.odd, packaged_default=False)
        for fragment in chunking.fragments or ()
        if fragment.odd is not None
    )
    return (main, *fragments)

chunk

chunk(
    source: Path | str,
    *,
    format: str = "html",
    output_dir: Path | str | None = None,
    template: Path | None = None,
    depth: int | None = None,
    odd: Path | str | None = None,
    doc_path: str | None = None,
    webcomponents: bool | None = None,
    xpath_extensions: Sequence[str] | None = None,
    overwrite: bool = False,
    on_document: Callable[[int, Path], None] | None = None,
    on_progress: Callable[[int, int], None] | None = None,
) -> ChunkRun

Split source into pages, as opm chunk does.

source is one XML file or a directory of them. For a directory, each document goes to a subdirectory named after it, and an HTML run also writes index.html listing the documents. pb-view output puts every document's data under <doc_path>/<name>.xml/ instead.

The arguments override the [chunking] settings of the same name.

Parameters:

Name Type Description Default
source Path | str

An XML file, or a directory of XML files (not searched recursively).

required
format str

html (pages rendered with the chunk template), json (one JSON file per chunk, for static site generators) or pb-view (data for the pb-view web component in static mode).

'html'
output_dir Path | str | None

Output directory, relative to root.

None
template Path | None

Page template for HTML output.

None
depth int | None

Maximum section depth to split at.

None
odd Path | str | None

The ODD to use instead of [chunking] odd.

None
doc_path str | None

For pb-view: the pb-document path the data is written under.

None
webcomponents bool | None

Enable web-component mode for HTML output. JSON and pb-view output always use it.

None
xpath_extensions Sequence[str] | None

Extension modules to use instead of the configured ones.

None
overwrite bool

Replace the output directory if it exists. Without it, an existing directory raises FileExistsError.

False
on_document Callable[[int, Path], None] | None

Called with the position and path of each document before it is chunked.

None
on_progress Callable[[int, int], None] | None

Called with (done, total) chunks as each document is processed.

None

Raises:

Type Description
ValueError

The config has no [chunking] section, format is unknown, or a directory contains no XML files.

FileExistsError

The output directory exists and overwrite does not allow replacing it.

Source code in src/opm/project.py
def chunk(
    self,
    source: Path | str,
    *,
    format: str = 'html',
    output_dir: Path | str | None = None,
    template: Path | None = None,
    depth: int | None = None,
    odd: Path | str | None = None,
    doc_path: str | None = None,
    webcomponents: bool | None = None,
    xpath_extensions: Sequence[str] | None = None,
    overwrite: bool = False,
    on_document: Callable[[int, Path], None] | None = None,
    on_progress: Callable[[int, int], None] | None = None,
) -> ChunkRun:
    """Split *source* into pages, as ``opm chunk`` does.

    *source* is one XML file or a directory of them. For a directory,
    each document goes to a subdirectory named after it, and an HTML run
    also writes ``index.html`` listing the documents. ``pb-view`` output
    puts every document's data under ``<doc_path>/<name>.xml/`` instead.

    The arguments override the ``[chunking]`` settings of the same name.

    Args:
        source: An XML file, or a directory of XML files (not searched
            recursively).
        format: ``html`` (pages rendered with the chunk template),
            ``json`` (one JSON file per chunk, for static site
            generators) or ``pb-view`` (data for the ``pb-view`` web
            component in static mode).
        output_dir: Output directory, relative to [`root`][opm.project.Project.root].
        template: Page template for HTML output.
        depth: Maximum section depth to split at.
        odd: The ODD to use instead of ``[chunking] odd``.
        doc_path: For ``pb-view``: the ``pb-document`` path the data is
            written under.
        webcomponents: Enable web-component mode for HTML output. JSON
            and ``pb-view`` output always use it.
        xpath_extensions: Extension modules to use instead of the
            configured ones.
        overwrite: Replace the output directory if it exists. Without
            it, an existing directory raises `FileExistsError`.
        on_document: Called with the position and path of each document
            before it is chunked.
        on_progress: Called with ``(done, total)`` chunks as each
            document is processed.

    Raises:
        ValueError: The config has no ``[chunking]`` section, *format* is
            unknown, or a directory contains no XML files.
        FileExistsError: The output directory exists and *overwrite*
            does not allow replacing it.
    """
    if format not in CHUNK_FORMATS:
        raise ValueError(f'format must be "html", "json" or "pb-view", got {format!r}')
    source = Path(source)
    files = chunk_input_files(source)
    by_directory = source.is_dir()
    if by_directory and not files:
        raise ValueError(f'no XML files found in directory {source}.')

    modules = self.chunk_modules(odd)
    chunking = self._chunking()
    changes: dict[str, Any] = {'module': modules[0].module_path}
    if output_dir is not None:
        changes['output_dir'] = str(output_dir)
    if template is not None:
        changes['template'] = Path(template)
    if depth is not None:
        changes['depth'] = depth
    if odd is not None:
        changes['odd'] = Path(odd)
    if chunking.fragments:
        compiled = iter(modules[1:])
        changes['fragments'] = [
            replace(fragment, module=next(compiled).module_path if fragment.odd else None)
            for fragment in chunking.fragments
        ]
    chunking = replace(chunking, **changes)

    out_dir = self.chunk_output_dir(chunking.output_dir)
    _clear_output_dir(out_dir, overwrite)

    enabled = (
        True if format in ('json', 'pb-view')
        else webcomponents if webcomponents is not None
        else bool(self.config.webcomponents_enabled)
    )
    extensions = tuple(xpath_extensions) if xpath_extensions else None
    base_doc_path = doc_path or chunking.doc_path
    # Built once and shared: templates test links against it.
    names = frozenset(path.name for path in files)

    for position, xml_file in enumerate(files):
        if on_document is not None:
            on_document(position, xml_file)
        if format == 'pb-view':
            # pb-view keeps its own layout: the data is fetched by path
            # rather than served as pages, and `doc_path` places it.
            document_config = chunking
        elif Path(chunking.output_dir).name == xml_file.name:
            # The output directory already names the document (-o site/doc.xml),
            # so take it as the per-document directory instead of nesting twice.
            document_config = replace(chunking, link_doc=xml_file.name)
        else:
            # One document or many, pages go to <output>/<name>.xml/ with the
            # stylesheets, assets and index shared at the root. Chunking a
            # single file therefore publishes the same URLs it will still
            # publish once a second document joins it.
            document_config = replace(
                chunking,
                output_dir=f'{chunking.output_dir.rstrip("/")}/{xml_file.name}',
                link_doc=xml_file.name,
            )
        if by_directory and format == 'pb-view':
            document_doc_path = (
                f'{base_doc_path.rstrip("/")}/{xml_file.name}' if base_doc_path
                else xml_file.name
            )
        else:
            document_doc_path = base_doc_path

        chunk_document(
            module_path=chunking.module,
            xml_path=xml_file,
            config=document_config,
            project_root=self.root,
            template_path=chunking.template,
            on_progress=on_progress,
            project_config=self.config,
            webcomponents=enabled,
            xpath_extensions=extensions,
            output_format=format,
            doc_path=document_doc_path,
            documents=names,
        )

    # Every HTML run leaves one subdirectory per document, which a web
    # server would otherwise show as a bare listing.
    index_file: Path | None = None
    if format == 'html':
        index_file = build_index(
            out_dir,
            template_path=chunking.index_template,
            title=chunking.index_title or source.name,
            module_path=chunking.module,
            project_config=self.config,
            project_root=self.root,
            chunking_config=chunking,
            webcomponents=enabled,
        )
    return ChunkRun(
        output_dir=out_dir,
        documents=tuple(files),
        format=format,
        modules=modules,
        index_file=index_file,
    )

index

index(
    sources: Path | str | Iterable[Path | str],
    *,
    odd: Path | str | None = None,
    options: IndexOptions | None = None,
) -> list[dict]

Search-index records for sources, as opm index makes them.

Parameters:

Name Type Description Default
sources Path | str | Iterable[Path | str]

An XML file, a directory (searched recursively), or a list of either.

required
odd Path | str | None

The ODD to use instead of [transform.json] odd.

None
options IndexOptions | None

Rollup settings. None uses [index] from the config.

None

Write the result with opm.indexing.write_jsonl.

Source code in src/opm/project.py
def index(
    self,
    sources: Path | str | Iterable[Path | str],
    *,
    odd: Path | str | None = None,
    options: IndexOptions | None = None,
) -> list[dict]:
    """Search-index records for *sources*, as ``opm index`` makes them.

    Args:
        sources: An XML file, a directory (searched recursively), or a
            list of either.
        odd: The ODD to use instead of ``[transform.json] odd``.
        options: Rollup settings. ``None`` uses ``[index]`` from the config.

    Write the result with [`opm.indexing.write_jsonl`][opm.indexing.write_jsonl].
    """
    from opm.indexing import index_document

    base_css = resolve_base_css(self.config.document_css, self.root)
    records: list[dict] = []
    for path in _corpus_files(sources):
        records.extend(
            index_document(
                path,
                cfg=self.config,
                odd=Path(odd) if odd is not None else None,
                project_root=self.root,
                options=options,
                base_css=base_css,
            ),
        )
    return records

coverage

coverage(
    sources: Path | str | Iterable[Path | str],
    *,
    mode: str = "json",
    odd: Path | str | None = None,
    parameters: dict[str, str] | None = None,
) -> CoverageReport

Measure the ODD against sources, as opm coverage does.

Parameters:

Name Type Description Default
sources Path | str | Iterable[Path | str]

An XML file, a directory (searched recursively), or a list of either.

required
mode str

json for the web channel, or json-<channel> for another one (json-print, json-typst, …).

'json'
odd Path | str | None

The ODD to use instead of [transform.json] odd.

None
parameters dict[str, str] | None

XPath $parameters, over [transform.parameters].

None
Source code in src/opm/project.py
def coverage(
    self,
    sources: Path | str | Iterable[Path | str],
    *,
    mode: str = 'json',
    odd: Path | str | None = None,
    parameters: dict[str, str] | None = None,
) -> CoverageReport:
    """Measure the ODD against *sources*, as ``opm coverage`` does.

    Args:
        sources: An XML file, a directory (searched recursively), or a
            list of either.
        mode: ``json`` for the web channel, or ``json-<channel>`` for
            another one (``json-print``, ``json-typst``, …).
        odd: The ODD to use instead of ``[transform.json] odd``.
        parameters: XPath ``$parameters``, over ``[transform.parameters]``.
    """
    from opm.coverage import analyze

    name = output_mode(mode).name
    chosen = Path(odd) if odd is not None else self.config.odd_for_type(name)
    return analyze(
        _corpus_files(sources),
        cfg=self.config,
        odd=chosen,
        output_mode=name,
        parameters=parameters,
        base_css=resolve_base_css(self.config.document_css, self.root),
    )

chunk_input_files

chunk_input_files(source: Path) -> list[Path]

The XML files a chunk run over source reads.

That is source itself, or the *.xml files directly inside it when it is a directory. Subdirectories are not searched: a chunk run publishes the pages of a given set of documents.

Source code in src/opm/project.py
def chunk_input_files(source: Path) -> list[Path]:
    """The XML files a chunk run over *source* reads.

    That is *source* itself, or the ``*.xml`` files directly inside it when it
    is a directory. Subdirectories are not searched: a chunk run publishes the
    pages of a given set of documents.
    """
    if source.is_dir():
        return sorted(
            path for path in source.iterdir()
            if path.is_file() and path.suffix.lower() == '.xml'
        )
    return [source]