tenkana, a spider that jumps from the checkpoints of your TeX document
----------------------------------------------------------------------

                            /\      ______      /\
                           /  \    / .  . \    /  \
                    _/\___/    \__|o(O)(O)o|__/    \___/\_
                   /      \  __   |   ''   |   __  /      \
                  /        \/  \   \______/   /  \/        \


tenkana is free software that makes repeated LaTeX builds faster.  Its command
is `tenkana'.  It needs no root access and changes neither the TeX engine nor
your document.

While TeX runs, tenkana keeps frozen copies ("snapshots") of the running
engine: one just before \begin{document} and one before each \include.
After an edit, tenkana continues from the last snapshot before the change, so
the preamble, its packages and fonts, and all unchanged chapters before
the edit are not processed again.

You give tenkana the TeX command you would type without it, as with env or
time; tenkana's own options come before it:

    tenkana lualatex thesis             draft build (default): fast,
                                        references not updated
    tenkana --final lualatex thesis     final build: references, citations
                                        and table of contents up to date
    tenkana -v pdflatex thesis          also show tenkana's details
                                        (passes, snapshots, memory)
    tenkana pdflatex -synctex=1 thesis  TeX's options go to TeX unchanged
    tenkana -q thesis                   short for `tenkana lualatex thesis'

`man tenkana' has the full manual.


Contents
--------

  1. What you see
  2. Installation
  3. How it works
  4. Snapshots on disk
  5. Measured results
  6. Building without snapshots
  7. Options and environment
  8. Limitations
  9. Files in this package
 10. Copyright and license


1. What you see
---------------

TeX's own terminal output, followed by tenkana's messages:

    Output written on talk.pdf (14 pages, 74150 bytes).
    Transcript written on talk.log.
    tenkana: jumped from ~5.6s for a plain run to 1.8s (68% less time).

Each message is one line and starts with `tenkana:', as with other Unix
commands; error messages go to stderr.  --silent (or TENKANA_SILENT) leaves out
all but error messages.  The comparison uses the time of the last full run of
the document.  When a build saves no time, tenkana says so and gives the
reason.  Every message tenkana can print is listed, with examples, in the file
MESSAGES; all texts live in src/tenkana/messages.py.

tenkana runs your command as you typed it, in the current folder, and adds
only -recorder right after the command name: it needs the list of files
TeX reads, which TeX writes to <job>.fls.  (XeTeX also gets -no-pdf, see
section 3.)  TeX's questions, after an error or from \typein, appear
on your terminal and take your answers, and Ctrl-C reaches TeX, which
answers with its "Interruption" prompt; this holds for resumed builds too.

On a resumed build, tenkana first prints what TeX printed up to the snapshot
(saved with it), then the live output, so the terminal transcript is the
same as a plain run's with the same options and -recorder.  So is the
.log file, except for two memory-statistics lines at the very end (tenkana
defines one extra macro).  Errors, warnings and SyncTeX refer to your
real file names and line numbers.


2. Installation
---------------

tenkana itself needs only Python 3.9 or newer.  It installs and runs without
TeX, but to turn a document into a PDF it calls a TeX engine (lualatex,
pdflatex, xelatex, ...), so for that you need a TeX distribution such as
TeX Live, with LaTeX 2020-10 or newer.

For snapshots, tenkana also needs:

  * GNU/Linux (tenkana uses LD_PRELOAD and /proc; it is not tested
    elsewhere);
  * a C compiler (cc, gcc or clang) for the snapshot library, which tenkana
    compiles from src/tenkana/tenkanasnap.c on first use and caches in
    ~/.cache/tenkana/ (in /tmp/tenkana-<uid>/ if that path contains a space or
    a colon, which LD_PRELOAD cannot handle);
  * an x86-64 machine.

Without a C compiler or on another processor, tenkana builds with plain runs
(section 6).

No root access is needed:

    ./install.sh              installs ~/.local/bin/tenkana and the man page
    ./install.sh --system     installs into /usr/local (uses sudo)
    ./install.sh --uninstall  removes tenkana again

The `tenkana' command is a single executable file (a Python zip application).
If ~/.local/bin is not on your PATH, the installer says what to add.


3. How it works
---------------

The core idea of process checkpointing is to capture a running process and
later continue from that exact point.  Tools such as CRIU do this from
outside the process, which needs root or special privileges.  tenkana lets the
engine copy itself instead, with fork(), which any program may do.

  * A small library, src/tenkana/tenkanasnap.c, is loaded into the unmodified
    engine with LD_PRELOAD.  The engine is started with your command, as
    in section 1 (`lualatex -recorder talk').

  * Pause points.  When the engine opens your main file, the library hands
    it the same text with two hooks added at the start of line 1, so line
    numbers do not change.  TeX still prints and records your file's real
    name.  The hooks make TeX open a special file just before
    \begin{document} and before every \include that TeX actually runs
    (not those skipped by \includeonly or a false conditional).  When the
    library sees that file opened, it notes the position of every open
    file and forks:

      - the copy freezes and becomes a snapshot.  It waits for commands
        and shows up in ps as `tenkana-snap-<n>'.  It marks itself as the first
        process the kernel's out-of-memory killer should stop, so if
        memory runs out, the kernel ends snapshots before your other
        programs (other memory managers, such as systemd-oomd or
        earlyoom, may choose differently);

      - the original reports to tenkana, waits while tenkana saves copies of
        the output files (.pdf, .aux, .log, ...), and carries on typesetting.

  * Resuming.  On the next run, tenkana puts the output files back as they
    were at the snapshot and tells that frozen copy to fork again.  The
    new copy restores every file position and continues typesetting from
    the pause point; for the main file it also discards text that had been
    read ahead, so an edited body is read fresh.  It reads its terminal
    input from the `tenkana' command that resumed it and gets back the
    engine's own signal handlers, so TeX can ask questions and Ctrl-C
    works as without tenkana.  The frozen copy stays frozen (holding no
    terminal) and can be resumed again and again.

  * Memory.  A snapshot holds what the engine holds at that point, mostly
    fonts and packages.  fork() is copy-on-write, so further snapshots of
    the same run only cost the memory pages changed in between: all 13
    snapshots of the 220-page test book took 330 to 400MB in our tests.
    `tenkana -v' shows the figure for your document.

  * Lifetime.  Snapshots are ordinary processes, so they survive between
    runs of tenkana and the closing of the terminal.  They end:

      - at a reboot, like every process;
      - at a logout only if the system ends a user's processes then (with
        systemd, when KillUserProcesses is on in logind.conf; distributions
        differ);
      - 8 hours after the document's last build (TENKANA_TTL; any build
        counts, even one that resumes no snapshot);
      - with `tenkana --clean', or within half a minute after tenkana's cache
        folder for the document is deleted;
      - if the kernel runs out of memory (see above).

    tenkana does not watch for any of this.  At the next build it finds that
    the snapshots are gone, and restores the preamble snapshot from disk
    (section 4) or makes a full run.

  * Which engine.  tenkana finds it from the program your command finally
    runs: pdflatex and pdflatex-dev run pdfTeX, lualatex runs LuaHBTeX,
    xelatex runs XeTeX.  Any command that runs pdfTeX, LuaTeX (also
    LuaHBTeX, LuaJITTeX) or XeTeX works.

  * Which file.  TeX's first line is what follows the options: usually the
    file name, as in `lualatex thesis'.  It may also be TeX code, as in
    `pdflatex '\nonstopmode\input paper.ltx''; tenkana then takes the file of
    the first \input as the main file.  Without an \input, tenkana runs the
    command as it is, without snapshots, and says so.  The file is looked
    for as TeX does: in the current folder, then on TeX's search path (as
    kpsewhich reports it), so `tenkana pdflatex sample2e' works.  If TeX
    would not find the file either, tenkana runs the command as it is and
    TeX reports the problem.

  * Where tenkana keeps its data.  Everything goes to ~/.cache/tenkana/
    ($XDG_CACHE_HOME/tenkana/ if that is set), one folder per document,
    readable only by you.  Nothing is written next to the document except
    TeX's own output (including <job>.fls), so a folder synchronised
    between computers (Nextcloud, Syncthing, ...) carries no snapshots;
    each computer keeps its own.  One build of a document runs at a time:
    a second one (from an editor that builds on every save, say) waits
    for the first.

  * Draft and final builds.  LaTeX reads the previous run's .aux (and
    .toc, ...) at \begin{document} and writes new ones during the run.  A
    normal build always does one pass, so a new or moved label shows as
    ?? or with an old number until you run `tenkana --final'.  That reruns
    from the preamble snapshot (taken before the .aux is read) until the
    .aux, .toc and similar files stop changing, like latexmk (at most 4
    passes).  In drafts, references keep the values read by the last
    build that started at \begin{document} (a full run, a final build, or
    an edit that resumed from the preamble snapshot); a draft that resumes
    from a chapter snapshot uses what that snapshot read.  The text and
    page layout are always current.

  * Change tracking.  Every file the engine reads is recorded (-recorder)
    and assigned to the part of the run that first read it: the preamble,
    the main file's body, or the stretch after one of the pauses.  The
    parts follow what TeX actually did, so \includeonly, conditionals and
    \include inside an \input file are handled correctly.  A change to a
    file read in the preamble needs a full run; any change to the main
    file's body resumes from the preamble snapshot; any other change
    resumes from the snapshot just before the part that first read the
    file.  The library also records every file TeX looked for but did not
    find in the document's folder, so a new chapter, image, .bbl (after
    biber) or .ind (after makeindex) is noticed too.  Files that other
    programs write for TeX are also watched by name: .bbl, .ind, .gls,
    .acr, .nls, .glstex, and glossary types declared in the .aux.  More
    can be added with TENKANA_WATCH.

  * Edits during a build.  A file saved while a build is running may or
    may not have been read by TeX already.  tenkana records each input as it
    was when the build started, so the next build takes such an edit as a
    change and none is lost.

  * Errors.  Snapshots taken before a LaTeX error stay valid, so after
    you fix the error tenkana resumes just before the chapter that failed.

  * Interrupted builds.  If tenkana is ended while it builds (kill, a closed
    terminal), it ends the TeX engine too.  The output files are then
    incomplete, and tenkana knows: the next build rebuilds them even if no
    input changed.

  * XeTeX runs with -no-pdf, so snapshots never involve its separate PDF
    converter (a second process, which a snapshot cannot hold); tenkana runs
    it once at the end, as XeTeX would (xdvipdfmx -q -E).  If you give
    -no-pdf yourself, tenkana leaves the .xdv file as it is.

  * SyncTeX.  TeX writes <job>.synctex(busy) during the run and renames
    it to <job>.synctex.gz at the end.  A snapshot holds the busy file
    open, so the snapshot library keeps that file in tenkana's cache (it
    catches the file however the engine opens it, also in engines with
    zlib built in, as TeX Live's own binaries are), and the renaming
    becomes a copy to <job>.synctex.gz.  So snapshots work with
    SyncTeX on.  For pdfTeX and LuaTeX the .synctex.gz file is the same as
    a plain run's; XeTeX's names the .xdv file as its output (it runs with
    -no-pdf), and viewers find the same positions in it.


4. Snapshots on disk
--------------------

After a build that made a new preamble snapshot, a copy of that snapshot
writes itself to tenkana's cache (section 3) in the background, unless it
would be larger than TENKANA_SAVE_LIMIT, so it survives a reboot.  Only what
is needed to rebuild it is stored:

  * memory backed by files (the engine, its libraries, mapped fonts) is
    mapped again from the files; only pages that differ are stored;
  * empty pages are skipped;
  * the rest is compressed with zstd (zlib if zstd is not installed).

Saving happens in a copy of the frozen engine, which reads the memory of a
pristine child of its own (a parent may read its child's memory without
root), so no build waits for it.

Restoring needs no root either.  tenkana starts the engine exactly as it
started the original: same command, recorded environment, and
address-space randomisation switched off for the engine (allowed for any
user), so the binary, libraries, stack and vDSO land at the same
addresses.  Before the engine's main() runs, the snapshot library
replaces the process's memory with the image, reopens its files at their
positions, restores signal handlers, working directory and thread pointer,
and continues from the frozen point.

An image is used only on the same kernel, with the same engine binary,
libraries and mapped files; otherwise tenkana makes a full run and says why.
TENKANA_SAVE_LIMIT (in MB, default 20) sets the largest image kept; 0 turns
saving off.  Images contain the engine's memory, including its environment
variables, which is why tenkana's cache is readable only by you.

Measured as an ordinary user:

    document                     engine memory  on disk  restore
    article, pdfLaTeX                     28MB    6.7MB     0.1s
    article, XeLaTeX                     113MB    8.4MB     0.2s
    article, LuaLaTeX                     96MB   13.5MB     0.2s
    220-page book, LuaLaTeX               90MB   13.3MB     0.2s
    talk with a Japanese font            719MB   55.5MB     1.1s
                                                 (over the default limit)


5. Measured results
-------------------

Test files are in testbook/ (run `python3 make_chapters.py' there first).
Machine: TeX Live 2023, LuaLaTeX, as an ordinary user.

book.tex: 220 pages, 12 \include'd chapters with TikZ, pgfplots, siunitx,
hyperref and cleveref.  A plain lualatex run takes 8.5s.  Medians of three
draft builds after an edit:

    edit in            tenkana  compared with a plain run
    chapter 12         0.77s   11 times as fast
    chapter 11         1.45s   5.8 times as fast
    chapter 9          2.63s   3.2 times as fast
    chapter 6          4.54s   1.9 times as fast
    chapter 1          7.54s   1.1 times as fast
    preamble           9.09s   7% slower (a full run)
    nothing changed    0.08s   build skipped

article.tex (the same preamble, one chapter, 20 pages): 2.07s plain,
0.76s with tenkana after an edit (63% less time).

Correctness: after `tenkana --final', the book's PDF was compared with plain
lualatex run on the same sources; all 220 pages were identical, also after
an edit that changes references across chapters.  The same held for
pdfLaTeX and XeLaTeX, and for builds restored from disk.


6. Building without snapshots
-----------------------------

When snapshots are not possible (no C compiler, not an x86-64 machine), or
with --no-snapshots, tenkana builds with plain runs:

  * pdfTeX, and XeTeX when the preamble loads no system fonts: the
    preamble is cached once in a private format (mylatexformat) and
    reused until it changes, unless TeX's first line is TeX code;
  * LuaTeX, and XeTeX with system fonts: these fonts cannot be stored in
    a format file, so the preamble is processed normally;
  * all engines: reruns until references settle with --final, and no run
    at all when no input changed.


7. Options and environment
--------------------------

    tenkana [tenkana options] TeX-command [TeX options] file
    tenkana [tenkana options] -q file

tenkana's options come first; the first word that is not one of them starts
the TeX command (`--' ends tenkana's options explicitly).

    -q, --quick          the file only, with the default engine:
                         `tenkana -q thesis' is `tenkana lualatex thesis'
    -F, --final          rerun until references, citations and table of
                         contents are up to date
    -d, --draft          one pass; references not updated (the default)
    -v, --verbose        also show passes, changed files, snapshots and
                         their memory
    --silent             show only TeX's output and tenkana's error messages
    --no-snapshots       build with plain runs instead of snapshots
    -f, --force          stop snapshots, discard cached state, build from
                         scratch
    --clean              stop a document's snapshots, delete its cache,
                         and exit.  Name the document as for a build
                         (`tenkana --clean lualatex thesis') or just by its
                         name (`tenkana --clean thesis'; thesis.tex and
                         thesis.ltx mean the same).  The file itself need
                         not exist any more.
    -V, --version        show version, copyright and license
    -h, --help           show a short help text

TeX's options go to TeX unchanged (-shell-escape, -synctex=1,
-interaction=batchmode, -file-line-error, -jobname=NAME, ...).  tenkana does
not support these yet, and says so: -ini, -fmt (and &format),
-output-directory, -aux-directory, -output-format and -output-driver.  A
change of the TeX command makes the next build a full run.

    TENKANA_ENGINE          the command for -q instead of lualatex; it may
                         have options ("pdflatex -shell-escape")
    TENKANA_SILENT          if set, as --silent
    TENKANA_TTL             seconds snapshots live after the document's last
                         build (default 28800, i.e. 8 hours)
    TENKANA_SAVE_LIMIT      largest preamble snapshot kept on disk, in MB
                         (default 20; 0 = never)
    TENKANA_WATCH           more generated-file extensions to watch
    TENKANA_SHIM            use this snapshot library instead of building one
    TENKANA_DEBUG           if set, explain why a snapshot could not be used


8. Limitations
--------------

  * Snapshots use memory while they live, roughly what the engine itself
    needs for your fonts and packages.  Use `tenkana --clean' when done, or a
    lower TENKANA_TTL.
  * Snapshots need \begin{document} in the main file itself.  Other
    documents -- plain TeX, or a main file that \inputs another with
    \begin{document} -- are built with plain runs (section 6); -v says
    so.
  * Chapter snapshots happen only at \include; documents that use only
    \input get the preamble snapshot.  With \includeonly, only the
    chapters it includes get snapshots.
  * The TeX options -ini, -fmt, -output-directory, -aux-directory,
    -output-format and -output-driver are not supported yet.
  * Snapshots keep the output files (.pdf, .aux, ...) open.  If another
    program replaces one of them (some synchronisation tools do), the
    snapshots that hold it cannot be used, and tenkana starts from an earlier
    one or makes a full run.
  * Only the preamble snapshot is saved on disk; after a reboot the first
    build resumes from the preamble and makes the chapter snapshots again.
  * Disk images need the same kernel and TeX installation as when they were
    saved (checked; otherwise a full run).  Where processes may not read
    their children's memory (Yama ptrace scope 2 or 3), images cannot be
    saved; snapshots in memory still work.
  * The engine runs with address-space randomisation switched off, which
    restoring images requires.  This affects only the engine tenkana starts.
  * tenkana's hooks are added to the start of line 1 of the copy of the main
    file that the engine reads, so a %&format line or a byte-order mark at
    the very start of the file is not recognised.
  * The pause uses read stream 15 (\openin15) for an instant; it would
    clash only with a document that keeps stream 15 open across
    \begin{document} or an \include.
  * \today, time stamps and \write18 side effects come from the run that
    made the snapshot.
  * The engine must be dynamically linked (true for TeX Live and the
    usual distribution packages), because the library is loaded with
    LD_PRELOAD.


9. Files in this package
------------------------

    README                     this file
    COPYING                    the GNU General Public License, version 3
    AUTHORS                    who wrote tenkana
    NEWS                       changes between versions
    MESSAGES                   every message, the help page and the man page
    tenkana.1                  the man page
    install.sh                 builds the single-file `tenkana' and installs it
    pyproject.toml             alternative installation with pip or pipx
    src/tenkana/cli.py         command line
    src/tenkana/messages.py    every text tenkana prints
    src/tenkana/snapshot.py    builds with snapshots
    src/tenkana/tenkanasnap.c  the snapshot library (LD_PRELOAD)
    src/tenkana/shim.py        compiles and locates the snapshot library
    src/tenkana/fallback.py    builds without snapshots
    src/tenkana/common.py      shared helpers
    tools/catalogue.py         regenerates MESSAGES from the source
    testbook/                  test documents (book.tex, article.tex,
                               talk.tex, make_chapters.py)


10. Copyright and license
-------------------------

Copyright © 2026 निरंजन (Niranjan)

tenkana is free software: you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free
Software Foundation, either version 3 of the License, or (at your
option) any later version.

tenkana is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
for more details.

You should have received a copy of the GNU General Public License
along with tenkana.  If not, see <https://www.gnu.org/licenses/>.

tenkana was written by Claude, a large language model (LLM) made by Anthropic.
Courts are divided on whether material generated by an LLM can be protected by
copyright.  Should it be found that the copyright claimant cannot hold
copyright in any part of tenkana, that part is in the public domain.  Wherever
copyright does hold, tenkana is distributed under the terms of the GNU General
Public License, version 3 or (at your option) any later version, as stated
above.
