Correct handling of lost chars in boxes with LuaTeX #1691

josephwright · 2025-03-04T17:35:43Z

READ ME FIRST: Please understand that in most cases we will not be able to merge a pull request because there are a lot of internal activities needed when updating the LaTeX2e sources. If you have a code suggestion please discuss it with the team first.

Internal housekeeping

Status of pull request

Feedback wanted
Under development
Ready to merge

Checklist of required changes before merge will be approved

Test file(s) added
Version and date string updated in changed source files
Relevant \changes entries in source included
Relevant changes.txt updated
Rollback provided (if necessary)?
ltnewsX.tex (and/or latexchanges.tex) updated

FrankMittelbach

That is unfortunately not enough for the issue at hand. LaTeX sets \tracinglostchars in several places, i.e., in \tracingall and \tracingnone and those need to be adjusted as well.

By the way what are the differences between the different values 2/3/4 ?

josephwright · 2025-03-04T17:58:15Z

OK, so perhaps plan B is \chardef the 'set on' value

\ifdefined\directlua
  \chardef\tracinglostchars@on=4
\else
  \chardef\tracinglostchars@on=3
\fi

then use this in the definition of \loggingall, etc.?

josephwright · 2025-03-04T17:59:06Z

By the way what are the differences between the different values 2/3/4 ?

For 4/5 instead of 2/3 - 'Hans agreed to match the classical behaviour when prodded'

josephwright · 2025-03-04T18:06:24Z

Plan B would presumably go in ltplain, btw.

davidcarlisle · 2025-03-04T18:45:18Z

OK, so perhaps plan B is \chardef the 'set on' value
\ifdefined\directlua
  \chardef\tracinglostchars@on=4
\else
  \chardef\tracinglostchars@on=3
\fi
then use this in the definition of \loggingall, etc.?

we could do, or alternatively as \loggingall is already set in an edef with some engine level tests could do

\tracinglostchars\ifx\directlua\undefined\thr@@\else 4 \fi

shouldn't it be 3+2=5 ?

davidcarlisle · 2025-03-04T18:54:32Z

shouldn't it be 3+2=5 ?

or more exactly shouldn't the tracing values be 2 and 4, was it intended to make \tracingall turn this in to an error condition?

davidcarlisle · 2025-03-04T19:23:19Z

so

\tracinglostchars\ifx\directlua\undefined\tw@\else 4 \fi

FrankMittelbach · 2025-03-04T20:06:33Z

shouldn't it be 3+2=5 ?

or more exactly shouldn't the tracing values be 2 and 4, was it intended to make \tracingall turn this in to an error condition?

I guess it was the intention back then, even though ltnews is a bit vague on the subject. However I'm okay with lkeeping it always just a warning which would also resolve the still open issue that \tracingnonesets it incorrectly back to 1. However that means

\tracinglostchars setting should be taken out of both \tracingall and \tracingnone
There should be an ltnews entry that \tracinglostchars is recommented to be set to 5 in luatex (not 3) because that is what we propagate in ltnews up to now

Or alternatively, we go \let\@@tracinglostchars=\tracinglostchars and redefine it to be a macro which does different things when in luatex (i.e., undo Hans's special design to give the warning always in LaTeX by default, i.e., somehting like

\def\tracinglostchars{\afterassignment\@tracinglostchars\@tmpcnta}
\def\@tracinglostchars{%
  \ifcase\@tmpcnta ...
}

I think I'm in favor of the latter solution so that in LaTeX \tracinglostchars has the usual meaning for 0,1,2,3 for all engines.

davidcarlisle · 2025-03-04T20:10:16Z

@FrankMittelbach yes it ocurred to me that that would mean we could/should remove it from \tracingall/tracingnone.

I'm not sure I like the macro setter, it would work well for setting but I think you'd need to support \showthe\tracinglostchars and that may be tricky with any kind of wrapper.

FrankMittelbach · 2025-03-04T20:14:35Z

@FrankMittelbach yes it ocurred to me that that would mean we could/should remove it from \tracingall/tracingnone.

I'm not sure I like the macro setter, it would work well for setting but I think you'd need to support \showthe\tracinglostchars and that may be tricky with any kind of wrapper.

true, that is a catch and not worth trying to find a solution for (even though there probably is one given that it is only needed in LuaTeX). But if not we need an ltnews entry that explain that you need to do =5 not =3 with LaTeX to get missing glyph errors.

josephwright · 2025-03-05T08:56:58Z

true, that is a catch and not worth trying to find a solution for (even though there probably is one given that it is only needed in LuaTeX). But if not we need an ltnews entry that explain that you need to do =5 not =3 with LaTeX to get missing glyph errors.

No, it's not that - you'll get errors if you set to 3, but only when the text is used. The new higher values are needed to get a missing char error when characters are typeset in a box, before the box is used.

davidcarlisle

would also need ltnews entry telling people to use 4 in luatex

davidcarlisle · 2025-03-05T09:01:33Z

base/ltplain.dtx

 %    \begin{macrocode}
 %<latexrelease>\IncludeInRelease{2021/06/01}{\loggingall}
 %<latexrelease>                 {\tracingstacklevels and \tracinglostchars=3}%
 %<*2ekernel|latexrelease>
 \edef\loggingall{%
  \tracingstats\tw@
  \tracingpages\@ne
-  \tracinglostchars\thr@@
+  \tracinglostchars


as discussed in the comments, 3/5 is consistent with the change making it 3 back in 2021 but I think that was in error and over-compensenation for the fact that we increased the default from 1 to 2 at that time.

I don't think it's expected that \tracingall makes non-error conditions into errors, so I would make this 2 and 4 or (more simply) remove the setting from here and from \tracingnone so the generic tracing on/off commands leave this at its default level.

If you want to set to 2 all of the time, does it even need to be in \tracingall, etc.? The standard setting is now 2 anyway. (Or are we worrying about places that set to zero/one not being reset?)

I agree \tracingall should not make it an error (it should simply leave the setting alone) but the recommendation for users in ltnews is to set it to 3 (to always make it an error) and that sensible and thus the new recommendation for LuaTeX would need to be to set it to 5 not 4.

@josephwright "does it even need to bein tracingall" No it doesn't, that's what @FrankMittelbach and I suggested: "or (more simply) remove the setting from here "

OK, so I remove the setting entirely from \tracingall and \loggingall, so we just have the initial setup plus the adjustment to \showhyphens, and I add something to ltnews41?

(Do we need rollback for that? I'm guessing no)

It doesn't need rollback but the whole setting of \tracinglostchars shoulw be removed including the next 9 lines setting it to 3 or 5. And the same with \tracingnone. It should stay transparent to whatever LaTeX has set or the user has changed to.

If there is a setting it should first save the current value (to return to that in \tracingnone) and then set it to 2/4 but I think it is over-engineering and we should take out the set as suggested by @davidcarlisle

josephwright · 2025-03-05T09:04:47Z

would also need ltnews entry telling people to use 4 in luatex

Again, only if they want identical behaviour to pdfTeX, etc.: setting to 2 or 3 still works, it's only an issue for boxes ...

FrankMittelbach · 2025-03-05T09:05:10Z

true, that is a catch and not worth trying to find a solution for (even though there probably is one given that it is only needed in LuaTeX). But if not we need an ltnews entry that explain that you need to do =5 not =3 with LaTeX to get missing glyph errors.

No, it's not that - you'll get errors if you set to 3, but only when the text is used. The new higher values are needed to get a missing char error when characters are typeset in a box, before the box is used.

but that is my point; yes you still get errors with a setting of 3 but not necessarily always (in contrast to pdftex, say) so to get the same behavior as with the other engines you need to set it to 5 not 3 in luatex

josephwright · 2025-03-05T09:10:58Z

but that is my point; yes you still get errors with a setting of 3 but not necessarily always (in contrast to pdftex, say) so to get the same behavior as with the other engines you need to set it to 5 not 3 in luatex

Yes, true - I guess I wonder if that's our job to document, as this is not something we control, it's an engine feature.

davidcarlisle · 2025-03-05T09:18:14Z

would also need ltnews entry telling people to use 4 in luatex

Again, only if they want identical behaviour to pdfTeX, etc.: setting to 2 or 3 still works, it's only an issue for boxes ...

3 only works if you use the box (and I think this is where this came up) it doesn't error at all if you don't use it, so

\settowidth\foo{abcdefghijk}

won't error if the font has no f if you just use \foo and don't typeset an f somewhere.

eg9 · 2025-03-05T09:31:48Z

I believe that LuaTeX has always been wrong in the treatment of \setbox. Not that we can do anything about this, except recommending to set \tracinglostchars to 5 in LuaLaTeX (error when a character is missing).

Likely \tracingall should set it to 2 or 4, so no error is raised. No need to restore back to the previous value: whoever uses \tracingall knows how to read its output.

Or, in the long run, we could ask the maintainers of pdftex to make 4 act the same as 2 and 5 the same as 3.

FrankMittelbach · 2025-03-05T10:02:13Z

base/ltplain.dtx

+  \tracinglostchars
+    \ifdefined\luatexversion
+      \ifnum\luatexversion>120 %
+        4 %
+      \else
+        \tw@
+      \fi
+    \else
+      \tw@
+    \fi


take out ...

I'm on it :)

FrankMittelbach · 2025-03-05T10:50:03Z

but that is my point; yes you still get errors with a setting of 3 but not necessarily always (in contrast to pdftex, say) so to get the same behavior as with the other engines you need to set it to 5 not 3 in luatex

Yes, true - I guess I wonder if that's our job to document, as this is not something we control, it's an engine feature.

but our recommendation is/was that people should set it so that they get an error if their document lost glyphs because the warnings are easily overlooked. And such an error is also important if there is some box setting going on even if the box is not use but only its features, e.g. its width or height. So yes, I think we should recomment that they set it to 5 with LuaTeX as painful as that may be.

base/ltplain.dtx

base/ltfssbas.dtx

muzimuzhi · 2025-03-05T18:45:04Z

base/doc/ltnews41.tex

+\subsection{Tracing lost characters}
+
+The \eTeX{} primitive \cs{tracinglostchars} allows the user to obtain a warning
+or error if a character is not available. The default value used in \LaTeX{} is
+set so that lost character information is written to the log and terminal.
+Users may wish to make this into an error, in which case they should set it to
+one higher. This is a \TeX{} primitive so the syntax for this is:
+\begin{verbatim} \advance\tracinglostchars by 1\relax \end{verbatim}


Do we need to also mention that \tracingall no longer sets \tracinglostchars=3?

When \tracinglostchars=3 was added to \tracingall, an entry was added to ltnews33, LaTeX release 2021-06-01.

One for @FrankMittelbach to decide about :)

Rather than advance, couldn't we just tell to use 5 which gives an error on all engines

As a correction to a previous statement I think it is worth pointing out, eg something like

In LaTeX news 33~\cite{41:ltnews33} we announced that \cs{tracingall} changes \cs{tracinglostchars} to an error condition. This change has been reverted and \cs{tracingall} and \cs{tracingnone} no longer alter \cs{tracinglostchars} but retain its current setting.

or something along these lines.

Rather than advance, couldn't we just tell to use 5 which gives an error on all engines

I agree that make the reading an explanation simpler.

Rather than advance, couldn't we just tell to use 5 which gives an error on all engines

OK, although I'm a bit wary that formally the behaviour is formally undefined for engines other than LuaTeX here.

In general all the tracing parameters are defined for all integer values, with all values larger acting like the largest defined value

TeXbook 212 but its only explicitly says so for \tracingcommands there. Anyway, I think David is right high values give you the largest defined one by convention in all engines

muzimuzhi · 2025-03-05T18:57:24Z

base/testfiles/github-0524.xetex.tlg

+{\tracingassigns}
+{into \tracingassigns=1}
+{the letter h}
+Missing character: There is no h ("68) in font nullfont!


Interesting XeTeX behaves differently than pdfTeX when \tracinglostchars=2, but the same with pdfTeX when \tracinglostchars=3.

% \tracinglostchars=2 % pdftex Missing character: There is no h in font nullfont! % xetex Missing character: There is no h ("68) in font nullfont!

Yeah, doubt it's deliberate - but I'm not sure there is much to be done

it isn't perfect but it isn't that much of a problem either (for us 1-2 extra tlg files due to the differences)

Co-authored-by: Yukai Chou <[email protected]>

josephwright linked an issue Mar 4, 2025 that may be closed by this pull request

Set \tracinglostchars to 4 or 5 in the LuaLaTeX initialization #1687

Closed

josephwright requested review from FrankMittelbach and eg9 March 4, 2025 17:35

FrankMittelbach reviewed Mar 4, 2025

View reviewed changes

josephwright closed this Mar 5, 2025

josephwright force-pushed the gh1687 branch from 9a90d50 to ad660ba Compare March 5, 2025 08:54

josephwright reopened this Mar 5, 2025

davidcarlisle requested changes Mar 5, 2025

View reviewed changes

FrankMittelbach reviewed Mar 5, 2025

View reviewed changes

josephwright force-pushed the gh1687 branch 2 times, most recently from d677ecb to 82056f5 Compare March 5, 2025 10:18

josephwright force-pushed the gh1687 branch 2 times, most recently from 26e8601 to a5a0303 Compare March 5, 2025 12:02

Update \tracinglostchars for LuaTeX in TL'25+

f4efc41

josephwright force-pushed the gh1687 branch from a5a0303 to f4efc41 Compare March 5, 2025 14:17

muzimuzhi reviewed Mar 5, 2025

View reviewed changes

josephwright and others added 3 commits March 6, 2025 07:40

Apply suggestions from code review

61f3920

Co-authored-by: Yukai Chou <[email protected]>

Adjust the News entry

f71e0b4

Further text refinement

b274ad6

FrankMittelbach approved these changes Mar 6, 2025

View reviewed changes

davidcarlisle approved these changes Mar 6, 2025

View reviewed changes

josephwright merged commit dfdf24d into develop Mar 6, 2025
88 checks passed

josephwright deleted the gh1687 branch March 6, 2025 09:20

Correct handling of lost chars in boxes with LuaTeX #1691

Correct handling of lost chars in boxes with LuaTeX #1691

Conversation

josephwright commented Mar 4, 2025

Internal housekeeping

Status of pull request

Checklist of required changes before merge will be approved

FrankMittelbach left a comment • edited Loading

Choose a reason for hiding this comment

josephwright commented Mar 4, 2025

josephwright commented Mar 4, 2025

josephwright commented Mar 4, 2025

davidcarlisle commented Mar 4, 2025

davidcarlisle commented Mar 4, 2025

davidcarlisle commented Mar 4, 2025

FrankMittelbach commented Mar 4, 2025

davidcarlisle commented Mar 4, 2025

FrankMittelbach commented Mar 4, 2025

josephwright commented Mar 5, 2025

davidcarlisle left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FrankMittelbach Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

davidcarlisle Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josephwright commented Mar 5, 2025

FrankMittelbach commented Mar 5, 2025

josephwright commented Mar 5, 2025

davidcarlisle commented Mar 5, 2025 • edited Loading

eg9 commented Mar 5, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FrankMittelbach commented Mar 5, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FrankMittelbach left a comment •

edited

Loading

FrankMittelbach Mar 5, 2025 •

edited

Loading

davidcarlisle Mar 5, 2025 •

edited

Loading

davidcarlisle commented Mar 5, 2025 •

edited

Loading