When an implicit token is allowed?
Clash Royale CLAN TAG#URR8PPP
up vote
7
down vote
favorite
Example: the latex code:
documentclassarticle
begindocument
% test-case 1
% Output: "b"
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup
% test-case 2
% Output: "b"
% Expansion is allowed, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup
% test-case 3
% Compile error: cq is undefined
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup
% test-case 4
% Output: "B"
% Expansion is allowed, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup
enddocument
Here are my questions:
Whether the implicit token is allowed if expansion is allowed in the context?
This might be the only way to explain the test-case 1, 2, 3.But it can not explain test-case 4.
If not, when it is allowed?
I have read the TeXByTopic but I can't find a certain conclusion.
tex-core
New contributor
add a comment |Â
up vote
7
down vote
favorite
Example: the latex code:
documentclassarticle
begindocument
% test-case 1
% Output: "b"
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup
% test-case 2
% Output: "b"
% Expansion is allowed, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup
% test-case 3
% Compile error: cq is undefined
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup
% test-case 4
% Output: "B"
% Expansion is allowed, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup
enddocument
Here are my questions:
Whether the implicit token is allowed if expansion is allowed in the context?
This might be the only way to explain the test-case 1, 2, 3.But it can not explain test-case 4.
If not, when it is allowed?
I have read the TeXByTopic but I can't find a certain conclusion.
tex-core
New contributor
Welcome to TeX.SX! Yes, that's the interesting thing about expansion anduppercase
is not expandable in itself.
â TeXnician
Sep 6 at 6:04
in test-case 4, you letimplicitToken
tob
which is not-expandable, soimplicitToken
does not expand in anedef
. Then you redefine its meaning so the new meaning is used.
â jfbu
Sep 6 at 8:57
I hope my answer is clear but to emphasis that implicit character tokens are not expanded to their meaning, they are equal to the character token. The classic example isletbgroup= letegroup=
.
â Joseph Wrightâ¦
Sep 6 at 8:58
you can achieve same as test-case 4 withletimplicitTokenignorespaces
then do theedef
then doletimplicitTokenhrule
and executecq
to see the rule. (if you try this test in isolation, beware there should also be something else printed on the page for thehrule
to show...)
â jfbu
Sep 6 at 8:58
there is no expansion in any of your examples except one stepcq
toimplicitToken
â David Carlisle
Sep 6 at 9:06
add a comment |Â
up vote
7
down vote
favorite
up vote
7
down vote
favorite
Example: the latex code:
documentclassarticle
begindocument
% test-case 1
% Output: "b"
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup
% test-case 2
% Output: "b"
% Expansion is allowed, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup
% test-case 3
% Compile error: cq is undefined
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup
% test-case 4
% Output: "B"
% Expansion is allowed, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup
enddocument
Here are my questions:
Whether the implicit token is allowed if expansion is allowed in the context?
This might be the only way to explain the test-case 1, 2, 3.But it can not explain test-case 4.
If not, when it is allowed?
I have read the TeXByTopic but I can't find a certain conclusion.
tex-core
New contributor
Example: the latex code:
documentclassarticle
begindocument
% test-case 1
% Output: "b"
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup
% test-case 2
% Output: "b"
% Expansion is allowed, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup
% test-case 3
% Compile error: cq is undefined
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup
% test-case 4
% Output: "B"
% Expansion is allowed, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup
enddocument
Here are my questions:
Whether the implicit token is allowed if expansion is allowed in the context?
This might be the only way to explain the test-case 1, 2, 3.But it can not explain test-case 4.
If not, when it is allowed?
I have read the TeXByTopic but I can't find a certain conclusion.
tex-core
New contributor
edited Sep 6 at 8:56
New contributor
asked Sep 6 at 3:45
Leslie Leigh
384
384
New contributor
New contributor
Welcome to TeX.SX! Yes, that's the interesting thing about expansion anduppercase
is not expandable in itself.
â TeXnician
Sep 6 at 6:04
in test-case 4, you letimplicitToken
tob
which is not-expandable, soimplicitToken
does not expand in anedef
. Then you redefine its meaning so the new meaning is used.
â jfbu
Sep 6 at 8:57
I hope my answer is clear but to emphasis that implicit character tokens are not expanded to their meaning, they are equal to the character token. The classic example isletbgroup= letegroup=
.
â Joseph Wrightâ¦
Sep 6 at 8:58
you can achieve same as test-case 4 withletimplicitTokenignorespaces
then do theedef
then doletimplicitTokenhrule
and executecq
to see the rule. (if you try this test in isolation, beware there should also be something else printed on the page for thehrule
to show...)
â jfbu
Sep 6 at 8:58
there is no expansion in any of your examples except one stepcq
toimplicitToken
â David Carlisle
Sep 6 at 9:06
add a comment |Â
Welcome to TeX.SX! Yes, that's the interesting thing about expansion anduppercase
is not expandable in itself.
â TeXnician
Sep 6 at 6:04
in test-case 4, you letimplicitToken
tob
which is not-expandable, soimplicitToken
does not expand in anedef
. Then you redefine its meaning so the new meaning is used.
â jfbu
Sep 6 at 8:57
I hope my answer is clear but to emphasis that implicit character tokens are not expanded to their meaning, they are equal to the character token. The classic example isletbgroup= letegroup=
.
â Joseph Wrightâ¦
Sep 6 at 8:58
you can achieve same as test-case 4 withletimplicitTokenignorespaces
then do theedef
then doletimplicitTokenhrule
and executecq
to see the rule. (if you try this test in isolation, beware there should also be something else printed on the page for thehrule
to show...)
â jfbu
Sep 6 at 8:58
there is no expansion in any of your examples except one stepcq
toimplicitToken
â David Carlisle
Sep 6 at 9:06
Welcome to TeX.SX! Yes, that's the interesting thing about expansion and
uppercase
is not expandable in itself.â TeXnician
Sep 6 at 6:04
Welcome to TeX.SX! Yes, that's the interesting thing about expansion and
uppercase
is not expandable in itself.â TeXnician
Sep 6 at 6:04
in test-case 4, you let
implicitToken
to b
which is not-expandable, so implicitToken
does not expand in an edef
. Then you redefine its meaning so the new meaning is used.â jfbu
Sep 6 at 8:57
in test-case 4, you let
implicitToken
to b
which is not-expandable, so implicitToken
does not expand in an edef
. Then you redefine its meaning so the new meaning is used.â jfbu
Sep 6 at 8:57
I hope my answer is clear but to emphasis that implicit character tokens are not expanded to their meaning, they are equal to the character token. The classic example is
letbgroup= letegroup=
.â Joseph Wrightâ¦
Sep 6 at 8:58
I hope my answer is clear but to emphasis that implicit character tokens are not expanded to their meaning, they are equal to the character token. The classic example is
letbgroup= letegroup=
.â Joseph Wrightâ¦
Sep 6 at 8:58
you can achieve same as test-case 4 with
letimplicitTokenignorespaces
then do the edef
then do letimplicitTokenhrule
and execute cq
to see the rule. (if you try this test in isolation, beware there should also be something else printed on the page for the hrule
to show...)â jfbu
Sep 6 at 8:58
you can achieve same as test-case 4 with
letimplicitTokenignorespaces
then do the edef
then do letimplicitTokenhrule
and execute cq
to see the rule. (if you try this test in isolation, beware there should also be something else printed on the page for the hrule
to show...)â jfbu
Sep 6 at 8:58
there is no expansion in any of your examples except one step
cq
to implicitToken
â David Carlisle
Sep 6 at 9:06
there is no expansion in any of your examples except one step
cq
to implicitToken
â David Carlisle
Sep 6 at 9:06
add a comment |Â
2 Answers
2
active
oldest
votes
up vote
4
down vote
accepted
Implicit tokens are not expanded to their meaning, they are their meaning. So with
lettokenAlias=b
tokenAlias
is an b
. That shows up in your second test: tokenAlias
and b
are entirely equivalent.
However, case changing takes place based on the character code of the token concerned. (As noted in a comment, uppercase
is also not expandable.) Thus tokenAlias
is not affected by uppercase
as it does not have a character code at all, being a control sequence.
The only way to get implicit character tokens to 'appear naturally' is to iterate over input and use a test on the meaning of each one, replacing every implicit token by it's meaning.
Taking the tests one at a time (following the edit), no expansion takes place in any cases. Thus:
implicitToken
remains unchanged as it does not have a character code in the upper case tableimplicitToken
isb
, so comparing the two usingif
is true: there is no expansion ofimplicitToken
.cq
is never defined, and so thelet
anddef
lines have no effectThe
edef
does nothing asimplicitToken
is not expanded. Rather, we end up withcq
with replacement textimplicitToken
: the effect this has thus depends on the meaning at point-of-use.
Note (again) that typesetting is not an expansion context: the result depends on the execution of the token.
uppercase
is also not expandable. SotokenAlias
->l
is an expansion?
â Leslie Leigh
Sep 6 at 7:36
@LeslieLeightokenAlias
is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, wheretokenAlias
isl
, which then produces the outputl
. That's not expansion.
â Joseph Wrightâ¦
Sep 6 at 7:41
Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it?
â Leslie Leigh
Sep 6 at 9:03
@LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for exampleuppercaseb
is the tokenB
but there is no token other thanimplicitToken
thatuppercaseimplicitToken
could be defined to be.
â David Carlisle
Sep 6 at 9:09
@LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic foruppercase
for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise).
â Joseph Wrightâ¦
Sep 6 at 11:46
add a comment |Â
up vote
4
down vote
Let's analyze your tests one by one.
An implicit token (more precisely an implicit character token) is a symbolic token defined by let<token>=<character>
.
Test 2
% test-case 2
% Output: "b"
% Expansion is enabled, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup
This outputs âÂÂbâÂÂ, because an implicit token behaves like the character it is defined to be equal to in the context of if
or ifcat
.
Test 3
% test-case 3
% Compile error: cq is undefined
% Expansion is disabled, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup
This is easier: the assignment defimplicitToken
overrides the let
. On the other hand, implicitToken
is never used and cq
(undefined) raises an error.
Test 4
% test-case 4
% Output: "B"
% Expansion is enabled, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup
If you add showcq
after doing the edef
you'll see
> cq=macro:
->implicitToken .
and the output will contain a âÂÂBâÂÂ, because of the subsequent redefinition.
An implicit token is not expandable, so the codes
letimplicitToken=b
edefcqimplicitToken
and
letimplicitToken=b
defcqimplicitToken
are completely equivalent. TeX will use the meaning of implicitToken
current at expansion of cq
time: in the case of your test it is âÂÂBâÂÂ.
Test 1
% test-case 1
% Output: "b"
% Expansion is disabled, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup
The primitive uppercase
examines its argument (without macro expansion, but it's irrelevant here) and transforms each explicit character token into its uppercase form (as stored in the uccode
vector). It doesn't act on implicit tokens.
Comments
There is an asymmetry: with letimplicitToken=b
, the test ifimplicitToken b
will return true. Similarly ifcat
will use the category code of the character current at definition time; for instance
letimplicitToken=b
catcode`b=12
ifcatimplicitToken a%
messageletter%
else
messagenonletter%
fi
will print letter
on the console, whereas ifcat ba
would print nonletter
.
Why this asymmetry? Sorry, but only Donald Knuth can answer. The most prominent usage of implicit tokens is with bgroup
and egroup
, which can be used as substitutes for and
in certain (not all) contexts. I guess that making them subject to transformations in
uppercase
or lowercase
would be irksome.
Expansion
Tokens in TeX can be expandable or unexpandable. A token of the latter kind, when found alone, is directly passed to the execution processor. What does this mean? Let's see an example. The primitive dimen
is not expandable. If TeX finds dimen2=12pt
, it will pass dimen
to the execution processor; the execution prompts the search in the input stream for a <number>
, an <optional equal>
and a suitable value (with expansion). On the contrary, in advancedimen2 by 2pt
, the token dimen
is absorbed as the argument to advance
for further processing. Non active (explicit) character tokens are not expandable; an active character can be or not, depending on how it's meaning is defined.
What tokens are expandable? Easy:
- some primitives, such as conditionals,
string
,noexpand
and others;
macros, that is, tokens defined withdef
,edef
,gdef
orxdef
;- tokens
let
equal to an expandable token.
Since your implicitToken
belongs to none of the above types, it is unexpandable.
add a comment |Â
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
4
down vote
accepted
Implicit tokens are not expanded to their meaning, they are their meaning. So with
lettokenAlias=b
tokenAlias
is an b
. That shows up in your second test: tokenAlias
and b
are entirely equivalent.
However, case changing takes place based on the character code of the token concerned. (As noted in a comment, uppercase
is also not expandable.) Thus tokenAlias
is not affected by uppercase
as it does not have a character code at all, being a control sequence.
The only way to get implicit character tokens to 'appear naturally' is to iterate over input and use a test on the meaning of each one, replacing every implicit token by it's meaning.
Taking the tests one at a time (following the edit), no expansion takes place in any cases. Thus:
implicitToken
remains unchanged as it does not have a character code in the upper case tableimplicitToken
isb
, so comparing the two usingif
is true: there is no expansion ofimplicitToken
.cq
is never defined, and so thelet
anddef
lines have no effectThe
edef
does nothing asimplicitToken
is not expanded. Rather, we end up withcq
with replacement textimplicitToken
: the effect this has thus depends on the meaning at point-of-use.
Note (again) that typesetting is not an expansion context: the result depends on the execution of the token.
uppercase
is also not expandable. SotokenAlias
->l
is an expansion?
â Leslie Leigh
Sep 6 at 7:36
@LeslieLeightokenAlias
is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, wheretokenAlias
isl
, which then produces the outputl
. That's not expansion.
â Joseph Wrightâ¦
Sep 6 at 7:41
Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it?
â Leslie Leigh
Sep 6 at 9:03
@LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for exampleuppercaseb
is the tokenB
but there is no token other thanimplicitToken
thatuppercaseimplicitToken
could be defined to be.
â David Carlisle
Sep 6 at 9:09
@LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic foruppercase
for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise).
â Joseph Wrightâ¦
Sep 6 at 11:46
add a comment |Â
up vote
4
down vote
accepted
Implicit tokens are not expanded to their meaning, they are their meaning. So with
lettokenAlias=b
tokenAlias
is an b
. That shows up in your second test: tokenAlias
and b
are entirely equivalent.
However, case changing takes place based on the character code of the token concerned. (As noted in a comment, uppercase
is also not expandable.) Thus tokenAlias
is not affected by uppercase
as it does not have a character code at all, being a control sequence.
The only way to get implicit character tokens to 'appear naturally' is to iterate over input and use a test on the meaning of each one, replacing every implicit token by it's meaning.
Taking the tests one at a time (following the edit), no expansion takes place in any cases. Thus:
implicitToken
remains unchanged as it does not have a character code in the upper case tableimplicitToken
isb
, so comparing the two usingif
is true: there is no expansion ofimplicitToken
.cq
is never defined, and so thelet
anddef
lines have no effectThe
edef
does nothing asimplicitToken
is not expanded. Rather, we end up withcq
with replacement textimplicitToken
: the effect this has thus depends on the meaning at point-of-use.
Note (again) that typesetting is not an expansion context: the result depends on the execution of the token.
uppercase
is also not expandable. SotokenAlias
->l
is an expansion?
â Leslie Leigh
Sep 6 at 7:36
@LeslieLeightokenAlias
is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, wheretokenAlias
isl
, which then produces the outputl
. That's not expansion.
â Joseph Wrightâ¦
Sep 6 at 7:41
Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it?
â Leslie Leigh
Sep 6 at 9:03
@LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for exampleuppercaseb
is the tokenB
but there is no token other thanimplicitToken
thatuppercaseimplicitToken
could be defined to be.
â David Carlisle
Sep 6 at 9:09
@LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic foruppercase
for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise).
â Joseph Wrightâ¦
Sep 6 at 11:46
add a comment |Â
up vote
4
down vote
accepted
up vote
4
down vote
accepted
Implicit tokens are not expanded to their meaning, they are their meaning. So with
lettokenAlias=b
tokenAlias
is an b
. That shows up in your second test: tokenAlias
and b
are entirely equivalent.
However, case changing takes place based on the character code of the token concerned. (As noted in a comment, uppercase
is also not expandable.) Thus tokenAlias
is not affected by uppercase
as it does not have a character code at all, being a control sequence.
The only way to get implicit character tokens to 'appear naturally' is to iterate over input and use a test on the meaning of each one, replacing every implicit token by it's meaning.
Taking the tests one at a time (following the edit), no expansion takes place in any cases. Thus:
implicitToken
remains unchanged as it does not have a character code in the upper case tableimplicitToken
isb
, so comparing the two usingif
is true: there is no expansion ofimplicitToken
.cq
is never defined, and so thelet
anddef
lines have no effectThe
edef
does nothing asimplicitToken
is not expanded. Rather, we end up withcq
with replacement textimplicitToken
: the effect this has thus depends on the meaning at point-of-use.
Note (again) that typesetting is not an expansion context: the result depends on the execution of the token.
Implicit tokens are not expanded to their meaning, they are their meaning. So with
lettokenAlias=b
tokenAlias
is an b
. That shows up in your second test: tokenAlias
and b
are entirely equivalent.
However, case changing takes place based on the character code of the token concerned. (As noted in a comment, uppercase
is also not expandable.) Thus tokenAlias
is not affected by uppercase
as it does not have a character code at all, being a control sequence.
The only way to get implicit character tokens to 'appear naturally' is to iterate over input and use a test on the meaning of each one, replacing every implicit token by it's meaning.
Taking the tests one at a time (following the edit), no expansion takes place in any cases. Thus:
implicitToken
remains unchanged as it does not have a character code in the upper case tableimplicitToken
isb
, so comparing the two usingif
is true: there is no expansion ofimplicitToken
.cq
is never defined, and so thelet
anddef
lines have no effectThe
edef
does nothing asimplicitToken
is not expanded. Rather, we end up withcq
with replacement textimplicitToken
: the effect this has thus depends on the meaning at point-of-use.
Note (again) that typesetting is not an expansion context: the result depends on the execution of the token.
edited Sep 6 at 8:57
answered Sep 6 at 6:35
Joseph Wrightâ¦
196k21540860
196k21540860
uppercase
is also not expandable. SotokenAlias
->l
is an expansion?
â Leslie Leigh
Sep 6 at 7:36
@LeslieLeightokenAlias
is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, wheretokenAlias
isl
, which then produces the outputl
. That's not expansion.
â Joseph Wrightâ¦
Sep 6 at 7:41
Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it?
â Leslie Leigh
Sep 6 at 9:03
@LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for exampleuppercaseb
is the tokenB
but there is no token other thanimplicitToken
thatuppercaseimplicitToken
could be defined to be.
â David Carlisle
Sep 6 at 9:09
@LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic foruppercase
for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise).
â Joseph Wrightâ¦
Sep 6 at 11:46
add a comment |Â
uppercase
is also not expandable. SotokenAlias
->l
is an expansion?
â Leslie Leigh
Sep 6 at 7:36
@LeslieLeightokenAlias
is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, wheretokenAlias
isl
, which then produces the outputl
. That's not expansion.
â Joseph Wrightâ¦
Sep 6 at 7:41
Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it?
â Leslie Leigh
Sep 6 at 9:03
@LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for exampleuppercaseb
is the tokenB
but there is no token other thanimplicitToken
thatuppercaseimplicitToken
could be defined to be.
â David Carlisle
Sep 6 at 9:09
@LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic foruppercase
for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise).
â Joseph Wrightâ¦
Sep 6 at 11:46
uppercase
is also not expandable. So tokenAlias
->l
is an expansion?â Leslie Leigh
Sep 6 at 7:36
uppercase
is also not expandable. So tokenAlias
->l
is an expansion?â Leslie Leigh
Sep 6 at 7:36
@LeslieLeigh
tokenAlias
is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, where tokenAlias
is l
, which then produces the output l
. That's not expansion.â Joseph Wrightâ¦
Sep 6 at 7:41
@LeslieLeigh
tokenAlias
is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, where tokenAlias
is l
, which then produces the output l
. That's not expansion.â Joseph Wrightâ¦
Sep 6 at 7:41
Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it?
â Leslie Leigh
Sep 6 at 9:03
Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it?
â Leslie Leigh
Sep 6 at 9:03
@LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for example
uppercaseb
is the token B
but there is no token other than implicitToken
that uppercaseimplicitToken
could be defined to be.â David Carlisle
Sep 6 at 9:09
@LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for example
uppercaseb
is the token B
but there is no token other than implicitToken
that uppercaseimplicitToken
could be defined to be.â David Carlisle
Sep 6 at 9:09
@LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic for
uppercase
for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise).â Joseph Wrightâ¦
Sep 6 at 11:46
@LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic for
uppercase
for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise).â Joseph Wrightâ¦
Sep 6 at 11:46
add a comment |Â
up vote
4
down vote
Let's analyze your tests one by one.
An implicit token (more precisely an implicit character token) is a symbolic token defined by let<token>=<character>
.
Test 2
% test-case 2
% Output: "b"
% Expansion is enabled, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup
This outputs âÂÂbâÂÂ, because an implicit token behaves like the character it is defined to be equal to in the context of if
or ifcat
.
Test 3
% test-case 3
% Compile error: cq is undefined
% Expansion is disabled, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup
This is easier: the assignment defimplicitToken
overrides the let
. On the other hand, implicitToken
is never used and cq
(undefined) raises an error.
Test 4
% test-case 4
% Output: "B"
% Expansion is enabled, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup
If you add showcq
after doing the edef
you'll see
> cq=macro:
->implicitToken .
and the output will contain a âÂÂBâÂÂ, because of the subsequent redefinition.
An implicit token is not expandable, so the codes
letimplicitToken=b
edefcqimplicitToken
and
letimplicitToken=b
defcqimplicitToken
are completely equivalent. TeX will use the meaning of implicitToken
current at expansion of cq
time: in the case of your test it is âÂÂBâÂÂ.
Test 1
% test-case 1
% Output: "b"
% Expansion is disabled, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup
The primitive uppercase
examines its argument (without macro expansion, but it's irrelevant here) and transforms each explicit character token into its uppercase form (as stored in the uccode
vector). It doesn't act on implicit tokens.
Comments
There is an asymmetry: with letimplicitToken=b
, the test ifimplicitToken b
will return true. Similarly ifcat
will use the category code of the character current at definition time; for instance
letimplicitToken=b
catcode`b=12
ifcatimplicitToken a%
messageletter%
else
messagenonletter%
fi
will print letter
on the console, whereas ifcat ba
would print nonletter
.
Why this asymmetry? Sorry, but only Donald Knuth can answer. The most prominent usage of implicit tokens is with bgroup
and egroup
, which can be used as substitutes for and
in certain (not all) contexts. I guess that making them subject to transformations in
uppercase
or lowercase
would be irksome.
Expansion
Tokens in TeX can be expandable or unexpandable. A token of the latter kind, when found alone, is directly passed to the execution processor. What does this mean? Let's see an example. The primitive dimen
is not expandable. If TeX finds dimen2=12pt
, it will pass dimen
to the execution processor; the execution prompts the search in the input stream for a <number>
, an <optional equal>
and a suitable value (with expansion). On the contrary, in advancedimen2 by 2pt
, the token dimen
is absorbed as the argument to advance
for further processing. Non active (explicit) character tokens are not expandable; an active character can be or not, depending on how it's meaning is defined.
What tokens are expandable? Easy:
- some primitives, such as conditionals,
string
,noexpand
and others;
macros, that is, tokens defined withdef
,edef
,gdef
orxdef
;- tokens
let
equal to an expandable token.
Since your implicitToken
belongs to none of the above types, it is unexpandable.
add a comment |Â
up vote
4
down vote
Let's analyze your tests one by one.
An implicit token (more precisely an implicit character token) is a symbolic token defined by let<token>=<character>
.
Test 2
% test-case 2
% Output: "b"
% Expansion is enabled, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup
This outputs âÂÂbâÂÂ, because an implicit token behaves like the character it is defined to be equal to in the context of if
or ifcat
.
Test 3
% test-case 3
% Compile error: cq is undefined
% Expansion is disabled, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup
This is easier: the assignment defimplicitToken
overrides the let
. On the other hand, implicitToken
is never used and cq
(undefined) raises an error.
Test 4
% test-case 4
% Output: "B"
% Expansion is enabled, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup
If you add showcq
after doing the edef
you'll see
> cq=macro:
->implicitToken .
and the output will contain a âÂÂBâÂÂ, because of the subsequent redefinition.
An implicit token is not expandable, so the codes
letimplicitToken=b
edefcqimplicitToken
and
letimplicitToken=b
defcqimplicitToken
are completely equivalent. TeX will use the meaning of implicitToken
current at expansion of cq
time: in the case of your test it is âÂÂBâÂÂ.
Test 1
% test-case 1
% Output: "b"
% Expansion is disabled, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup
The primitive uppercase
examines its argument (without macro expansion, but it's irrelevant here) and transforms each explicit character token into its uppercase form (as stored in the uccode
vector). It doesn't act on implicit tokens.
Comments
There is an asymmetry: with letimplicitToken=b
, the test ifimplicitToken b
will return true. Similarly ifcat
will use the category code of the character current at definition time; for instance
letimplicitToken=b
catcode`b=12
ifcatimplicitToken a%
messageletter%
else
messagenonletter%
fi
will print letter
on the console, whereas ifcat ba
would print nonletter
.
Why this asymmetry? Sorry, but only Donald Knuth can answer. The most prominent usage of implicit tokens is with bgroup
and egroup
, which can be used as substitutes for and
in certain (not all) contexts. I guess that making them subject to transformations in
uppercase
or lowercase
would be irksome.
Expansion
Tokens in TeX can be expandable or unexpandable. A token of the latter kind, when found alone, is directly passed to the execution processor. What does this mean? Let's see an example. The primitive dimen
is not expandable. If TeX finds dimen2=12pt
, it will pass dimen
to the execution processor; the execution prompts the search in the input stream for a <number>
, an <optional equal>
and a suitable value (with expansion). On the contrary, in advancedimen2 by 2pt
, the token dimen
is absorbed as the argument to advance
for further processing. Non active (explicit) character tokens are not expandable; an active character can be or not, depending on how it's meaning is defined.
What tokens are expandable? Easy:
- some primitives, such as conditionals,
string
,noexpand
and others;
macros, that is, tokens defined withdef
,edef
,gdef
orxdef
;- tokens
let
equal to an expandable token.
Since your implicitToken
belongs to none of the above types, it is unexpandable.
add a comment |Â
up vote
4
down vote
up vote
4
down vote
Let's analyze your tests one by one.
An implicit token (more precisely an implicit character token) is a symbolic token defined by let<token>=<character>
.
Test 2
% test-case 2
% Output: "b"
% Expansion is enabled, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup
This outputs âÂÂbâÂÂ, because an implicit token behaves like the character it is defined to be equal to in the context of if
or ifcat
.
Test 3
% test-case 3
% Compile error: cq is undefined
% Expansion is disabled, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup
This is easier: the assignment defimplicitToken
overrides the let
. On the other hand, implicitToken
is never used and cq
(undefined) raises an error.
Test 4
% test-case 4
% Output: "B"
% Expansion is enabled, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup
If you add showcq
after doing the edef
you'll see
> cq=macro:
->implicitToken .
and the output will contain a âÂÂBâÂÂ, because of the subsequent redefinition.
An implicit token is not expandable, so the codes
letimplicitToken=b
edefcqimplicitToken
and
letimplicitToken=b
defcqimplicitToken
are completely equivalent. TeX will use the meaning of implicitToken
current at expansion of cq
time: in the case of your test it is âÂÂBâÂÂ.
Test 1
% test-case 1
% Output: "b"
% Expansion is disabled, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup
The primitive uppercase
examines its argument (without macro expansion, but it's irrelevant here) and transforms each explicit character token into its uppercase form (as stored in the uccode
vector). It doesn't act on implicit tokens.
Comments
There is an asymmetry: with letimplicitToken=b
, the test ifimplicitToken b
will return true. Similarly ifcat
will use the category code of the character current at definition time; for instance
letimplicitToken=b
catcode`b=12
ifcatimplicitToken a%
messageletter%
else
messagenonletter%
fi
will print letter
on the console, whereas ifcat ba
would print nonletter
.
Why this asymmetry? Sorry, but only Donald Knuth can answer. The most prominent usage of implicit tokens is with bgroup
and egroup
, which can be used as substitutes for and
in certain (not all) contexts. I guess that making them subject to transformations in
uppercase
or lowercase
would be irksome.
Expansion
Tokens in TeX can be expandable or unexpandable. A token of the latter kind, when found alone, is directly passed to the execution processor. What does this mean? Let's see an example. The primitive dimen
is not expandable. If TeX finds dimen2=12pt
, it will pass dimen
to the execution processor; the execution prompts the search in the input stream for a <number>
, an <optional equal>
and a suitable value (with expansion). On the contrary, in advancedimen2 by 2pt
, the token dimen
is absorbed as the argument to advance
for further processing. Non active (explicit) character tokens are not expandable; an active character can be or not, depending on how it's meaning is defined.
What tokens are expandable? Easy:
- some primitives, such as conditionals,
string
,noexpand
and others;
macros, that is, tokens defined withdef
,edef
,gdef
orxdef
;- tokens
let
equal to an expandable token.
Since your implicitToken
belongs to none of the above types, it is unexpandable.
Let's analyze your tests one by one.
An implicit token (more precisely an implicit character token) is a symbolic token defined by let<token>=<character>
.
Test 2
% test-case 2
% Output: "b"
% Expansion is enabled, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup
This outputs âÂÂbâÂÂ, because an implicit token behaves like the character it is defined to be equal to in the context of if
or ifcat
.
Test 3
% test-case 3
% Compile error: cq is undefined
% Expansion is disabled, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup
This is easier: the assignment defimplicitToken
overrides the let
. On the other hand, implicitToken
is never used and cq
(undefined) raises an error.
Test 4
% test-case 4
% Output: "B"
% Expansion is enabled, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup
If you add showcq
after doing the edef
you'll see
> cq=macro:
->implicitToken .
and the output will contain a âÂÂBâÂÂ, because of the subsequent redefinition.
An implicit token is not expandable, so the codes
letimplicitToken=b
edefcqimplicitToken
and
letimplicitToken=b
defcqimplicitToken
are completely equivalent. TeX will use the meaning of implicitToken
current at expansion of cq
time: in the case of your test it is âÂÂBâÂÂ.
Test 1
% test-case 1
% Output: "b"
% Expansion is disabled, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup
The primitive uppercase
examines its argument (without macro expansion, but it's irrelevant here) and transforms each explicit character token into its uppercase form (as stored in the uccode
vector). It doesn't act on implicit tokens.
Comments
There is an asymmetry: with letimplicitToken=b
, the test ifimplicitToken b
will return true. Similarly ifcat
will use the category code of the character current at definition time; for instance
letimplicitToken=b
catcode`b=12
ifcatimplicitToken a%
messageletter%
else
messagenonletter%
fi
will print letter
on the console, whereas ifcat ba
would print nonletter
.
Why this asymmetry? Sorry, but only Donald Knuth can answer. The most prominent usage of implicit tokens is with bgroup
and egroup
, which can be used as substitutes for and
in certain (not all) contexts. I guess that making them subject to transformations in
uppercase
or lowercase
would be irksome.
Expansion
Tokens in TeX can be expandable or unexpandable. A token of the latter kind, when found alone, is directly passed to the execution processor. What does this mean? Let's see an example. The primitive dimen
is not expandable. If TeX finds dimen2=12pt
, it will pass dimen
to the execution processor; the execution prompts the search in the input stream for a <number>
, an <optional equal>
and a suitable value (with expansion). On the contrary, in advancedimen2 by 2pt
, the token dimen
is absorbed as the argument to advance
for further processing. Non active (explicit) character tokens are not expandable; an active character can be or not, depending on how it's meaning is defined.
What tokens are expandable? Easy:
- some primitives, such as conditionals,
string
,noexpand
and others;
macros, that is, tokens defined withdef
,edef
,gdef
orxdef
;- tokens
let
equal to an expandable token.
Since your implicitToken
belongs to none of the above types, it is unexpandable.
answered Sep 6 at 9:32
egreg
681k8318103058
681k8318103058
add a comment |Â
add a comment |Â
Leslie Leigh is a new contributor. Be nice, and check out our Code of Conduct.
Leslie Leigh is a new contributor. Be nice, and check out our Code of Conduct.
Leslie Leigh is a new contributor. Be nice, and check out our Code of Conduct.
Leslie Leigh is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2ftex.stackexchange.com%2fquestions%2f449597%2fwhen-an-implicit-token-is-allowed%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Welcome to TeX.SX! Yes, that's the interesting thing about expansion and
uppercase
is not expandable in itself.â TeXnician
Sep 6 at 6:04
in test-case 4, you let
implicitToken
tob
which is not-expandable, soimplicitToken
does not expand in anedef
. Then you redefine its meaning so the new meaning is used.â jfbu
Sep 6 at 8:57
I hope my answer is clear but to emphasis that implicit character tokens are not expanded to their meaning, they are equal to the character token. The classic example is
letbgroup= letegroup=
.â Joseph Wrightâ¦
Sep 6 at 8:58
you can achieve same as test-case 4 with
letimplicitTokenignorespaces
then do theedef
then doletimplicitTokenhrule
and executecq
to see the rule. (if you try this test in isolation, beware there should also be something else printed on the page for thehrule
to show...)â jfbu
Sep 6 at 8:58
there is no expansion in any of your examples except one step
cq
toimplicitToken
â David Carlisle
Sep 6 at 9:06