When an implicit token is allowed?

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
7
down vote

favorite












Example: the latex code:



documentclassarticle
begindocument

% test-case 1
% Output: "b"
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup

% test-case 2
% Output: "b"
% Expansion is allowed, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup

% test-case 3
% Compile error: cq is undefined
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup

% test-case 4
% Output: "B"
% Expansion is allowed, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup

enddocument


Here are my questions:




  • Whether the implicit token is allowed if expansion is allowed in the context?



    This might be the only way to explain the test-case 1, 2, 3.But it can not explain test-case 4.



  • If not, when it is allowed?



I have read the TeXByTopic but I can't find a certain conclusion.







share|improve this question









New contributor




Leslie Leigh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.


















  • Welcome to TeX.SX! Yes, that's the interesting thing about expansion and uppercase is not expandable in itself.
    – TeXnician
    Sep 6 at 6:04










  • in test-case 4, you let implicitToken to b which is not-expandable, so implicitToken does not expand in an edef. Then you redefine its meaning so the new meaning is used.
    – jfbu
    Sep 6 at 8:57










  • I hope my answer is clear but to emphasis that implicit character tokens are not expanded to their meaning, they are equal to the character token. The classic example is letbgroup= letegroup=.
    – Joseph Wright♦
    Sep 6 at 8:58










  • you can achieve same as test-case 4 with letimplicitTokenignorespaces then do the edef then do letimplicitTokenhrule and execute cq to see the rule. (if you try this test in isolation, beware there should also be something else printed on the page for the hrule to show...)
    – jfbu
    Sep 6 at 8:58











  • there is no expansion in any of your examples except one step cq to implicitToken
    – David Carlisle
    Sep 6 at 9:06














up vote
7
down vote

favorite












Example: the latex code:



documentclassarticle
begindocument

% test-case 1
% Output: "b"
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup

% test-case 2
% Output: "b"
% Expansion is allowed, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup

% test-case 3
% Compile error: cq is undefined
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup

% test-case 4
% Output: "B"
% Expansion is allowed, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup

enddocument


Here are my questions:




  • Whether the implicit token is allowed if expansion is allowed in the context?



    This might be the only way to explain the test-case 1, 2, 3.But it can not explain test-case 4.



  • If not, when it is allowed?



I have read the TeXByTopic but I can't find a certain conclusion.







share|improve this question









New contributor




Leslie Leigh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.


















  • Welcome to TeX.SX! Yes, that's the interesting thing about expansion and uppercase is not expandable in itself.
    – TeXnician
    Sep 6 at 6:04










  • in test-case 4, you let implicitToken to b which is not-expandable, so implicitToken does not expand in an edef. Then you redefine its meaning so the new meaning is used.
    – jfbu
    Sep 6 at 8:57










  • I hope my answer is clear but to emphasis that implicit character tokens are not expanded to their meaning, they are equal to the character token. The classic example is letbgroup= letegroup=.
    – Joseph Wright♦
    Sep 6 at 8:58










  • you can achieve same as test-case 4 with letimplicitTokenignorespaces then do the edef then do letimplicitTokenhrule and execute cq to see the rule. (if you try this test in isolation, beware there should also be something else printed on the page for the hrule to show...)
    – jfbu
    Sep 6 at 8:58











  • there is no expansion in any of your examples except one step cq to implicitToken
    – David Carlisle
    Sep 6 at 9:06












up vote
7
down vote

favorite









up vote
7
down vote

favorite











Example: the latex code:



documentclassarticle
begindocument

% test-case 1
% Output: "b"
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup

% test-case 2
% Output: "b"
% Expansion is allowed, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup

% test-case 3
% Compile error: cq is undefined
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup

% test-case 4
% Output: "B"
% Expansion is allowed, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup

enddocument


Here are my questions:




  • Whether the implicit token is allowed if expansion is allowed in the context?



    This might be the only way to explain the test-case 1, 2, 3.But it can not explain test-case 4.



  • If not, when it is allowed?



I have read the TeXByTopic but I can't find a certain conclusion.







share|improve this question









New contributor




Leslie Leigh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.










Example: the latex code:



documentclassarticle
begindocument

% test-case 1
% Output: "b"
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup

% test-case 2
% Output: "b"
% Expansion is allowed, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup

% test-case 3
% Compile error: cq is undefined
% Expansion is not allowed, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup

% test-case 4
% Output: "B"
% Expansion is allowed, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup

enddocument


Here are my questions:




  • Whether the implicit token is allowed if expansion is allowed in the context?



    This might be the only way to explain the test-case 1, 2, 3.But it can not explain test-case 4.



  • If not, when it is allowed?



I have read the TeXByTopic but I can't find a certain conclusion.









share|improve this question









New contributor




Leslie Leigh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









share|improve this question




share|improve this question








edited Sep 6 at 8:56





















New contributor




Leslie Leigh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









asked Sep 6 at 3:45









Leslie Leigh

384




384




New contributor




Leslie Leigh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.





New contributor





Leslie Leigh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.






Leslie Leigh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











  • Welcome to TeX.SX! Yes, that's the interesting thing about expansion and uppercase is not expandable in itself.
    – TeXnician
    Sep 6 at 6:04










  • in test-case 4, you let implicitToken to b which is not-expandable, so implicitToken does not expand in an edef. Then you redefine its meaning so the new meaning is used.
    – jfbu
    Sep 6 at 8:57










  • I hope my answer is clear but to emphasis that implicit character tokens are not expanded to their meaning, they are equal to the character token. The classic example is letbgroup= letegroup=.
    – Joseph Wright♦
    Sep 6 at 8:58










  • you can achieve same as test-case 4 with letimplicitTokenignorespaces then do the edef then do letimplicitTokenhrule and execute cq to see the rule. (if you try this test in isolation, beware there should also be something else printed on the page for the hrule to show...)
    – jfbu
    Sep 6 at 8:58











  • there is no expansion in any of your examples except one step cq to implicitToken
    – David Carlisle
    Sep 6 at 9:06
















  • Welcome to TeX.SX! Yes, that's the interesting thing about expansion and uppercase is not expandable in itself.
    – TeXnician
    Sep 6 at 6:04










  • in test-case 4, you let implicitToken to b which is not-expandable, so implicitToken does not expand in an edef. Then you redefine its meaning so the new meaning is used.
    – jfbu
    Sep 6 at 8:57










  • I hope my answer is clear but to emphasis that implicit character tokens are not expanded to their meaning, they are equal to the character token. The classic example is letbgroup= letegroup=.
    – Joseph Wright♦
    Sep 6 at 8:58










  • you can achieve same as test-case 4 with letimplicitTokenignorespaces then do the edef then do letimplicitTokenhrule and execute cq to see the rule. (if you try this test in isolation, beware there should also be something else printed on the page for the hrule to show...)
    – jfbu
    Sep 6 at 8:58











  • there is no expansion in any of your examples except one step cq to implicitToken
    – David Carlisle
    Sep 6 at 9:06















Welcome to TeX.SX! Yes, that's the interesting thing about expansion and uppercase is not expandable in itself.
– TeXnician
Sep 6 at 6:04




Welcome to TeX.SX! Yes, that's the interesting thing about expansion and uppercase is not expandable in itself.
– TeXnician
Sep 6 at 6:04












in test-case 4, you let implicitToken to b which is not-expandable, so implicitToken does not expand in an edef. Then you redefine its meaning so the new meaning is used.
– jfbu
Sep 6 at 8:57




in test-case 4, you let implicitToken to b which is not-expandable, so implicitToken does not expand in an edef. Then you redefine its meaning so the new meaning is used.
– jfbu
Sep 6 at 8:57












I hope my answer is clear but to emphasis that implicit character tokens are not expanded to their meaning, they are equal to the character token. The classic example is letbgroup= letegroup=.
– Joseph Wright♦
Sep 6 at 8:58




I hope my answer is clear but to emphasis that implicit character tokens are not expanded to their meaning, they are equal to the character token. The classic example is letbgroup= letegroup=.
– Joseph Wright♦
Sep 6 at 8:58












you can achieve same as test-case 4 with letimplicitTokenignorespaces then do the edef then do letimplicitTokenhrule and execute cq to see the rule. (if you try this test in isolation, beware there should also be something else printed on the page for the hrule to show...)
– jfbu
Sep 6 at 8:58





you can achieve same as test-case 4 with letimplicitTokenignorespaces then do the edef then do letimplicitTokenhrule and execute cq to see the rule. (if you try this test in isolation, beware there should also be something else printed on the page for the hrule to show...)
– jfbu
Sep 6 at 8:58













there is no expansion in any of your examples except one step cq to implicitToken
– David Carlisle
Sep 6 at 9:06




there is no expansion in any of your examples except one step cq to implicitToken
– David Carlisle
Sep 6 at 9:06










2 Answers
2






active

oldest

votes

















up vote
4
down vote



accepted










Implicit tokens are not expanded to their meaning, they are their meaning. So with



lettokenAlias=b


tokenAlias is an b. That shows up in your second test: tokenAlias and b are entirely equivalent.



However, case changing takes place based on the character code of the token concerned. (As noted in a comment, uppercase is also not expandable.) Thus tokenAlias is not affected by uppercase as it does not have a character code at all, being a control sequence.



The only way to get implicit character tokens to 'appear naturally' is to iterate over input and use a test on the meaning of each one, replacing every implicit token by it's meaning.




Taking the tests one at a time (following the edit), no expansion takes place in any cases. Thus:



  1. implicitToken remains unchanged as it does not have a character code in the upper case table


  2. implicitToken is b, so comparing the two using if is true: there is no expansion of implicitToken.


  3. cq is never defined, and so the let and def lines have no effect


  4. The edef does nothing as implicitToken is not expanded. Rather, we end up with cq with replacement text implicitToken: the effect this has thus depends on the meaning at point-of-use.


Note (again) that typesetting is not an expansion context: the result depends on the execution of the token.






share|improve this answer






















  • uppercase is also not expandable. So tokenAlias->l is an expansion?
    – Leslie Leigh
    Sep 6 at 7:36










  • @LeslieLeigh tokenAlias is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, where tokenAlias is l, which then produces the output l. That's not expansion.
    – Joseph Wright♦
    Sep 6 at 7:41










  • Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it?
    – Leslie Leigh
    Sep 6 at 9:03











  • @LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for example uppercaseb is the token B but there is no token other than implicitToken that uppercaseimplicitToken could be defined to be.
    – David Carlisle
    Sep 6 at 9:09










  • @LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic for uppercase for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise).
    – Joseph Wright♦
    Sep 6 at 11:46

















up vote
4
down vote













Let's analyze your tests one by one.



An implicit token (more precisely an implicit character token) is a symbolic token defined by let<token>=<character>.



Test 2



% test-case 2
% Output: "b"
% Expansion is enabled, and implicit token takes effect.
begingroup
letimplicitToken=b
if implicitToken b
b
else
B
fi
endgroup


This outputs “b”, because an implicit token behaves like the character it is defined to be equal to in the context of if or ifcat.



Test 3



% test-case 3
% Compile error: cq is undefined
% Expansion is disabled, and implicit token doesn't take effect.
begingroup
letimplicitTokencq
defimplicitToken
cq
endgroup


This is easier: the assignment defimplicitToken overrides the let. On the other hand, implicitToken is never used and cq (undefined) raises an error.



Test 4



% test-case 4
% Output: "B"
% Expansion is enabled, but implicit token doesn't take effect.
begingroup
letimplicitToken=b
edefcqimplicitToken
letimplicitToken=B
cq
endgroup


If you add showcq after doing the edef you'll see



> cq=macro:
->implicitToken .


and the output will contain a “B”, because of the subsequent redefinition.



An implicit token is not expandable, so the codes



letimplicitToken=b
edefcqimplicitToken


and



letimplicitToken=b
defcqimplicitToken


are completely equivalent. TeX will use the meaning of implicitToken current at expansion of cq time: in the case of your test it is “B”.



Test 1



% test-case 1
% Output: "b"
% Expansion is disabled, and implicit token doesn't take effect.
begingroup
letimplicitToken=b
uppercaseimplicitToken
endgroup


The primitive uppercase examines its argument (without macro expansion, but it's irrelevant here) and transforms each explicit character token into its uppercase form (as stored in the uccode vector). It doesn't act on implicit tokens.



Comments



There is an asymmetry: with letimplicitToken=b, the test ifimplicitToken b will return true. Similarly ifcat will use the category code of the character current at definition time; for instance



letimplicitToken=b
catcode`b=12
ifcatimplicitToken a%
messageletter%
else
messagenonletter%
fi


will print letter on the console, whereas ifcat ba would print nonletter.



Why this asymmetry? Sorry, but only Donald Knuth can answer. The most prominent usage of implicit tokens is with bgroup and egroup, which can be used as substitutes for and in certain (not all) contexts. I guess that making them subject to transformations in uppercase or lowercase would be irksome.



Expansion



Tokens in TeX can be expandable or unexpandable. A token of the latter kind, when found alone, is directly passed to the execution processor. What does this mean? Let's see an example. The primitive dimen is not expandable. If TeX finds dimen2=12pt, it will pass dimen to the execution processor; the execution prompts the search in the input stream for a <number>, an <optional equal> and a suitable value (with expansion). On the contrary, in advancedimen2 by 2pt, the token dimen is absorbed as the argument to advance for further processing. Non active (explicit) character tokens are not expandable; an active character can be or not, depending on how it's meaning is defined.



What tokens are expandable? Easy:



  1. some primitives, such as conditionals, string, noexpand and others;


  2. macros, that is, tokens defined with def, edef, gdef or xdef;

  3. tokens let equal to an expandable token.

Since your implicitToken belongs to none of the above types, it is unexpandable.






share|improve this answer




















    Your Answer







    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "85"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    convertImagesToLinks: false,
    noModals: false,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );






    Leslie Leigh is a new contributor. Be nice, and check out our Code of Conduct.









     

    draft saved


    draft discarded


















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2ftex.stackexchange.com%2fquestions%2f449597%2fwhen-an-implicit-token-is-allowed%23new-answer', 'question_page');

    );

    Post as a guest






























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes








    up vote
    4
    down vote



    accepted










    Implicit tokens are not expanded to their meaning, they are their meaning. So with



    lettokenAlias=b


    tokenAlias is an b. That shows up in your second test: tokenAlias and b are entirely equivalent.



    However, case changing takes place based on the character code of the token concerned. (As noted in a comment, uppercase is also not expandable.) Thus tokenAlias is not affected by uppercase as it does not have a character code at all, being a control sequence.



    The only way to get implicit character tokens to 'appear naturally' is to iterate over input and use a test on the meaning of each one, replacing every implicit token by it's meaning.




    Taking the tests one at a time (following the edit), no expansion takes place in any cases. Thus:



    1. implicitToken remains unchanged as it does not have a character code in the upper case table


    2. implicitToken is b, so comparing the two using if is true: there is no expansion of implicitToken.


    3. cq is never defined, and so the let and def lines have no effect


    4. The edef does nothing as implicitToken is not expanded. Rather, we end up with cq with replacement text implicitToken: the effect this has thus depends on the meaning at point-of-use.


    Note (again) that typesetting is not an expansion context: the result depends on the execution of the token.






    share|improve this answer






















    • uppercase is also not expandable. So tokenAlias->l is an expansion?
      – Leslie Leigh
      Sep 6 at 7:36










    • @LeslieLeigh tokenAlias is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, where tokenAlias is l, which then produces the output l. That's not expansion.
      – Joseph Wright♦
      Sep 6 at 7:41










    • Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it?
      – Leslie Leigh
      Sep 6 at 9:03











    • @LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for example uppercaseb is the token B but there is no token other than implicitToken that uppercaseimplicitToken could be defined to be.
      – David Carlisle
      Sep 6 at 9:09










    • @LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic for uppercase for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise).
      – Joseph Wright♦
      Sep 6 at 11:46














    up vote
    4
    down vote



    accepted










    Implicit tokens are not expanded to their meaning, they are their meaning. So with



    lettokenAlias=b


    tokenAlias is an b. That shows up in your second test: tokenAlias and b are entirely equivalent.



    However, case changing takes place based on the character code of the token concerned. (As noted in a comment, uppercase is also not expandable.) Thus tokenAlias is not affected by uppercase as it does not have a character code at all, being a control sequence.



    The only way to get implicit character tokens to 'appear naturally' is to iterate over input and use a test on the meaning of each one, replacing every implicit token by it's meaning.




    Taking the tests one at a time (following the edit), no expansion takes place in any cases. Thus:



    1. implicitToken remains unchanged as it does not have a character code in the upper case table


    2. implicitToken is b, so comparing the two using if is true: there is no expansion of implicitToken.


    3. cq is never defined, and so the let and def lines have no effect


    4. The edef does nothing as implicitToken is not expanded. Rather, we end up with cq with replacement text implicitToken: the effect this has thus depends on the meaning at point-of-use.


    Note (again) that typesetting is not an expansion context: the result depends on the execution of the token.






    share|improve this answer






















    • uppercase is also not expandable. So tokenAlias->l is an expansion?
      – Leslie Leigh
      Sep 6 at 7:36










    • @LeslieLeigh tokenAlias is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, where tokenAlias is l, which then produces the output l. That's not expansion.
      – Joseph Wright♦
      Sep 6 at 7:41










    • Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it?
      – Leslie Leigh
      Sep 6 at 9:03











    • @LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for example uppercaseb is the token B but there is no token other than implicitToken that uppercaseimplicitToken could be defined to be.
      – David Carlisle
      Sep 6 at 9:09










    • @LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic for uppercase for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise).
      – Joseph Wright♦
      Sep 6 at 11:46












    up vote
    4
    down vote



    accepted







    up vote
    4
    down vote



    accepted






    Implicit tokens are not expanded to their meaning, they are their meaning. So with



    lettokenAlias=b


    tokenAlias is an b. That shows up in your second test: tokenAlias and b are entirely equivalent.



    However, case changing takes place based on the character code of the token concerned. (As noted in a comment, uppercase is also not expandable.) Thus tokenAlias is not affected by uppercase as it does not have a character code at all, being a control sequence.



    The only way to get implicit character tokens to 'appear naturally' is to iterate over input and use a test on the meaning of each one, replacing every implicit token by it's meaning.




    Taking the tests one at a time (following the edit), no expansion takes place in any cases. Thus:



    1. implicitToken remains unchanged as it does not have a character code in the upper case table


    2. implicitToken is b, so comparing the two using if is true: there is no expansion of implicitToken.


    3. cq is never defined, and so the let and def lines have no effect


    4. The edef does nothing as implicitToken is not expanded. Rather, we end up with cq with replacement text implicitToken: the effect this has thus depends on the meaning at point-of-use.


    Note (again) that typesetting is not an expansion context: the result depends on the execution of the token.






    share|improve this answer














    Implicit tokens are not expanded to their meaning, they are their meaning. So with



    lettokenAlias=b


    tokenAlias is an b. That shows up in your second test: tokenAlias and b are entirely equivalent.



    However, case changing takes place based on the character code of the token concerned. (As noted in a comment, uppercase is also not expandable.) Thus tokenAlias is not affected by uppercase as it does not have a character code at all, being a control sequence.



    The only way to get implicit character tokens to 'appear naturally' is to iterate over input and use a test on the meaning of each one, replacing every implicit token by it's meaning.




    Taking the tests one at a time (following the edit), no expansion takes place in any cases. Thus:



    1. implicitToken remains unchanged as it does not have a character code in the upper case table


    2. implicitToken is b, so comparing the two using if is true: there is no expansion of implicitToken.


    3. cq is never defined, and so the let and def lines have no effect


    4. The edef does nothing as implicitToken is not expanded. Rather, we end up with cq with replacement text implicitToken: the effect this has thus depends on the meaning at point-of-use.


    Note (again) that typesetting is not an expansion context: the result depends on the execution of the token.







    share|improve this answer














    share|improve this answer



    share|improve this answer








    edited Sep 6 at 8:57

























    answered Sep 6 at 6:35









    Joseph Wright♦

    196k21540860




    196k21540860











    • uppercase is also not expandable. So tokenAlias->l is an expansion?
      – Leslie Leigh
      Sep 6 at 7:36










    • @LeslieLeigh tokenAlias is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, where tokenAlias is l, which then produces the output l. That's not expansion.
      – Joseph Wright♦
      Sep 6 at 7:41










    • Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it?
      – Leslie Leigh
      Sep 6 at 9:03











    • @LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for example uppercaseb is the token B but there is no token other than implicitToken that uppercaseimplicitToken could be defined to be.
      – David Carlisle
      Sep 6 at 9:09










    • @LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic for uppercase for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise).
      – Joseph Wright♦
      Sep 6 at 11:46
















    • uppercase is also not expandable. So tokenAlias->l is an expansion?
      – Leslie Leigh
      Sep 6 at 7:36










    • @LeslieLeigh tokenAlias is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, where tokenAlias is l, which then produces the output l. That's not expansion.
      – Joseph Wright♦
      Sep 6 at 7:41










    • Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it?
      – Leslie Leigh
      Sep 6 at 9:03











    • @LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for example uppercaseb is the token B but there is no token other than implicitToken that uppercaseimplicitToken could be defined to be.
      – David Carlisle
      Sep 6 at 9:09










    • @LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic for uppercase for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise).
      – Joseph Wright♦
      Sep 6 at 11:46















    uppercase is also not expandable. So tokenAlias->l is an expansion?
    – Leslie Leigh
    Sep 6 at 7:36




    uppercase is also not expandable. So tokenAlias->l is an expansion?
    – Leslie Leigh
    Sep 6 at 7:36












    @LeslieLeigh tokenAlias is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, where tokenAlias is l, which then produces the output l. That's not expansion.
    – Joseph Wright♦
    Sep 6 at 7:41




    @LeslieLeigh tokenAlias is never expanded, as it's an implicit token. I think you mean for example when it's used in a typesetting context, where tokenAlias is l, which then produces the output l. That's not expansion.
    – Joseph Wright♦
    Sep 6 at 7:41












    Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it?
    – Leslie Leigh
    Sep 6 at 9:03





    Hi Joseph, I have admitted your words 'implicit tokens don't expand', but why implicit tokens sometimes take effect, sometimes don't? How do I decide it?
    – Leslie Leigh
    Sep 6 at 9:03













    @LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for example uppercaseb is the token B but there is no token other than implicitToken that uppercaseimplicitToken could be defined to be.
    – David Carlisle
    Sep 6 at 9:09




    @LeslieLeigh ultimately the TeXBook documents which tex primitives act on what kind of token. although it mostly followed some kind of logic, for example uppercaseb is the token B but there is no token other than implicitToken that uppercaseimplicitToken could be defined to be.
    – David Carlisle
    Sep 6 at 9:09












    @LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic for uppercase for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise).
    – Joseph Wright♦
    Sep 6 at 11:46




    @LeslieLeigh They always 'take effect', it just depends on what operation is being applied. As David has said, the logic for uppercase for example is purely about character code, so it's entirely logical that it doesn't affect a control sequence (implict char or otherwise).
    – Joseph Wright♦
    Sep 6 at 11:46










    up vote
    4
    down vote













    Let's analyze your tests one by one.



    An implicit token (more precisely an implicit character token) is a symbolic token defined by let<token>=<character>.



    Test 2



    % test-case 2
    % Output: "b"
    % Expansion is enabled, and implicit token takes effect.
    begingroup
    letimplicitToken=b
    if implicitToken b
    b
    else
    B
    fi
    endgroup


    This outputs “b”, because an implicit token behaves like the character it is defined to be equal to in the context of if or ifcat.



    Test 3



    % test-case 3
    % Compile error: cq is undefined
    % Expansion is disabled, and implicit token doesn't take effect.
    begingroup
    letimplicitTokencq
    defimplicitToken
    cq
    endgroup


    This is easier: the assignment defimplicitToken overrides the let. On the other hand, implicitToken is never used and cq (undefined) raises an error.



    Test 4



    % test-case 4
    % Output: "B"
    % Expansion is enabled, but implicit token doesn't take effect.
    begingroup
    letimplicitToken=b
    edefcqimplicitToken
    letimplicitToken=B
    cq
    endgroup


    If you add showcq after doing the edef you'll see



    > cq=macro:
    ->implicitToken .


    and the output will contain a “B”, because of the subsequent redefinition.



    An implicit token is not expandable, so the codes



    letimplicitToken=b
    edefcqimplicitToken


    and



    letimplicitToken=b
    defcqimplicitToken


    are completely equivalent. TeX will use the meaning of implicitToken current at expansion of cq time: in the case of your test it is “B”.



    Test 1



    % test-case 1
    % Output: "b"
    % Expansion is disabled, and implicit token doesn't take effect.
    begingroup
    letimplicitToken=b
    uppercaseimplicitToken
    endgroup


    The primitive uppercase examines its argument (without macro expansion, but it's irrelevant here) and transforms each explicit character token into its uppercase form (as stored in the uccode vector). It doesn't act on implicit tokens.



    Comments



    There is an asymmetry: with letimplicitToken=b, the test ifimplicitToken b will return true. Similarly ifcat will use the category code of the character current at definition time; for instance



    letimplicitToken=b
    catcode`b=12
    ifcatimplicitToken a%
    messageletter%
    else
    messagenonletter%
    fi


    will print letter on the console, whereas ifcat ba would print nonletter.



    Why this asymmetry? Sorry, but only Donald Knuth can answer. The most prominent usage of implicit tokens is with bgroup and egroup, which can be used as substitutes for and in certain (not all) contexts. I guess that making them subject to transformations in uppercase or lowercase would be irksome.



    Expansion



    Tokens in TeX can be expandable or unexpandable. A token of the latter kind, when found alone, is directly passed to the execution processor. What does this mean? Let's see an example. The primitive dimen is not expandable. If TeX finds dimen2=12pt, it will pass dimen to the execution processor; the execution prompts the search in the input stream for a <number>, an <optional equal> and a suitable value (with expansion). On the contrary, in advancedimen2 by 2pt, the token dimen is absorbed as the argument to advance for further processing. Non active (explicit) character tokens are not expandable; an active character can be or not, depending on how it's meaning is defined.



    What tokens are expandable? Easy:



    1. some primitives, such as conditionals, string, noexpand and others;


    2. macros, that is, tokens defined with def, edef, gdef or xdef;

    3. tokens let equal to an expandable token.

    Since your implicitToken belongs to none of the above types, it is unexpandable.






    share|improve this answer
























      up vote
      4
      down vote













      Let's analyze your tests one by one.



      An implicit token (more precisely an implicit character token) is a symbolic token defined by let<token>=<character>.



      Test 2



      % test-case 2
      % Output: "b"
      % Expansion is enabled, and implicit token takes effect.
      begingroup
      letimplicitToken=b
      if implicitToken b
      b
      else
      B
      fi
      endgroup


      This outputs “b”, because an implicit token behaves like the character it is defined to be equal to in the context of if or ifcat.



      Test 3



      % test-case 3
      % Compile error: cq is undefined
      % Expansion is disabled, and implicit token doesn't take effect.
      begingroup
      letimplicitTokencq
      defimplicitToken
      cq
      endgroup


      This is easier: the assignment defimplicitToken overrides the let. On the other hand, implicitToken is never used and cq (undefined) raises an error.



      Test 4



      % test-case 4
      % Output: "B"
      % Expansion is enabled, but implicit token doesn't take effect.
      begingroup
      letimplicitToken=b
      edefcqimplicitToken
      letimplicitToken=B
      cq
      endgroup


      If you add showcq after doing the edef you'll see



      > cq=macro:
      ->implicitToken .


      and the output will contain a “B”, because of the subsequent redefinition.



      An implicit token is not expandable, so the codes



      letimplicitToken=b
      edefcqimplicitToken


      and



      letimplicitToken=b
      defcqimplicitToken


      are completely equivalent. TeX will use the meaning of implicitToken current at expansion of cq time: in the case of your test it is “B”.



      Test 1



      % test-case 1
      % Output: "b"
      % Expansion is disabled, and implicit token doesn't take effect.
      begingroup
      letimplicitToken=b
      uppercaseimplicitToken
      endgroup


      The primitive uppercase examines its argument (without macro expansion, but it's irrelevant here) and transforms each explicit character token into its uppercase form (as stored in the uccode vector). It doesn't act on implicit tokens.



      Comments



      There is an asymmetry: with letimplicitToken=b, the test ifimplicitToken b will return true. Similarly ifcat will use the category code of the character current at definition time; for instance



      letimplicitToken=b
      catcode`b=12
      ifcatimplicitToken a%
      messageletter%
      else
      messagenonletter%
      fi


      will print letter on the console, whereas ifcat ba would print nonletter.



      Why this asymmetry? Sorry, but only Donald Knuth can answer. The most prominent usage of implicit tokens is with bgroup and egroup, which can be used as substitutes for and in certain (not all) contexts. I guess that making them subject to transformations in uppercase or lowercase would be irksome.



      Expansion



      Tokens in TeX can be expandable or unexpandable. A token of the latter kind, when found alone, is directly passed to the execution processor. What does this mean? Let's see an example. The primitive dimen is not expandable. If TeX finds dimen2=12pt, it will pass dimen to the execution processor; the execution prompts the search in the input stream for a <number>, an <optional equal> and a suitable value (with expansion). On the contrary, in advancedimen2 by 2pt, the token dimen is absorbed as the argument to advance for further processing. Non active (explicit) character tokens are not expandable; an active character can be or not, depending on how it's meaning is defined.



      What tokens are expandable? Easy:



      1. some primitives, such as conditionals, string, noexpand and others;


      2. macros, that is, tokens defined with def, edef, gdef or xdef;

      3. tokens let equal to an expandable token.

      Since your implicitToken belongs to none of the above types, it is unexpandable.






      share|improve this answer






















        up vote
        4
        down vote










        up vote
        4
        down vote









        Let's analyze your tests one by one.



        An implicit token (more precisely an implicit character token) is a symbolic token defined by let<token>=<character>.



        Test 2



        % test-case 2
        % Output: "b"
        % Expansion is enabled, and implicit token takes effect.
        begingroup
        letimplicitToken=b
        if implicitToken b
        b
        else
        B
        fi
        endgroup


        This outputs “b”, because an implicit token behaves like the character it is defined to be equal to in the context of if or ifcat.



        Test 3



        % test-case 3
        % Compile error: cq is undefined
        % Expansion is disabled, and implicit token doesn't take effect.
        begingroup
        letimplicitTokencq
        defimplicitToken
        cq
        endgroup


        This is easier: the assignment defimplicitToken overrides the let. On the other hand, implicitToken is never used and cq (undefined) raises an error.



        Test 4



        % test-case 4
        % Output: "B"
        % Expansion is enabled, but implicit token doesn't take effect.
        begingroup
        letimplicitToken=b
        edefcqimplicitToken
        letimplicitToken=B
        cq
        endgroup


        If you add showcq after doing the edef you'll see



        > cq=macro:
        ->implicitToken .


        and the output will contain a “B”, because of the subsequent redefinition.



        An implicit token is not expandable, so the codes



        letimplicitToken=b
        edefcqimplicitToken


        and



        letimplicitToken=b
        defcqimplicitToken


        are completely equivalent. TeX will use the meaning of implicitToken current at expansion of cq time: in the case of your test it is “B”.



        Test 1



        % test-case 1
        % Output: "b"
        % Expansion is disabled, and implicit token doesn't take effect.
        begingroup
        letimplicitToken=b
        uppercaseimplicitToken
        endgroup


        The primitive uppercase examines its argument (without macro expansion, but it's irrelevant here) and transforms each explicit character token into its uppercase form (as stored in the uccode vector). It doesn't act on implicit tokens.



        Comments



        There is an asymmetry: with letimplicitToken=b, the test ifimplicitToken b will return true. Similarly ifcat will use the category code of the character current at definition time; for instance



        letimplicitToken=b
        catcode`b=12
        ifcatimplicitToken a%
        messageletter%
        else
        messagenonletter%
        fi


        will print letter on the console, whereas ifcat ba would print nonletter.



        Why this asymmetry? Sorry, but only Donald Knuth can answer. The most prominent usage of implicit tokens is with bgroup and egroup, which can be used as substitutes for and in certain (not all) contexts. I guess that making them subject to transformations in uppercase or lowercase would be irksome.



        Expansion



        Tokens in TeX can be expandable or unexpandable. A token of the latter kind, when found alone, is directly passed to the execution processor. What does this mean? Let's see an example. The primitive dimen is not expandable. If TeX finds dimen2=12pt, it will pass dimen to the execution processor; the execution prompts the search in the input stream for a <number>, an <optional equal> and a suitable value (with expansion). On the contrary, in advancedimen2 by 2pt, the token dimen is absorbed as the argument to advance for further processing. Non active (explicit) character tokens are not expandable; an active character can be or not, depending on how it's meaning is defined.



        What tokens are expandable? Easy:



        1. some primitives, such as conditionals, string, noexpand and others;


        2. macros, that is, tokens defined with def, edef, gdef or xdef;

        3. tokens let equal to an expandable token.

        Since your implicitToken belongs to none of the above types, it is unexpandable.






        share|improve this answer












        Let's analyze your tests one by one.



        An implicit token (more precisely an implicit character token) is a symbolic token defined by let<token>=<character>.



        Test 2



        % test-case 2
        % Output: "b"
        % Expansion is enabled, and implicit token takes effect.
        begingroup
        letimplicitToken=b
        if implicitToken b
        b
        else
        B
        fi
        endgroup


        This outputs “b”, because an implicit token behaves like the character it is defined to be equal to in the context of if or ifcat.



        Test 3



        % test-case 3
        % Compile error: cq is undefined
        % Expansion is disabled, and implicit token doesn't take effect.
        begingroup
        letimplicitTokencq
        defimplicitToken
        cq
        endgroup


        This is easier: the assignment defimplicitToken overrides the let. On the other hand, implicitToken is never used and cq (undefined) raises an error.



        Test 4



        % test-case 4
        % Output: "B"
        % Expansion is enabled, but implicit token doesn't take effect.
        begingroup
        letimplicitToken=b
        edefcqimplicitToken
        letimplicitToken=B
        cq
        endgroup


        If you add showcq after doing the edef you'll see



        > cq=macro:
        ->implicitToken .


        and the output will contain a “B”, because of the subsequent redefinition.



        An implicit token is not expandable, so the codes



        letimplicitToken=b
        edefcqimplicitToken


        and



        letimplicitToken=b
        defcqimplicitToken


        are completely equivalent. TeX will use the meaning of implicitToken current at expansion of cq time: in the case of your test it is “B”.



        Test 1



        % test-case 1
        % Output: "b"
        % Expansion is disabled, and implicit token doesn't take effect.
        begingroup
        letimplicitToken=b
        uppercaseimplicitToken
        endgroup


        The primitive uppercase examines its argument (without macro expansion, but it's irrelevant here) and transforms each explicit character token into its uppercase form (as stored in the uccode vector). It doesn't act on implicit tokens.



        Comments



        There is an asymmetry: with letimplicitToken=b, the test ifimplicitToken b will return true. Similarly ifcat will use the category code of the character current at definition time; for instance



        letimplicitToken=b
        catcode`b=12
        ifcatimplicitToken a%
        messageletter%
        else
        messagenonletter%
        fi


        will print letter on the console, whereas ifcat ba would print nonletter.



        Why this asymmetry? Sorry, but only Donald Knuth can answer. The most prominent usage of implicit tokens is with bgroup and egroup, which can be used as substitutes for and in certain (not all) contexts. I guess that making them subject to transformations in uppercase or lowercase would be irksome.



        Expansion



        Tokens in TeX can be expandable or unexpandable. A token of the latter kind, when found alone, is directly passed to the execution processor. What does this mean? Let's see an example. The primitive dimen is not expandable. If TeX finds dimen2=12pt, it will pass dimen to the execution processor; the execution prompts the search in the input stream for a <number>, an <optional equal> and a suitable value (with expansion). On the contrary, in advancedimen2 by 2pt, the token dimen is absorbed as the argument to advance for further processing. Non active (explicit) character tokens are not expandable; an active character can be or not, depending on how it's meaning is defined.



        What tokens are expandable? Easy:



        1. some primitives, such as conditionals, string, noexpand and others;


        2. macros, that is, tokens defined with def, edef, gdef or xdef;

        3. tokens let equal to an expandable token.

        Since your implicitToken belongs to none of the above types, it is unexpandable.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Sep 6 at 9:32









        egreg

        681k8318103058




        681k8318103058




















            Leslie Leigh is a new contributor. Be nice, and check out our Code of Conduct.









             

            draft saved


            draft discarded


















            Leslie Leigh is a new contributor. Be nice, and check out our Code of Conduct.












            Leslie Leigh is a new contributor. Be nice, and check out our Code of Conduct.











            Leslie Leigh is a new contributor. Be nice, and check out our Code of Conduct.













             


            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2ftex.stackexchange.com%2fquestions%2f449597%2fwhen-an-implicit-token-is-allowed%23new-answer', 'question_page');

            );

            Post as a guest













































































            Comments

            Popular posts from this blog

            Long meetings (6-7 hours a day): Being “babysat” by supervisor

            Is the Concept of Multiple Fantasy Races Scientifically Flawed? [closed]

            Confectionery