Mixing
scantokens not playing nice with def or gdef etc
Clash Royale CLAN TAG#URR8PPP
up vote
2
down vote
favorite
I working in a context where I'm changing catcodes and need to rescan the tokens before proceeding to process the text. But, I'm running into problems. Even with this bare minimum code, I still have issues.
documentclassarticle
makeatletter
newcommandaeinput[1]%%
begingroup
scantokensgdefae@input::#1::%%
endgroup
ae@input
%%
makeatother
begindocument
aeinputHELLO
enddocument
Resulting in the error:
! Undefined control sequence.
aeinput ...ae@input ::#1::}endgroup ae@input
I get the same error if I write:
scantokensxdefae@input::#1::%%
Why is this not compiling?
Also, if I do something like
expandaftergdefexpandafterae@inputexpandafterscantokens::#1::%%
I get the following error:
! File ended while scanning definition of ae@input.
Even adding everyoefnoexpand doesn't help.
HOWEVER, the following does work (which makes me even more confused).
xdefae@inputscantokens::#1::%%
So, what's happening here?
Where does this question come from?
I thought I had a simple answer for inputing text within an environment for which I've already changed the catcode of . Basically, in that question I have an environment
beginspeech
inside here periods have a different catcode
endspeech
The OP for that question wanted to do something like
beginspeech
inputtext.tex
endspeech
Clearly, input is not going to work here because . has the wrong catcode. I thought I could easily remedy this (and allow the OP to continue writing inputtext.tex) by doing something like
makeatletter
newcommandaeinput[1]%%
begingroup
catcode`.=periodcatcode
scantokensgdefae@inputinput#1%%
endgroup
ae@input
%%
makeatother
An alternative approach would have been to do something like
makeatletter
newcommandaeinput[1]%%
begingroup
catcode`.=periodcatcode
scantokensgdefae@filehandle#1%%
endgroup
inputae@filehandle%%
%%
makeatother
That way, the input file would handle periods appropriately according to the speech environment.
Something like
newcommandaeinput[1]%%
begingroup
catcode`.=periodcatcode
scantokensinput#1%%
endgroup
Only superficially looks like it could work, but the input file would have not have the correct catcode for the periods anymore.
input catcodes e-tex
asked 1 hour ago
A.Ellett
34.8k1063162
 |Â
show 3 more comments
up vote
2
down vote
favorite
I working in a context where I'm changing catcodes and need to rescan the tokens before proceeding to process the text. But, I'm running into problems. Even with this bare minimum code, I still have issues.
documentclassarticle
makeatletter
newcommandaeinput[1]%%
begingroup
scantokensgdefae@input::#1::%%
endgroup
ae@input
%%
makeatother
begindocument
aeinputHELLO
enddocument
Resulting in the error:
! Undefined control sequence.
aeinput ...ae@input ::#1::}endgroup ae@input
I get the same error if I write:
scantokensxdefae@input::#1::%%
Why is this not compiling?
Also, if I do something like
expandaftergdefexpandafterae@inputexpandafterscantokens::#1::%%
I get the following error:
! File ended while scanning definition of ae@input.
Even adding everyoefnoexpand doesn't help.
HOWEVER, the following does work (which makes me even more confused).
xdefae@inputscantokens::#1::%%
So, what's happening here?
Where does this question come from?
I thought I had a simple answer for inputing text within an environment for which I've already changed the catcode of . Basically, in that question I have an environment
beginspeech
inside here periods have a different catcode
endspeech
The OP for that question wanted to do something like
beginspeech
inputtext.tex
endspeech
Clearly, input is not going to work here because . has the wrong catcode. I thought I could easily remedy this (and allow the OP to continue writing inputtext.tex) by doing something like
makeatletter
newcommandaeinput[1]%%
begingroup
catcode`.=periodcatcode
scantokensgdefae@inputinput#1%%
endgroup
ae@input
%%
makeatother
An alternative approach would have been to do something like
makeatletter
newcommandaeinput[1]%%
begingroup
catcode`.=periodcatcode
scantokensgdefae@filehandle#1%%
endgroup
inputae@filehandle%%
%%
makeatother
That way, the input file would handle periods appropriately according to the speech environment.
Something like
newcommandaeinput[1]%%
begingroup
catcode`.=periodcatcode
scantokensinput#1%%
endgroup
Only superficially looks like it could work, but the input file would have not have the correct catcode for the periods anymore.
input catcodes e-tex
asked 1 hour ago
A.Ellett
34.8k1063162
What is the purpose ofxdef? This will also fully expand#1. Why not justnewcommandaeinput[1]scantokens#1ignorespaces?
– Henri Menke
1 hour ago
@HenriMenke According to theetexdocumentationscantokensis expandable. But, let's say it's not. Why in the world wouldxdefae@inputscantokens....be the only instance for which my code compiled?
– A.Ellett
1 hour ago
@HenriMenke You changed your comment before I noticed.
– A.Ellett
1 hour ago
Have you had a look at tex.stackexchange.com/questions/117906/… ?
– Henri Menke
1 hour ago
@HenriMenke Yes, I have and I can't seem to make things work. I've updated my answer to explain why your first suggestion is not tenable.
– A.Ellett
1 hour ago
 |Â
show 3 more comments
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I working in a context where I'm changing catcodes and need to rescan the tokens before proceeding to process the text. But, I'm running into problems. Even with this bare minimum code, I still have issues.
documentclassarticle
makeatletter
newcommandaeinput[1]%%
begingroup
scantokensgdefae@input::#1::%%
endgroup
ae@input
%%
makeatother
begindocument
aeinputHELLO
enddocument
Resulting in the error:
! Undefined control sequence.
aeinput ...ae@input ::#1::}endgroup ae@input
I get the same error if I write:
scantokensxdefae@input::#1::%%
Why is this not compiling?
Also, if I do something like
expandaftergdefexpandafterae@inputexpandafterscantokens::#1::%%
I get the following error:
! File ended while scanning definition of ae@input.
Even adding everyoefnoexpand doesn't help.
HOWEVER, the following does work (which makes me even more confused).
xdefae@inputscantokens::#1::%%
So, what's happening here?
Where does this question come from?
I thought I had a simple answer for inputing text within an environment for which I've already changed the catcode of . Basically, in that question I have an environment
beginspeech
inside here periods have a different catcode
endspeech
The OP for that question wanted to do something like
beginspeech
inputtext.tex
endspeech
Clearly, input is not going to work here because . has the wrong catcode. I thought I could easily remedy this (and allow the OP to continue writing inputtext.tex) by doing something like
makeatletter
newcommandaeinput[1]%%
begingroup
catcode`.=periodcatcode
scantokensgdefae@inputinput#1%%
endgroup
ae@input
%%
makeatother
An alternative approach would have been to do something like
makeatletter
newcommandaeinput[1]%%
begingroup
catcode`.=periodcatcode
scantokensgdefae@filehandle#1%%
endgroup
inputae@filehandle%%
%%
makeatother
That way, the input file would handle periods appropriately according to the speech environment.
Something like
newcommandaeinput[1]%%
begingroup
catcode`.=periodcatcode
scantokensinput#1%%
endgroup
Only superficially looks like it could work, but the input file would have not have the correct catcode for the periods anymore.
input catcodes e-tex
asked 1 hour ago
A.Ellett
34.8k1063162
I working in a context where I'm changing catcodes and need to rescan the tokens before proceeding to process the text. But, I'm running into problems. Even with this bare minimum code, I still have issues.
documentclassarticle
makeatletter
newcommandaeinput[1]%%
begingroup
scantokensgdefae@input::#1::%%
endgroup
ae@input
%%
makeatother
begindocument
aeinputHELLO
enddocument
Resulting in the error:
! Undefined control sequence.
aeinput ...ae@input ::#1::}endgroup ae@input
I get the same error if I write:
scantokensxdefae@input::#1::%%
Why is this not compiling?
Also, if I do something like
expandaftergdefexpandafterae@inputexpandafterscantokens::#1::%%
I get the following error:
! File ended while scanning definition of ae@input.
Even adding everyoefnoexpand doesn't help.
HOWEVER, the following does work (which makes me even more confused).
xdefae@inputscantokens::#1::%%
So, what's happening here?
Where does this question come from?
I thought I had a simple answer for inputing text within an environment for which I've already changed the catcode of . Basically, in that question I have an environment
beginspeech
inside here periods have a different catcode
endspeech
The OP for that question wanted to do something like
beginspeech
inputtext.tex
endspeech
Clearly, input is not going to work here because . has the wrong catcode. I thought I could easily remedy this (and allow the OP to continue writing inputtext.tex) by doing something like
makeatletter
newcommandaeinput[1]%%
begingroup
catcode`.=periodcatcode
scantokensgdefae@inputinput#1%%
endgroup
ae@input
%%
makeatother
An alternative approach would have been to do something like
makeatletter
newcommandaeinput[1]%%
begingroup
catcode`.=periodcatcode
scantokensgdefae@filehandle#1%%
endgroup
inputae@filehandle%%
%%
makeatother
That way, the input file would handle periods appropriately according to the speech environment.
Something like
newcommandaeinput[1]%%
begingroup
catcode`.=periodcatcode
scantokensinput#1%%
endgroup
Only superficially looks like it could work, but the input file would have not have the correct catcode for the periods anymore.
input catcodes e-tex
input catcodes e-tex
asked 1 hour ago
A.Ellett
34.8k1063162
asked 1 hour ago
A.Ellett
34.8k1063162
edited 1 hour ago
asked 1 hour ago
A.Ellett
34.8k1063162
asked 1 hour ago
A.Ellett
34.8k1063162
asked 1 hour ago
A.Ellett
34.8k1063162
34.8k1063162
What is the purpose ofxdef? This will also fully expand#1. Why not justnewcommandaeinput[1]scantokens#1ignorespaces?
– Henri Menke
1 hour ago
@HenriMenke According to theetexdocumentationscantokensis expandable. But, let's say it's not. Why in the world wouldxdefae@inputscantokens....be the only instance for which my code compiled?
– A.Ellett
1 hour ago
@HenriMenke You changed your comment before I noticed.
– A.Ellett
1 hour ago
Have you had a look at tex.stackexchange.com/questions/117906/… ?
– Henri Menke
1 hour ago
@HenriMenke Yes, I have and I can't seem to make things work. I've updated my answer to explain why your first suggestion is not tenable.
– A.Ellett
1 hour ago
 |Â
show 3 more comments
What is the purpose ofxdef? This will also fully expand#1. Why not justnewcommandaeinput[1]scantokens#1ignorespaces?
– Henri Menke
1 hour ago
@HenriMenke According to theetexdocumentationscantokensis expandable. But, let's say it's not. Why in the world wouldxdefae@inputscantokens....be the only instance for which my code compiled?
– A.Ellett
1 hour ago
@HenriMenke You changed your comment before I noticed.
– A.Ellett
1 hour ago
Have you had a look at tex.stackexchange.com/questions/117906/… ?
– Henri Menke
1 hour ago
@HenriMenke Yes, I have and I can't seem to make things work. I've updated my answer to explain why your first suggestion is not tenable.
– A.Ellett
1 hour ago
What is the purpose of
xdef? This will also fully expand #1. Why not just newcommandaeinput[1]scantokens#1ignorespaces?– Henri Menke
1 hour ago
What is the purpose of
xdef? This will also fully expand #1. Why not just newcommandaeinput[1]scantokens#1ignorespaces?– Henri Menke
1 hour ago
@HenriMenke According to the
etex documentation scantokens is expandable. But, let's say it's not. Why in the world would xdefae@inputscantokens.... be the only instance for which my code compiled?– A.Ellett
1 hour ago
@HenriMenke According to the
etex documentation scantokens is expandable. But, let's say it's not. Why in the world would xdefae@inputscantokens.... be the only instance for which my code compiled?– A.Ellett
1 hour ago
@HenriMenke You changed your comment before I noticed.
– A.Ellett
1 hour ago
@HenriMenke You changed your comment before I noticed.
– A.Ellett
1 hour ago
Have you had a look at tex.stackexchange.com/questions/117906/… ?
– Henri Menke
1 hour ago
Have you had a look at tex.stackexchange.com/questions/117906/… ?
– Henri Menke
1 hour ago
@HenriMenke Yes, I have and I can't seem to make things work. I've updated my answer to explain why your first suggestion is not tenable.
– A.Ellett
1 hour ago
@HenriMenke Yes, I have and I can't seem to make things work. I've updated my answer to explain why your first suggestion is not tenable.
– A.Ellett
1 hour ago
 |Â
show 3 more comments
1 Answer
1
active
oldest
votes
up vote
3
down vote
accepted
Remember that scantokens applies the category codes in force when it's used. Thus in
documentclassarticle
makeatletter
newcommandaeinput[1]%%
begingroup
scantokensgdefae@input::#1::%%
endgroup
ae@input
%%
makeatother
begindocument
aeinputHELLO
enddocument
the scantokens line is inserted in the doucment, where @ is not a letter. Sogdefae@input defines ae as a macro to be followed by @input with replacement text ::<#1>::. You need to change the category codes inside your group
documentclassarticle
makeatletter
newcommandaeinput[1]%%
begingroup
makeatletter
scantokensgdefae@input::#1::%%
endgroup
ae@input
%%
makeatother
begindocument
aeinputHELLO
enddocument
As you observe, whilst scantokens is expandable, getting that right is a bit tricky as it has some oddities about the 'end of file' for the virtual file it uses. Those can be got around (see the tl_rescan:nn code and similar in expl3), but it's non-trivial.
answered 29 mins ago
Joseph Wright♦
197k21543863
OMG. How did I not see that! Yikes. Subtle, but also kind of obvious---at least for the amount of monkeying around I do with this stuff. Thank you.
– A.Ellett
25 mins ago
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
3
down vote
accepted
Remember that scantokens applies the category codes in force when it's used. Thus in
documentclassarticle
makeatletter
newcommandaeinput[1]%%
begingroup
scantokensgdefae@input::#1::%%
endgroup
ae@input
%%
makeatother
begindocument
aeinputHELLO
enddocument
the scantokens line is inserted in the doucment, where @ is not a letter. Sogdefae@input defines ae as a macro to be followed by @input with replacement text ::<#1>::. You need to change the category codes inside your group
documentclassarticle
makeatletter
newcommandaeinput[1]%%
begingroup
makeatletter
scantokensgdefae@input::#1::%%
endgroup
ae@input
%%
makeatother
begindocument
aeinputHELLO
enddocument
As you observe, whilst scantokens is expandable, getting that right is a bit tricky as it has some oddities about the 'end of file' for the virtual file it uses. Those can be got around (see the tl_rescan:nn code and similar in expl3), but it's non-trivial.
answered 29 mins ago
Joseph Wright♦
197k21543863
OMG. How did I not see that! Yikes. Subtle, but also kind of obvious---at least for the amount of monkeying around I do with this stuff. Thank you.
– A.Ellett
25 mins ago
add a comment |Â
up vote
3
down vote
accepted
Remember that scantokens applies the category codes in force when it's used. Thus in
documentclassarticle
makeatletter
newcommandaeinput[1]%%
begingroup
scantokensgdefae@input::#1::%%
endgroup
ae@input
%%
makeatother
begindocument
aeinputHELLO
enddocument
the scantokens line is inserted in the doucment, where @ is not a letter. Sogdefae@input defines ae as a macro to be followed by @input with replacement text ::<#1>::. You need to change the category codes inside your group
documentclassarticle
makeatletter
newcommandaeinput[1]%%
begingroup
makeatletter
scantokensgdefae@input::#1::%%
endgroup
ae@input
%%
makeatother
begindocument
aeinputHELLO
enddocument
As you observe, whilst scantokens is expandable, getting that right is a bit tricky as it has some oddities about the 'end of file' for the virtual file it uses. Those can be got around (see the tl_rescan:nn code and similar in expl3), but it's non-trivial.
answered 29 mins ago
Joseph Wright♦
197k21543863
OMG. How did I not see that! Yikes. Subtle, but also kind of obvious---at least for the amount of monkeying around I do with this stuff. Thank you.
– A.Ellett
25 mins ago
add a comment |Â
up vote
3
down vote
accepted
up vote
3
down vote
accepted
Remember that scantokens applies the category codes in force when it's used. Thus in
documentclassarticle
makeatletter
newcommandaeinput[1]%%
begingroup
scantokensgdefae@input::#1::%%
endgroup
ae@input
%%
makeatother
begindocument
aeinputHELLO
enddocument
the scantokens line is inserted in the doucment, where @ is not a letter. Sogdefae@input defines ae as a macro to be followed by @input with replacement text ::<#1>::. You need to change the category codes inside your group
documentclassarticle
makeatletter
newcommandaeinput[1]%%
begingroup
makeatletter
scantokensgdefae@input::#1::%%
endgroup
ae@input
%%
makeatother
begindocument
aeinputHELLO
enddocument
As you observe, whilst scantokens is expandable, getting that right is a bit tricky as it has some oddities about the 'end of file' for the virtual file it uses. Those can be got around (see the tl_rescan:nn code and similar in expl3), but it's non-trivial.
answered 29 mins ago
Joseph Wright♦
197k21543863
Remember that scantokens applies the category codes in force when it's used. Thus in
documentclassarticle
makeatletter
newcommandaeinput[1]%%
begingroup
scantokensgdefae@input::#1::%%
endgroup
ae@input
%%
makeatother
begindocument
aeinputHELLO
enddocument
the scantokens line is inserted in the doucment, where @ is not a letter. Sogdefae@input defines ae as a macro to be followed by @input with replacement text ::<#1>::. You need to change the category codes inside your group
documentclassarticle
makeatletter
newcommandaeinput[1]%%
begingroup
makeatletter
scantokensgdefae@input::#1::%%
endgroup
ae@input
%%
makeatother
begindocument
aeinputHELLO
enddocument
As you observe, whilst scantokens is expandable, getting that right is a bit tricky as it has some oddities about the 'end of file' for the virtual file it uses. Those can be got around (see the tl_rescan:nn code and similar in expl3), but it's non-trivial.
answered 29 mins ago
Joseph Wright♦
197k21543863
answered 29 mins ago
Joseph Wright♦
197k21543863
answered 29 mins ago
Joseph Wright♦
197k21543863
answered 29 mins ago
Joseph Wright♦
197k21543863
197k21543863
OMG. How did I not see that! Yikes. Subtle, but also kind of obvious---at least for the amount of monkeying around I do with this stuff. Thank you.
– A.Ellett
25 mins ago
add a comment |Â
OMG. How did I not see that! Yikes. Subtle, but also kind of obvious---at least for the amount of monkeying around I do with this stuff. Thank you.
– A.Ellett
25 mins ago
OMG. How did I not see that! Yikes. Subtle, but also kind of obvious---at least for the amount of monkeying around I do with this stuff. Thank you.
– A.Ellett
25 mins ago
OMG. How did I not see that! Yikes. Subtle, but also kind of obvious---at least for the amount of monkeying around I do with this stuff. Thank you.
– A.Ellett
25 mins ago
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2ftex.stackexchange.com%2fquestions%2f453568%2fscantokens-not-playing-nice-with-def-or-gdef-etc%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
What is the purpose of
xdef? This will also fully expand#1. Why not justnewcommandaeinput[1]scantokens#1ignorespaces?– Henri Menke
1 hour ago
@HenriMenke According to the
etexdocumentationscantokensis expandable. But, let's say it's not. Why in the world wouldxdefae@inputscantokens....be the only instance for which my code compiled?– A.Ellett
1 hour ago
@HenriMenke You changed your comment before I noticed.
– A.Ellett
1 hour ago
Have you had a look at tex.stackexchange.com/questions/117906/… ?
– Henri Menke
1 hour ago
@HenriMenke Yes, I have and I can't seem to make things work. I've updated my answer to explain why your first suggestion is not tenable.
– A.Ellett
1 hour ago