Using uniq on unicode text

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
1
down vote

favorite












I want to remove duplicate lines from a file with words of Syriac script. The source file has 3 lines, 1st and 3rd are identical.



$ cat file.txt 
ܐܒܘܢ
ܢܗܘܐ
ܐܒܘܢ


When I use sort and uniq, the result presumes that all the 3 lines are identical, which is wrong:



$ cat file.txt | sort | uniq -c
 3 ܐܒܘܢ


Explicitly setting locale to Syriac doesn't help either.



$ LC_COLLATE=syr_SY.utf8 cat file.txt | sort | uniq -c 
 3 ܐܒܘܢ


Why would that happen?
I'm using Kubuntu 18 and bash, if that matters.






sort unicode uniq







share|improve this question



















  • 2




    Possible duplicate of Why is uniq ignoring Unicode and lines with a single letter?
    – Michael Homer
    5 hours ago






  • 1




    Note that both the sort and the uniq need to have the right collation to work here, so you'd want LC_COLLATE=syr_SY.utf8 sort file.txt | LC_COLLATE=syr_SY.utf8 uniq -c (or perhaps better yet in the regular environment).
    – Michael Homer
    5 hours ago














up vote
1
down vote

favorite












I want to remove duplicate lines from a file with words of Syriac script. The source file has 3 lines, 1st and 3rd are identical.



$ cat file.txt 
ܐܒܘܢ
ܢܗܘܐ
ܐܒܘܢ


When I use sort and uniq, the result presumes that all the 3 lines are identical, which is wrong:



$ cat file.txt | sort | uniq -c
 3 ܐܒܘܢ


Explicitly setting locale to Syriac doesn't help either.



$ LC_COLLATE=syr_SY.utf8 cat file.txt | sort | uniq -c 
 3 ܐܒܘܢ


Why would that happen?
I'm using Kubuntu 18 and bash, if that matters.






sort unicode uniq







share|improve this question



















  • 2




    Possible duplicate of Why is uniq ignoring Unicode and lines with a single letter?
    – Michael Homer
    5 hours ago






  • 1




    Note that both the sort and the uniq need to have the right collation to work here, so you'd want LC_COLLATE=syr_SY.utf8 sort file.txt | LC_COLLATE=syr_SY.utf8 uniq -c (or perhaps better yet in the regular environment).
    – Michael Homer
    5 hours ago












up vote
1
down vote

favorite









up vote
1
down vote

favorite











I want to remove duplicate lines from a file with words of Syriac script. The source file has 3 lines, 1st and 3rd are identical.



$ cat file.txt 
ܐܒܘܢ
ܢܗܘܐ
ܐܒܘܢ


When I use sort and uniq, the result presumes that all the 3 lines are identical, which is wrong:



$ cat file.txt | sort | uniq -c
 3 ܐܒܘܢ


Explicitly setting locale to Syriac doesn't help either.



$ LC_COLLATE=syr_SY.utf8 cat file.txt | sort | uniq -c 
 3 ܐܒܘܢ


Why would that happen?
I'm using Kubuntu 18 and bash, if that matters.






sort unicode uniq







share|improve this question















I want to remove duplicate lines from a file with words of Syriac script. The source file has 3 lines, 1st and 3rd are identical.



$ cat file.txt 
ܐܒܘܢ
ܢܗܘܐ
ܐܒܘܢ


When I use sort and uniq, the result presumes that all the 3 lines are identical, which is wrong:



$ cat file.txt | sort | uniq -c
 3 ܐܒܘܢ


Explicitly setting locale to Syriac doesn't help either.



$ LC_COLLATE=syr_SY.utf8 cat file.txt | sort | uniq -c 
 3 ܐܒܘܢ


Why would that happen?
I'm using Kubuntu 18 and bash, if that matters.






sort unicode uniq




sort unicode uniq






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 5 hours ago









神秘德里克

1559




1559










asked 6 hours ago









evb

1085




1085







  • 2




    Possible duplicate of Why is uniq ignoring Unicode and lines with a single letter?
    – Michael Homer
    5 hours ago






  • 1




    Note that both the sort and the uniq need to have the right collation to work here, so you'd want LC_COLLATE=syr_SY.utf8 sort file.txt | LC_COLLATE=syr_SY.utf8 uniq -c (or perhaps better yet in the regular environment).
    – Michael Homer
    5 hours ago












  • 2




    Possible duplicate of Why is uniq ignoring Unicode and lines with a single letter?
    – Michael Homer
    5 hours ago






  • 1




    Note that both the sort and the uniq need to have the right collation to work here, so you'd want LC_COLLATE=syr_SY.utf8 sort file.txt | LC_COLLATE=syr_SY.utf8 uniq -c (or perhaps better yet in the regular environment).
    – Michael Homer
    5 hours ago







2




2




Possible duplicate of Why is uniq ignoring Unicode and lines with a single letter?
– Michael Homer
5 hours ago




Possible duplicate of Why is uniq ignoring Unicode and lines with a single letter?
– Michael Homer
5 hours ago




1




1




Note that both the sort and the uniq need to have the right collation to work here, so you'd want LC_COLLATE=syr_SY.utf8 sort file.txt | LC_COLLATE=syr_SY.utf8 uniq -c (or perhaps better yet in the regular environment).
– Michael Homer
5 hours ago




Note that both the sort and the uniq need to have the right collation to work here, so you'd want LC_COLLATE=syr_SY.utf8 sort file.txt | LC_COLLATE=syr_SY.utf8 uniq -c (or perhaps better yet in the regular environment).
– Michael Homer
5 hours ago










2 Answers
2






active

oldest

votes

















up vote
2
down vote



accepted










First set CTYPE:



$ export LC_CTYPE=syr_SY.utf8
$ cat file.txt |sort |uniq -c
 2 ܐܒܘܢ
 1 ܢܗܘܐ





share|improve this answer




















  • thanks! The only problem is that I get a warning: bash: warning: setlocale: LC_CTYPE: cannot change locale (syr_SY.utf8)
    – evb
    5 hours ago






  • 1




    There is no syr_SY locale defined yet. As it doesn't exist, the locale in effect drops to default: C. That is why the commands did work. @evb
    – Isaac
    14 mins ago






  • 1




    Also, there is no need to use cat.
    – Isaac
    14 mins ago

















up vote
2
down vote













A (simplistic) portable solution:



$ ( LC_ALL=C sort syriac.txt | LC_ALL=C uniq -c )
 2 ܐܒܘܢ
 1 ܢܗܘܐ





share|improve this answer




















    Your Answer







    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "106"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    convertImagesToLinks: false,
    noModals: false,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );













     

    draft saved


    draft discarded


















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f469347%2fusing-uniq-on-unicode-text%23new-answer', 'question_page');

    );

    Post as a guest

















    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes








    up vote
    2
    down vote



    accepted










    First set CTYPE:



    $ export LC_CTYPE=syr_SY.utf8
    $ cat file.txt |sort |uniq -c
     2 ܐܒܘܢ
     1 ܢܗܘܐ





    share|improve this answer




















    • thanks! The only problem is that I get a warning: bash: warning: setlocale: LC_CTYPE: cannot change locale (syr_SY.utf8)
      – evb
      5 hours ago






    • 1




      There is no syr_SY locale defined yet. As it doesn't exist, the locale in effect drops to default: C. That is why the commands did work. @evb
      – Isaac
      14 mins ago






    • 1




      Also, there is no need to use cat.
      – Isaac
      14 mins ago














    up vote
    2
    down vote



    accepted










    First set CTYPE:



    $ export LC_CTYPE=syr_SY.utf8
    $ cat file.txt |sort |uniq -c
     2 ܐܒܘܢ
     1 ܢܗܘܐ





    share|improve this answer




















    • thanks! The only problem is that I get a warning: bash: warning: setlocale: LC_CTYPE: cannot change locale (syr_SY.utf8)
      – evb
      5 hours ago






    • 1




      There is no syr_SY locale defined yet. As it doesn't exist, the locale in effect drops to default: C. That is why the commands did work. @evb
      – Isaac
      14 mins ago






    • 1




      Also, there is no need to use cat.
      – Isaac
      14 mins ago












    up vote
    2
    down vote



    accepted







    up vote
    2
    down vote



    accepted






    First set CTYPE:



    $ export LC_CTYPE=syr_SY.utf8
    $ cat file.txt |sort |uniq -c
     2 ܐܒܘܢ
     1 ܢܗܘܐ





    share|improve this answer












    First set CTYPE:



    $ export LC_CTYPE=syr_SY.utf8
    $ cat file.txt |sort |uniq -c
     2 ܐܒܘܢ
     1 ܢܗܘܐ






    share|improve this answer












    share|improve this answer



    share|improve this answer










    answered 5 hours ago









    Ipor Sircer

    9,2331920




    9,2331920











    • thanks! The only problem is that I get a warning: bash: warning: setlocale: LC_CTYPE: cannot change locale (syr_SY.utf8)
      – evb
      5 hours ago






    • 1




      There is no syr_SY locale defined yet. As it doesn't exist, the locale in effect drops to default: C. That is why the commands did work. @evb
      – Isaac
      14 mins ago






    • 1




      Also, there is no need to use cat.
      – Isaac
      14 mins ago
















    • thanks! The only problem is that I get a warning: bash: warning: setlocale: LC_CTYPE: cannot change locale (syr_SY.utf8)
      – evb
      5 hours ago






    • 1




      There is no syr_SY locale defined yet. As it doesn't exist, the locale in effect drops to default: C. That is why the commands did work. @evb
      – Isaac
      14 mins ago






    • 1




      Also, there is no need to use cat.
      – Isaac
      14 mins ago















    thanks! The only problem is that I get a warning: bash: warning: setlocale: LC_CTYPE: cannot change locale (syr_SY.utf8)
    – evb
    5 hours ago




    thanks! The only problem is that I get a warning: bash: warning: setlocale: LC_CTYPE: cannot change locale (syr_SY.utf8)
    – evb
    5 hours ago




    1




    1




    There is no syr_SY locale defined yet. As it doesn't exist, the locale in effect drops to default: C. That is why the commands did work. @evb
    – Isaac
    14 mins ago




    There is no syr_SY locale defined yet. As it doesn't exist, the locale in effect drops to default: C. That is why the commands did work. @evb
    – Isaac
    14 mins ago




    1




    1




    Also, there is no need to use cat.
    – Isaac
    14 mins ago




    Also, there is no need to use cat.
    – Isaac
    14 mins ago












    up vote
    2
    down vote













    A (simplistic) portable solution:



    $ ( LC_ALL=C sort syriac.txt | LC_ALL=C uniq -c )
     2 ܐܒܘܢ
     1 ܢܗܘܐ





    share|improve this answer
























      up vote
      2
      down vote













      A (simplistic) portable solution:



      $ ( LC_ALL=C sort syriac.txt | LC_ALL=C uniq -c )
       2 ܐܒܘܢ
       1 ܢܗܘܐ





      share|improve this answer






















        up vote
        2
        down vote










        up vote
        2
        down vote









        A (simplistic) portable solution:



        $ ( LC_ALL=C sort syriac.txt | LC_ALL=C uniq -c )
         2 ܐܒܘܢ
         1 ܢܗܘܐ





        share|improve this answer












        A (simplistic) portable solution:



        $ ( LC_ALL=C sort syriac.txt | LC_ALL=C uniq -c )
         2 ܐܒܘܢ
         1 ܢܗܘܐ






        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered 14 mins ago









        Isaac

        7,0121834




        7,0121834



























             

            draft saved


            draft discarded















































             


            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f469347%2fusing-uniq-on-unicode-text%23new-answer', 'question_page');

            );

            Post as a guest
































































            Comments

            Post a Comment

            Popular posts from this blog

            What does second last employer means? [closed]

            Image
            Clash Royale CLAN TAG #URR8PPP .everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty margin-bottom:0; up vote -3 down vote favorite Say, for example I have worked in the following companies, Company A => 2010 - 2011 Company B => 2011 - 2012 Company C => 2012 - Till Date Company D => Have offer Now assume Company D is asking to submit documents from my second last employer. So from the list of companies given, which company would be qualified as second last employer? software-industry employer share | improve this question asked Aug 10 '14 at 5:16 Rajaprabhu Aravindasamy 559 1 6 13 closed as off-topic by Jim G., jmort253 ♦ Aug 11 '14 at 1:58 This question appears to be off-topic. The users who voted to close gave this specific reason: "Questions seeking advice on company-specific regulations, agreements, or policies should be directed to your manager or HR department. Questions that address...
            Read more

            List of Gilmore Girls characters

            Image
            The season three cast—(Back row): Lane, Michel, Paris, Emily, Richard, Sookie, Miss Patty, Kirk; Front row: Jess, Luke, Lorelai, Rory, Dean This is a list of characters for the comedy-drama television series Gilmore Girls . Contents 1 Main characters 1.1 Lorelai Gilmore 1.2 Rory Gilmore 1.3 Sookie St. James 1.4 Lane Kim 1.5 Michel Gerard 1.6 Luke Danes 1.7 Emily Gilmore 1.8 Richard Gilmore 1.9 Paris Geller 1.10 Dean Forester 1.11 Jess Mariano 1.12 Kirk Gleason 1.13 Jason Stiles 1.14 Logan Huntzberger 2 Major recurring characters 2.1 Christopher Hayden 2.2 Jackson Belleville 2.3 Mrs. Kim 2.4 Tristin DuGray 2.5 Max Medina 2.6 Taylor Doose 2.7 Dave Rygalski 2.8 Marty 3 Recurring characters 3.1 Stars Hollow 3.2 Chilton 3.3 Yale 3.4 Other 4 Notable guest stars 5 Unseen characters 6 Notes 7 References Main characters Actor Character Appearances Season 1 Season 2 Season 3 Season 4 Season 5 Season 6 Season 7 A Year in the Life Lauren Graham Lorelai Gilmore...
            Read more

            Confectionery

            Image
            Prepared foods made with a lot of sugar or other cabohydrates "Sweetmeat" redirects here. For the racehorse, see Sweetmeat (horse). Not to be confused with Sweetbread. This Kransekake is a traditional Scandinavian baker's confection. Confectionery is the art of making confections , which are food items that are rich in sugar and carbohydrates. Exact definitions are difficult. [1] In general, though, confectionery is divided into two broad and somewhat overlapping categories, bakers' confections and sugar confections. [2] Bakers' confectionery, also called flour confections , includes principally sweet pastries, cakes, and similar baked goods. Sugar confectionery includes candies ( sweets in British English), candied nuts, chocolates, chewing gum, bubble gum, pastillage, and other confections that are made primarily of sugar. In some cases, chocolate confections (confections made of chocolate) are treated as a separate category, as are sugar-free versions of ...
            Read more