Using uniq on unicode text

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
1
down vote

favorite












I want to remove duplicate lines from a file with words of Syriac script. The source file has 3 lines, 1st and 3rd are identical.



$ cat file.txt 
ܐܒܘܢ
ܢܗܘܐ
ܐܒܘܢ


When I use sort and uniq, the result presumes that all the 3 lines are identical, which is wrong:



$ cat file.txt | sort | uniq -c
 3 ܐܒܘܢ


Explicitly setting locale to Syriac doesn't help either.



$ LC_COLLATE=syr_SY.utf8 cat file.txt | sort | uniq -c 
 3 ܐܒܘܢ


Why would that happen?
I'm using Kubuntu 18 and bash, if that matters.






sort unicode uniq







share|improve this question



















  • 2




    Possible duplicate of Why is uniq ignoring Unicode and lines with a single letter?
    – Michael Homer
    5 hours ago






  • 1




    Note that both the sort and the uniq need to have the right collation to work here, so you'd want LC_COLLATE=syr_SY.utf8 sort file.txt | LC_COLLATE=syr_SY.utf8 uniq -c (or perhaps better yet in the regular environment).
    – Michael Homer
    5 hours ago














up vote
1
down vote

favorite












I want to remove duplicate lines from a file with words of Syriac script. The source file has 3 lines, 1st and 3rd are identical.



$ cat file.txt 
ܐܒܘܢ
ܢܗܘܐ
ܐܒܘܢ


When I use sort and uniq, the result presumes that all the 3 lines are identical, which is wrong:



$ cat file.txt | sort | uniq -c
 3 ܐܒܘܢ


Explicitly setting locale to Syriac doesn't help either.



$ LC_COLLATE=syr_SY.utf8 cat file.txt | sort | uniq -c 
 3 ܐܒܘܢ


Why would that happen?
I'm using Kubuntu 18 and bash, if that matters.






sort unicode uniq







share|improve this question



















  • 2




    Possible duplicate of Why is uniq ignoring Unicode and lines with a single letter?
    – Michael Homer
    5 hours ago






  • 1




    Note that both the sort and the uniq need to have the right collation to work here, so you'd want LC_COLLATE=syr_SY.utf8 sort file.txt | LC_COLLATE=syr_SY.utf8 uniq -c (or perhaps better yet in the regular environment).
    – Michael Homer
    5 hours ago












up vote
1
down vote

favorite









up vote
1
down vote

favorite











I want to remove duplicate lines from a file with words of Syriac script. The source file has 3 lines, 1st and 3rd are identical.



$ cat file.txt 
ܐܒܘܢ
ܢܗܘܐ
ܐܒܘܢ


When I use sort and uniq, the result presumes that all the 3 lines are identical, which is wrong:



$ cat file.txt | sort | uniq -c
 3 ܐܒܘܢ


Explicitly setting locale to Syriac doesn't help either.



$ LC_COLLATE=syr_SY.utf8 cat file.txt | sort | uniq -c 
 3 ܐܒܘܢ


Why would that happen?
I'm using Kubuntu 18 and bash, if that matters.






sort unicode uniq







share|improve this question















I want to remove duplicate lines from a file with words of Syriac script. The source file has 3 lines, 1st and 3rd are identical.



$ cat file.txt 
ܐܒܘܢ
ܢܗܘܐ
ܐܒܘܢ


When I use sort and uniq, the result presumes that all the 3 lines are identical, which is wrong:



$ cat file.txt | sort | uniq -c
 3 ܐܒܘܢ


Explicitly setting locale to Syriac doesn't help either.



$ LC_COLLATE=syr_SY.utf8 cat file.txt | sort | uniq -c 
 3 ܐܒܘܢ


Why would that happen?
I'm using Kubuntu 18 and bash, if that matters.






sort unicode uniq




sort unicode uniq






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 5 hours ago









神秘德里克

1559




1559










asked 6 hours ago









evb

1085




1085







  • 2




    Possible duplicate of Why is uniq ignoring Unicode and lines with a single letter?
    – Michael Homer
    5 hours ago






  • 1




    Note that both the sort and the uniq need to have the right collation to work here, so you'd want LC_COLLATE=syr_SY.utf8 sort file.txt | LC_COLLATE=syr_SY.utf8 uniq -c (or perhaps better yet in the regular environment).
    – Michael Homer
    5 hours ago












  • 2




    Possible duplicate of Why is uniq ignoring Unicode and lines with a single letter?
    – Michael Homer
    5 hours ago






  • 1




    Note that both the sort and the uniq need to have the right collation to work here, so you'd want LC_COLLATE=syr_SY.utf8 sort file.txt | LC_COLLATE=syr_SY.utf8 uniq -c (or perhaps better yet in the regular environment).
    – Michael Homer
    5 hours ago







2




2




Possible duplicate of Why is uniq ignoring Unicode and lines with a single letter?
– Michael Homer
5 hours ago




Possible duplicate of Why is uniq ignoring Unicode and lines with a single letter?
– Michael Homer
5 hours ago




1




1




Note that both the sort and the uniq need to have the right collation to work here, so you'd want LC_COLLATE=syr_SY.utf8 sort file.txt | LC_COLLATE=syr_SY.utf8 uniq -c (or perhaps better yet in the regular environment).
– Michael Homer
5 hours ago




Note that both the sort and the uniq need to have the right collation to work here, so you'd want LC_COLLATE=syr_SY.utf8 sort file.txt | LC_COLLATE=syr_SY.utf8 uniq -c (or perhaps better yet in the regular environment).
– Michael Homer
5 hours ago










2 Answers
2






active

oldest

votes

















up vote
2
down vote



accepted










First set CTYPE:



$ export LC_CTYPE=syr_SY.utf8
$ cat file.txt |sort |uniq -c
 2 ܐܒܘܢ
 1 ܢܗܘܐ





share|improve this answer




















  • thanks! The only problem is that I get a warning: bash: warning: setlocale: LC_CTYPE: cannot change locale (syr_SY.utf8)
    – evb
    5 hours ago






  • 1




    There is no syr_SY locale defined yet. As it doesn't exist, the locale in effect drops to default: C. That is why the commands did work. @evb
    – Isaac
    14 mins ago






  • 1




    Also, there is no need to use cat.
    – Isaac
    14 mins ago

















up vote
2
down vote













A (simplistic) portable solution:



$ ( LC_ALL=C sort syriac.txt | LC_ALL=C uniq -c )
 2 ܐܒܘܢ
 1 ܢܗܘܐ





share|improve this answer




















    Your Answer







    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "106"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    convertImagesToLinks: false,
    noModals: false,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );













     

    draft saved


    draft discarded


















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f469347%2fusing-uniq-on-unicode-text%23new-answer', 'question_page');

    );

    Post as a guest

















    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes








    up vote
    2
    down vote



    accepted










    First set CTYPE:



    $ export LC_CTYPE=syr_SY.utf8
    $ cat file.txt |sort |uniq -c
     2 ܐܒܘܢ
     1 ܢܗܘܐ





    share|improve this answer




















    • thanks! The only problem is that I get a warning: bash: warning: setlocale: LC_CTYPE: cannot change locale (syr_SY.utf8)
      – evb
      5 hours ago






    • 1




      There is no syr_SY locale defined yet. As it doesn't exist, the locale in effect drops to default: C. That is why the commands did work. @evb
      – Isaac
      14 mins ago






    • 1




      Also, there is no need to use cat.
      – Isaac
      14 mins ago














    up vote
    2
    down vote



    accepted










    First set CTYPE:



    $ export LC_CTYPE=syr_SY.utf8
    $ cat file.txt |sort |uniq -c
     2 ܐܒܘܢ
     1 ܢܗܘܐ





    share|improve this answer




















    • thanks! The only problem is that I get a warning: bash: warning: setlocale: LC_CTYPE: cannot change locale (syr_SY.utf8)
      – evb
      5 hours ago






    • 1




      There is no syr_SY locale defined yet. As it doesn't exist, the locale in effect drops to default: C. That is why the commands did work. @evb
      – Isaac
      14 mins ago






    • 1




      Also, there is no need to use cat.
      – Isaac
      14 mins ago












    up vote
    2
    down vote



    accepted







    up vote
    2
    down vote



    accepted






    First set CTYPE:



    $ export LC_CTYPE=syr_SY.utf8
    $ cat file.txt |sort |uniq -c
     2 ܐܒܘܢ
     1 ܢܗܘܐ





    share|improve this answer












    First set CTYPE:



    $ export LC_CTYPE=syr_SY.utf8
    $ cat file.txt |sort |uniq -c
     2 ܐܒܘܢ
     1 ܢܗܘܐ






    share|improve this answer












    share|improve this answer



    share|improve this answer










    answered 5 hours ago









    Ipor Sircer

    9,2331920




    9,2331920











    • thanks! The only problem is that I get a warning: bash: warning: setlocale: LC_CTYPE: cannot change locale (syr_SY.utf8)
      – evb
      5 hours ago






    • 1




      There is no syr_SY locale defined yet. As it doesn't exist, the locale in effect drops to default: C. That is why the commands did work. @evb
      – Isaac
      14 mins ago






    • 1




      Also, there is no need to use cat.
      – Isaac
      14 mins ago
















    • thanks! The only problem is that I get a warning: bash: warning: setlocale: LC_CTYPE: cannot change locale (syr_SY.utf8)
      – evb
      5 hours ago






    • 1




      There is no syr_SY locale defined yet. As it doesn't exist, the locale in effect drops to default: C. That is why the commands did work. @evb
      – Isaac
      14 mins ago






    • 1




      Also, there is no need to use cat.
      – Isaac
      14 mins ago















    thanks! The only problem is that I get a warning: bash: warning: setlocale: LC_CTYPE: cannot change locale (syr_SY.utf8)
    – evb
    5 hours ago




    thanks! The only problem is that I get a warning: bash: warning: setlocale: LC_CTYPE: cannot change locale (syr_SY.utf8)
    – evb
    5 hours ago




    1




    1




    There is no syr_SY locale defined yet. As it doesn't exist, the locale in effect drops to default: C. That is why the commands did work. @evb
    – Isaac
    14 mins ago




    There is no syr_SY locale defined yet. As it doesn't exist, the locale in effect drops to default: C. That is why the commands did work. @evb
    – Isaac
    14 mins ago




    1




    1




    Also, there is no need to use cat.
    – Isaac
    14 mins ago




    Also, there is no need to use cat.
    – Isaac
    14 mins ago












    up vote
    2
    down vote













    A (simplistic) portable solution:



    $ ( LC_ALL=C sort syriac.txt | LC_ALL=C uniq -c )
     2 ܐܒܘܢ
     1 ܢܗܘܐ





    share|improve this answer
























      up vote
      2
      down vote













      A (simplistic) portable solution:



      $ ( LC_ALL=C sort syriac.txt | LC_ALL=C uniq -c )
       2 ܐܒܘܢ
       1 ܢܗܘܐ





      share|improve this answer






















        up vote
        2
        down vote










        up vote
        2
        down vote









        A (simplistic) portable solution:



        $ ( LC_ALL=C sort syriac.txt | LC_ALL=C uniq -c )
         2 ܐܒܘܢ
         1 ܢܗܘܐ





        share|improve this answer












        A (simplistic) portable solution:



        $ ( LC_ALL=C sort syriac.txt | LC_ALL=C uniq -c )
         2 ܐܒܘܢ
         1 ܢܗܘܐ






        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered 14 mins ago









        Isaac

        7,0121834




        7,0121834



























             

            draft saved


            draft discarded















































             


            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f469347%2fusing-uniq-on-unicode-text%23new-answer', 'question_page');

            );

            Post as a guest
































































            Comments

            Post a Comment

            Popular posts from this blog

            White Anglo-Saxon Protestant

            Image
            White Anglo-Saxon Protestants ( WASPs ) are a social group of wealthy and well-connected white Americans, of Protestant and predominantly British ancestry, who trace their ancestry to the American colonial period. Until at least the 1960s, this group has dominated American society and culture and dominated in the leadership of the Whig and Republican parties. They usually are very well placed in major financial, business, legal and academic institutions and had close to a monopoly of elite society due to intermarriage and nepotism. [1] During the latter half of the twentieth century, outsider ethnic and racial groups grew in influence and WASP dominance weakened. Americans are increasingly criticizing the WASP hegemony and disparaging WASPs as the epitome of "the Establishment". The Random House Unabridged Dictionary (1998) says the term is "Sometimes Disparaging and Offensive". [2] [3] The term WASP is often used as a pejorative to classify their historical dom...
            Read more

            BuddyTV

            BuddyTV Owner David Niu, Andy Liu Created by Various contributors Website www.buddytv.com Launched July 31, 2007 Current status Offline BuddyTV was an entertainment-based website based in Seattle, Washington, which generated content about television programs and sporting events. The website published information about celebrity and related entertainment news through a series of articles, entertainment profiles, actor biographies and user forums. [1] On 31 December 2014, Smart TV manufacturer VIZIO acquired BuddyTV's parent Advanced Media Research Group, Inc., in order to expand content and service offerings. [2] The site was shut down on 22 May 2018. [3] Contents 1 History 2 Business model 3 Impact 4 See also 5 References 6 External links History BuddyTV was co-founded in 2005 by Andy Liu and David Niu. The pair graduated from the University of Pennsylvania with a Master of Business Administration. They sold their previous company, NetConversions, to aQuantive in 2004...
            Read more

            Conflict (narrative)

            Image
            Conflict in narrative comes in many forms. "Man versus man", such as is depicted here in the battle between King Arthur and Mordred, is particularly common in traditional literature, fairy tales and myths. [1] In works of narrative, conflict is the challenge main characters need to solve to achieve their goals. Traditionally, conflict is a major literary element that creates challenges in a story by adding uncertainty to if the goal would be achieved. A narrative is not limited to a single conflict. While conflicts may not always resolve in narrative, the resolution of a conflict creates closure or fulfillment, which may or may not occur at a story's end. Contents 1 Basic nature 2 Classification 2.1 Man against man 2.2 Man against nature 2.3 Man against self 2.4 Man against society 3 History 4 See also 5 References 6 External links Basic nature Conflict in literature refers to the different drives of the characters or forces involved. Conflict may be internal o...
            Read more