Serialization and serialization times in 40G/10G and 100G/25G Ethernet

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
3
down vote

favorite
1












I've recently been involved in discussions about lowest-latency requirements for a Leaf/Spine (or CLOS) network to host an OpenStack platform.



System architects are striving for the lowest possible RTT for their transactions (block storage and future RDMA scenarios), and the claim was that 100G/25G offered greatly reduced serialization delays compared to 40G/10G. All persons involved are aware that there's a lot more factors in the end to end game (any of which can hurt or help RTT) than just the NICs and switch ports serialization delays. Still, the topic about serialization delays keeps popping up, as they are one thing that is difficult to optimize without jumping a possibly very costly technology gap.



A bit over-simplified (leaving out the encoding schemes), serialization time can be calculated as number-of-bits/bit rate, which lets us start at ~1.2μs for 10G (also see wiki.geant.org).



For a 1518 byte frame with 12'144bits,
at 10G (assuming 10*10^9 bits/s), this will give us ~1.2μs
at 25G (assuming 25*10^9 bits/s), this would be reduced to ~0.48μs 
at 40G (assuming 40*10^9 bits/s), one might expect to see ~0.3μs
at 100G (assuming 100*10^9 bits/s), one might expect to see ~0.12μs


Now for the interesting bit. At the physical layer, 40G is commonly done as 4 lanes of 10G and 100G is done as 4 lanes of 25G. Depending on QSFP+ or QSFP28 variant, this is sometimes done with 4 pairs of fibre strands, sometimes it is split by lambdas on a single fibre pair, where the QSFP module does some xWDM on its own. I do know that there's specs for 1x 40G or or 2x 50G or even 1x 100G lanes, but let's leave those aside for the moment.



To estimate serialization delays in the context of multi-lane 40G or 100G, one needs to know how 100G and 40G NICs and switch ports actually "distribute the bits to the (set of) wire(s)", so to speak. What is being done here?



Is it a bit like Etherchannel/LAG? The NIC/switchports send frames of one "flow" (read: same hashing result of whatever hashing algorithm is used across which scope of the frame) across one given channel? In that case, we'd expect serialization delays like 10G and 25G, respectively. But essentially, that would make a 40G link just a LAG of 4x10G, reducing single flow throughput to 1x10G.



Is it something like bit-wise round-robin? Each bit is round-robin distributed across the 4 (sub)channels? That might actually result in lower serialization delays because of parallelization, but raises some questions about in-order-delivery.



Is it something like frame-wise round-robin? Entire ethernet frames (or other suitably sized chunks of bits) are sent over the 4 channels, distributed in round robin fashion?



Is it someting else entirely, such as...



Thanks for your comments and pointers.






latency serialization rtt 40g 100g







share|improve this question



























    up vote
    3
    down vote

    favorite
    1












    I've recently been involved in discussions about lowest-latency requirements for a Leaf/Spine (or CLOS) network to host an OpenStack platform.



    System architects are striving for the lowest possible RTT for their transactions (block storage and future RDMA scenarios), and the claim was that 100G/25G offered greatly reduced serialization delays compared to 40G/10G. All persons involved are aware that there's a lot more factors in the end to end game (any of which can hurt or help RTT) than just the NICs and switch ports serialization delays. Still, the topic about serialization delays keeps popping up, as they are one thing that is difficult to optimize without jumping a possibly very costly technology gap.



    A bit over-simplified (leaving out the encoding schemes), serialization time can be calculated as number-of-bits/bit rate, which lets us start at ~1.2μs for 10G (also see wiki.geant.org).



    For a 1518 byte frame with 12'144bits,
    at 10G (assuming 10*10^9 bits/s), this will give us ~1.2μs
    at 25G (assuming 25*10^9 bits/s), this would be reduced to ~0.48μs 
    at 40G (assuming 40*10^9 bits/s), one might expect to see ~0.3μs
    at 100G (assuming 100*10^9 bits/s), one might expect to see ~0.12μs


    Now for the interesting bit. At the physical layer, 40G is commonly done as 4 lanes of 10G and 100G is done as 4 lanes of 25G. Depending on QSFP+ or QSFP28 variant, this is sometimes done with 4 pairs of fibre strands, sometimes it is split by lambdas on a single fibre pair, where the QSFP module does some xWDM on its own. I do know that there's specs for 1x 40G or or 2x 50G or even 1x 100G lanes, but let's leave those aside for the moment.



    To estimate serialization delays in the context of multi-lane 40G or 100G, one needs to know how 100G and 40G NICs and switch ports actually "distribute the bits to the (set of) wire(s)", so to speak. What is being done here?



    Is it a bit like Etherchannel/LAG? The NIC/switchports send frames of one "flow" (read: same hashing result of whatever hashing algorithm is used across which scope of the frame) across one given channel? In that case, we'd expect serialization delays like 10G and 25G, respectively. But essentially, that would make a 40G link just a LAG of 4x10G, reducing single flow throughput to 1x10G.



    Is it something like bit-wise round-robin? Each bit is round-robin distributed across the 4 (sub)channels? That might actually result in lower serialization delays because of parallelization, but raises some questions about in-order-delivery.



    Is it something like frame-wise round-robin? Entire ethernet frames (or other suitably sized chunks of bits) are sent over the 4 channels, distributed in round robin fashion?



    Is it someting else entirely, such as...



    Thanks for your comments and pointers.






    latency serialization rtt 40g 100g







    share|improve this question

























      up vote
      3
      down vote

      favorite
      1









      up vote
      3
      down vote

      favorite
      1






      1





      I've recently been involved in discussions about lowest-latency requirements for a Leaf/Spine (or CLOS) network to host an OpenStack platform.



      System architects are striving for the lowest possible RTT for their transactions (block storage and future RDMA scenarios), and the claim was that 100G/25G offered greatly reduced serialization delays compared to 40G/10G. All persons involved are aware that there's a lot more factors in the end to end game (any of which can hurt or help RTT) than just the NICs and switch ports serialization delays. Still, the topic about serialization delays keeps popping up, as they are one thing that is difficult to optimize without jumping a possibly very costly technology gap.



      A bit over-simplified (leaving out the encoding schemes), serialization time can be calculated as number-of-bits/bit rate, which lets us start at ~1.2μs for 10G (also see wiki.geant.org).



      For a 1518 byte frame with 12'144bits,
      at 10G (assuming 10*10^9 bits/s), this will give us ~1.2μs
      at 25G (assuming 25*10^9 bits/s), this would be reduced to ~0.48μs 
      at 40G (assuming 40*10^9 bits/s), one might expect to see ~0.3μs
      at 100G (assuming 100*10^9 bits/s), one might expect to see ~0.12μs


      Now for the interesting bit. At the physical layer, 40G is commonly done as 4 lanes of 10G and 100G is done as 4 lanes of 25G. Depending on QSFP+ or QSFP28 variant, this is sometimes done with 4 pairs of fibre strands, sometimes it is split by lambdas on a single fibre pair, where the QSFP module does some xWDM on its own. I do know that there's specs for 1x 40G or or 2x 50G or even 1x 100G lanes, but let's leave those aside for the moment.



      To estimate serialization delays in the context of multi-lane 40G or 100G, one needs to know how 100G and 40G NICs and switch ports actually "distribute the bits to the (set of) wire(s)", so to speak. What is being done here?



      Is it a bit like Etherchannel/LAG? The NIC/switchports send frames of one "flow" (read: same hashing result of whatever hashing algorithm is used across which scope of the frame) across one given channel? In that case, we'd expect serialization delays like 10G and 25G, respectively. But essentially, that would make a 40G link just a LAG of 4x10G, reducing single flow throughput to 1x10G.



      Is it something like bit-wise round-robin? Each bit is round-robin distributed across the 4 (sub)channels? That might actually result in lower serialization delays because of parallelization, but raises some questions about in-order-delivery.



      Is it something like frame-wise round-robin? Entire ethernet frames (or other suitably sized chunks of bits) are sent over the 4 channels, distributed in round robin fashion?



      Is it someting else entirely, such as...



      Thanks for your comments and pointers.






      latency serialization rtt 40g 100g







      share|improve this question















      I've recently been involved in discussions about lowest-latency requirements for a Leaf/Spine (or CLOS) network to host an OpenStack platform.



      System architects are striving for the lowest possible RTT for their transactions (block storage and future RDMA scenarios), and the claim was that 100G/25G offered greatly reduced serialization delays compared to 40G/10G. All persons involved are aware that there's a lot more factors in the end to end game (any of which can hurt or help RTT) than just the NICs and switch ports serialization delays. Still, the topic about serialization delays keeps popping up, as they are one thing that is difficult to optimize without jumping a possibly very costly technology gap.



      A bit over-simplified (leaving out the encoding schemes), serialization time can be calculated as number-of-bits/bit rate, which lets us start at ~1.2μs for 10G (also see wiki.geant.org).



      For a 1518 byte frame with 12'144bits,
      at 10G (assuming 10*10^9 bits/s), this will give us ~1.2μs
      at 25G (assuming 25*10^9 bits/s), this would be reduced to ~0.48μs 
      at 40G (assuming 40*10^9 bits/s), one might expect to see ~0.3μs
      at 100G (assuming 100*10^9 bits/s), one might expect to see ~0.12μs


      Now for the interesting bit. At the physical layer, 40G is commonly done as 4 lanes of 10G and 100G is done as 4 lanes of 25G. Depending on QSFP+ or QSFP28 variant, this is sometimes done with 4 pairs of fibre strands, sometimes it is split by lambdas on a single fibre pair, where the QSFP module does some xWDM on its own. I do know that there's specs for 1x 40G or or 2x 50G or even 1x 100G lanes, but let's leave those aside for the moment.



      To estimate serialization delays in the context of multi-lane 40G or 100G, one needs to know how 100G and 40G NICs and switch ports actually "distribute the bits to the (set of) wire(s)", so to speak. What is being done here?



      Is it a bit like Etherchannel/LAG? The NIC/switchports send frames of one "flow" (read: same hashing result of whatever hashing algorithm is used across which scope of the frame) across one given channel? In that case, we'd expect serialization delays like 10G and 25G, respectively. But essentially, that would make a 40G link just a LAG of 4x10G, reducing single flow throughput to 1x10G.



      Is it something like bit-wise round-robin? Each bit is round-robin distributed across the 4 (sub)channels? That might actually result in lower serialization delays because of parallelization, but raises some questions about in-order-delivery.



      Is it something like frame-wise round-robin? Entire ethernet frames (or other suitably sized chunks of bits) are sent over the 4 channels, distributed in round robin fashion?



      Is it someting else entirely, such as...



      Thanks for your comments and pointers.






      latency serialization rtt 40g 100g




      latency serialization rtt 40g 100g






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited 1 hour ago

























      asked 2 hours ago









      Marc 'netztier' Luethi

      1,982313




      1,982313




















          1 Answer
          1






          active

          oldest

          votes

















          up vote
          3
          down vote













          This is overthinking.



          The number of lanes used doesn't really matter. Whether you transport 50 Gbit/s over 1, 2, or 5 lanes, the serialization delay is 20 ps/bit. So, you'd get 5 bits every 100 ps, regardless of the lanes used.



          Naturally, 100 Gbit/s has half the delay of 50 Gbit/s and so on, so the faster you serialize, the faster a frame is transmitted.



          If you're interested in the internal serialization in the interface you'd need to look at the MII variant that is being used for the speed class. However, this serialization takes place on-the-fly or in parallel with the actual MDI serialization - it does take a minute amount of time but that's up to the actual piece of hardware and probably impossible to predict (something along 2-5 ps would be my guess for 100 Gbit/s). I wouldn't actually worry about this as there are much larger factors involved. 10 ps is the order of transmission latency you'd get from an additional 2 millimeters(!) of cable.



          Using four lanes of 10 Gbit/s each for 40 Gbit/s is NOT the same as aggregating four 10 Gbit/s links. A 40 Gbit/s link - regardless of the number of lanes - can transport a single 40 Gbit/s stream which LAGged 10 Gbit/s links can't. Also, the serialization delay of 40G is only 1/4 that of 10G.






          share|improve this answer






















            Your Answer







            StackExchange.ready(function()
            var channelOptions =
            tags: "".split(" "),
            id: "496"
            ;
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function()
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled)
            StackExchange.using("snippets", function()
            createEditor();
            );

            else
            createEditor();

            );

            function createEditor()
            StackExchange.prepareEditor(
            heartbeatType: 'answer',
            convertImagesToLinks: false,
            noModals: false,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: null,
            bindNavPrevention: true,
            postfix: "",
            noCode: true, onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            );



            );













             

            draft saved


            draft discarded


















            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fnetworkengineering.stackexchange.com%2fquestions%2f53688%2fserialization-and-serialization-times-in-40g-10g-and-100g-25g-ethernet%23new-answer', 'question_page');

            );

            Post as a guest

















            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes








            up vote
            3
            down vote













            This is overthinking.



            The number of lanes used doesn't really matter. Whether you transport 50 Gbit/s over 1, 2, or 5 lanes, the serialization delay is 20 ps/bit. So, you'd get 5 bits every 100 ps, regardless of the lanes used.



            Naturally, 100 Gbit/s has half the delay of 50 Gbit/s and so on, so the faster you serialize, the faster a frame is transmitted.



            If you're interested in the internal serialization in the interface you'd need to look at the MII variant that is being used for the speed class. However, this serialization takes place on-the-fly or in parallel with the actual MDI serialization - it does take a minute amount of time but that's up to the actual piece of hardware and probably impossible to predict (something along 2-5 ps would be my guess for 100 Gbit/s). I wouldn't actually worry about this as there are much larger factors involved. 10 ps is the order of transmission latency you'd get from an additional 2 millimeters(!) of cable.



            Using four lanes of 10 Gbit/s each for 40 Gbit/s is NOT the same as aggregating four 10 Gbit/s links. A 40 Gbit/s link - regardless of the number of lanes - can transport a single 40 Gbit/s stream which LAGged 10 Gbit/s links can't. Also, the serialization delay of 40G is only 1/4 that of 10G.






            share|improve this answer


























              up vote
              3
              down vote













              This is overthinking.



              The number of lanes used doesn't really matter. Whether you transport 50 Gbit/s over 1, 2, or 5 lanes, the serialization delay is 20 ps/bit. So, you'd get 5 bits every 100 ps, regardless of the lanes used.



              Naturally, 100 Gbit/s has half the delay of 50 Gbit/s and so on, so the faster you serialize, the faster a frame is transmitted.



              If you're interested in the internal serialization in the interface you'd need to look at the MII variant that is being used for the speed class. However, this serialization takes place on-the-fly or in parallel with the actual MDI serialization - it does take a minute amount of time but that's up to the actual piece of hardware and probably impossible to predict (something along 2-5 ps would be my guess for 100 Gbit/s). I wouldn't actually worry about this as there are much larger factors involved. 10 ps is the order of transmission latency you'd get from an additional 2 millimeters(!) of cable.



              Using four lanes of 10 Gbit/s each for 40 Gbit/s is NOT the same as aggregating four 10 Gbit/s links. A 40 Gbit/s link - regardless of the number of lanes - can transport a single 40 Gbit/s stream which LAGged 10 Gbit/s links can't. Also, the serialization delay of 40G is only 1/4 that of 10G.






              share|improve this answer
























                up vote
                3
                down vote










                up vote
                3
                down vote









                This is overthinking.



                The number of lanes used doesn't really matter. Whether you transport 50 Gbit/s over 1, 2, or 5 lanes, the serialization delay is 20 ps/bit. So, you'd get 5 bits every 100 ps, regardless of the lanes used.



                Naturally, 100 Gbit/s has half the delay of 50 Gbit/s and so on, so the faster you serialize, the faster a frame is transmitted.



                If you're interested in the internal serialization in the interface you'd need to look at the MII variant that is being used for the speed class. However, this serialization takes place on-the-fly or in parallel with the actual MDI serialization - it does take a minute amount of time but that's up to the actual piece of hardware and probably impossible to predict (something along 2-5 ps would be my guess for 100 Gbit/s). I wouldn't actually worry about this as there are much larger factors involved. 10 ps is the order of transmission latency you'd get from an additional 2 millimeters(!) of cable.



                Using four lanes of 10 Gbit/s each for 40 Gbit/s is NOT the same as aggregating four 10 Gbit/s links. A 40 Gbit/s link - regardless of the number of lanes - can transport a single 40 Gbit/s stream which LAGged 10 Gbit/s links can't. Also, the serialization delay of 40G is only 1/4 that of 10G.






                share|improve this answer














                This is overthinking.



                The number of lanes used doesn't really matter. Whether you transport 50 Gbit/s over 1, 2, or 5 lanes, the serialization delay is 20 ps/bit. So, you'd get 5 bits every 100 ps, regardless of the lanes used.



                Naturally, 100 Gbit/s has half the delay of 50 Gbit/s and so on, so the faster you serialize, the faster a frame is transmitted.



                If you're interested in the internal serialization in the interface you'd need to look at the MII variant that is being used for the speed class. However, this serialization takes place on-the-fly or in parallel with the actual MDI serialization - it does take a minute amount of time but that's up to the actual piece of hardware and probably impossible to predict (something along 2-5 ps would be my guess for 100 Gbit/s). I wouldn't actually worry about this as there are much larger factors involved. 10 ps is the order of transmission latency you'd get from an additional 2 millimeters(!) of cable.



                Using four lanes of 10 Gbit/s each for 40 Gbit/s is NOT the same as aggregating four 10 Gbit/s links. A 40 Gbit/s link - regardless of the number of lanes - can transport a single 40 Gbit/s stream which LAGged 10 Gbit/s links can't. Also, the serialization delay of 40G is only 1/4 that of 10G.







                share|improve this answer














                share|improve this answer



                share|improve this answer








                edited 27 mins ago









                jonathanjo

                6,170423




                6,170423










                answered 59 mins ago









                Zac67

                20.5k21047




                20.5k21047



























                     

                    draft saved


                    draft discarded















































                     


                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function ()
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fnetworkengineering.stackexchange.com%2fquestions%2f53688%2fserialization-and-serialization-times-in-40g-10g-and-100g-25g-ethernet%23new-answer', 'question_page');

                    );

                    Post as a guest
































































                    Comments

                    Post a Comment

                    Popular posts from this blog

                    What does second last employer means? [closed]

                    Image
                    Clash Royale CLAN TAG #URR8PPP .everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty margin-bottom:0; up vote -3 down vote favorite Say, for example I have worked in the following companies, Company A => 2010 - 2011 Company B => 2011 - 2012 Company C => 2012 - Till Date Company D => Have offer Now assume Company D is asking to submit documents from my second last employer. So from the list of companies given, which company would be qualified as second last employer? software-industry employer share | improve this question asked Aug 10 '14 at 5:16 Rajaprabhu Aravindasamy 559 1 6 13 closed as off-topic by Jim G., jmort253 ♦ Aug 11 '14 at 1:58 This question appears to be off-topic. The users who voted to close gave this specific reason: "Questions seeking advice on company-specific regulations, agreements, or policies should be directed to your manager or HR department. Questions that address...
                    Read more

                    List of Gilmore Girls characters

                    Image
                    The season three cast—(Back row): Lane, Michel, Paris, Emily, Richard, Sookie, Miss Patty, Kirk; Front row: Jess, Luke, Lorelai, Rory, Dean This is a list of characters for the comedy-drama television series Gilmore Girls . Contents 1 Main characters 1.1 Lorelai Gilmore 1.2 Rory Gilmore 1.3 Sookie St. James 1.4 Lane Kim 1.5 Michel Gerard 1.6 Luke Danes 1.7 Emily Gilmore 1.8 Richard Gilmore 1.9 Paris Geller 1.10 Dean Forester 1.11 Jess Mariano 1.12 Kirk Gleason 1.13 Jason Stiles 1.14 Logan Huntzberger 2 Major recurring characters 2.1 Christopher Hayden 2.2 Jackson Belleville 2.3 Mrs. Kim 2.4 Tristin DuGray 2.5 Max Medina 2.6 Taylor Doose 2.7 Dave Rygalski 2.8 Marty 3 Recurring characters 3.1 Stars Hollow 3.2 Chilton 3.3 Yale 3.4 Other 4 Notable guest stars 5 Unseen characters 6 Notes 7 References Main characters Actor Character Appearances Season 1 Season 2 Season 3 Season 4 Season 5 Season 6 Season 7 A Year in the Life Lauren Graham Lorelai Gilmore...
                    Read more

                    One-line joke

                    A one-liner is a joke that is delivered in a single line. A good one-liner is said to be pithy - concise and meaningful. [1] Comedians and actors use this comedic method as part of their act, e.g. Jimmy Carr, Tommy Cooper, Rodney Dangerfield, Norm Macdonald, Ken Dodd, Stewart Francis, Zach Galifianakis, Mitch Hedberg, Anthony Jeselnik, Milton Jones, Shappi Khorsandi, Jay London, Mark Linn-Baker, Demetri Martin, Groucho Marx, Dan Mintz, Emo Philips, Tim Vine, Steven Wright and Henny Youngman and Arnold Schwarzenegger. [ citation needed ] Many fictional characters are also known to deliver one-liners, including James Bond, who usually includes pithy and laconic quips after disposing of a villain. [ citation needed ] Examples "A baby seal walks into a club." "A dyslexic man walks into a bra." (Anonymous) "There are three types of people, those who can count and those who can't." "Venison's dear, isn't it?" (Jimmy Carr) "An escala...
                    Read more