> I looked at your network diagram. Try checking the configuration of the > Ethernet ports on the firewall and the Asterisk box. Make sure they are > set to auto-negotiate and not set to a fixed speed and fixed duplex. > I have found in the past that if one end of a link is expecting auto- > negotiation (as the switches probably are) and the other end is expecting > a fixed configuration, things can degrade to half-duplex trying to talk > to full-duplex, resulting in lots of collisions and packet loss when there > is any kind of significant traffic. > > Your description would be consistent with the firewall introducing lots of > LAN collisions when busy, in the central gigabit switch, even if the VoIP > traffic isn't passing through the firewall.Also, check the wiring. Check each individual RJ-45 jumper, *and* the in-house wiring, with a proper tester that can verify that the individual pairs are hooked up correctly. I've seen all kinds of hell occur, in situations where somebody used telco-type RJ-45 connecting cables, in place of proper Ethernet connecting cables. The problem is this: in a telco RJ-45 cable (such as was/is often used for proprietary telephone systems) the individual wires are either not in twisted pairs, or are twisted-pairs in a 1-2 3-4 5-6 7-8 arrangement. These work fine for analog connections. They're latent-death-on-wheels for Ethernet. Ethernet only works well if you connect the pairs as a 1-2, 3-6, 4-5, 7-8 arrangement, because this is how the signals are sent electrically. Using the correct connections ensures that the signals on each pair are "balanced" electrically - that is, the two wires in each twisted pair are carrying equal-but-opposite currents for the two sides of an individual signal. This minimizes electrical coupling between pairs, and thus minimizes crosstalk. If you use a telco-style cable (these are often black, and flat), or if you use what looks like an Ethernet cable but which had its wires "punched down" to the connector in the wrong pairing, things go very badly indeed. One twisted pair might be carrying one TX signal and one RX signal. This pretty much *guarantees* terrible cross-talk between the two. The symptoms of this can be as was related... the connection appears to work OK under light load, when there's usually traffic flowing in only one direction at a time. However, when you put a bidirectional load on the connection, the signals going from A to B and from B to A cross-talk with one another, leading to a very high rate of corrupted/dropped packets on the network. This will often show up in the end device's Ethernet packet statistics, if you can get to them... look for a high rate of dropped or "bad" packets, FCS (frame sequence check) errors, etc. I've seen a fair number of cheap "Ethernet" cables that had been manufactured wrong. You should see a color pairing such as http://www.hardwaresecrets.com/how-gigabit-ethernet-works/ indicates - pins 4 and 5 are a pair (blue, and white-and-blue), and the next-outer pins are also a pair (orange, and white-with-orange). If you see a pattern such as "white-with-green, green, white-with-blue, blue, white-with-orange, orange, white-with-brown, brown" where there are four color-matched pairs of wires next to one another, you've got a bad cable. The same error can occur when building wiring is "punched down" to the RJ-45 jacks. A good Ethernet cable-pair tester can spot such things pretty quickly.
On Wednesday 04 April 2018 at 00:20:29, Dave Platt wrote:> Also, check the wiring. Check each individual RJ-45 jumper, *and* the > in-house wiring, with a proper tester that can verify that the > individual pairs are hooked up correctly.Can you provide a link to such a tester?> The problem is this: in a telco RJ-45 cable (such as was/is often used > for proprietary telephone systems) the individual wires are either not > in twisted pairs, or are twisted-pairs in a 1-2 3-4 5-6 7-8 arrangement. > These work fine for analog connections. They're latent-death-on-wheels > for Ethernet.> A good Ethernet cable-pair tester can spot such things pretty quickly.I disagree. *Certainly*, incorrect pair terminations can cause the sort of problems described, however I haven't yet come across a cable tester which can identify that a cable correctly connected from end to end with wires {1..8} <-> {1..8} is in fact not correctly connected in ethernet 1-2 3-6 4-5 7-8 pairs. All cable testers I have so far encountered will check that all wires are correctly connected end to end and not cross connected etc., but have no clue whether the wire joining pin 3 at one end to pin 3 at the other end is twisted together with the cable from pin 4 or pin 6. I would be very interested to find such a tester, if you can point us at one. Thanks, Antony. -- "Can you keep a secret?" "Well, I shouldn't really tell you this, but... no." Please reply to the list; please *don't* CC me.
At the first office, I replaced all the wiring except the in-wall stuff.? Checked all the cables to make sure they were correct.? I've done cabling for the last 20+ years, so I've usually got a good feel for that.? Make all my own cables and do all my own wiring.? I still make a habit of checking that first because you never know when somebody is going to decide to swap out a cable with one they just pulled out of hammerspace for one reason or another. All of the duplex and flow control settings are set to auto-negotiate.? The switch logs don't show any unexpected amount of collisions, and no receive or transmit errors. I might add that I have the same setup in 8 offices.? Right now, only two of the offices are reporting problems.? All of these offices were previously operating fine with Asterisk 1.4 installations.? Over the past year all offices were upgraded to new phone/fax servers running Asterisk 13.? All offices ran fine for several months until the one problem office started having the audio drop-outs, and then a few weeks later the second office started having the same issue. Is there anything in the pjsip code that might cause RTP latency if reverse DNS lookup timed out for one reason or another? ** On 4/3/2018 5:20 PM, Dave Platt wrote:>> I looked at your network diagram. Try checking the configuration of the >> Ethernet ports on the firewall and the Asterisk box. Make sure they are >> set to auto-negotiate and not set to a fixed speed and fixed duplex. >> I have found in the past that if one end of a link is expecting auto- >> negotiation (as the switches probably are) and the other end is expecting >> a fixed configuration, things can degrade to half-duplex trying to talk >> to full-duplex, resulting in lots of collisions and packet loss when there >> is any kind of significant traffic. >> >> Your description would be consistent with the firewall introducing lots of >> LAN collisions when busy, in the central gigabit switch, even if the VoIP >> traffic isn't passing through the firewall. > Also, check the wiring. Check each individual RJ-45 jumper, *and* the > in-house wiring, with a proper tester that can verify that the > individual pairs are hooked up correctly. > > I've seen all kinds of hell occur, in situations where somebody used > telco-type RJ-45 connecting cables, in place of proper Ethernet > connecting cables. > > The problem is this: in a telco RJ-45 cable (such as was/is often used > for proprietary telephone systems) the individual wires are either not > in twisted pairs, or are twisted-pairs in a 1-2 3-4 5-6 7-8 arrangement. > These work fine for analog connections. They're latent-death-on-wheels > for Ethernet. > > Ethernet only works well if you connect the pairs as a 1-2, 3-6, 4-5, > 7-8 arrangement, because this is how the signals are sent electrically. > Using the correct connections ensures that the signals on each pair are > "balanced" electrically - that is, the two wires in each twisted pair > are carrying equal-but-opposite currents for the two sides of an > individual signal. This minimizes electrical coupling between pairs, > and thus minimizes crosstalk. > > If you use a telco-style cable (these are often black, and flat), or if > you use what looks like an Ethernet cable but which had its wires > "punched down" to the connector in the wrong pairing, things go very > badly indeed. One twisted pair might be carrying one TX signal and one > RX signal. This pretty much *guarantees* terrible cross-talk between > the two. > > The symptoms of this can be as was related... the connection appears to > work OK under light load, when there's usually traffic flowing in only > one direction at a time. However, when you put a bidirectional load on > the connection, the signals going from A to B and from B to A cross-talk > with one another, leading to a very high rate of corrupted/dropped > packets on the network. > > This will often show up in the end device's Ethernet packet statistics, > if you can get to them... look for a high rate of dropped or "bad" > packets, FCS (frame sequence check) errors, etc. > > I've seen a fair number of cheap "Ethernet" cables that had been > manufactured wrong. You should see a color pairing such as > > http://www.hardwaresecrets.com/how-gigabit-ethernet-works/ > > indicates - pins 4 and 5 are a pair (blue, and white-and-blue), and the > next-outer pins are also a pair (orange, and white-with-orange). > > If you see a pattern such as "white-with-green, green, white-with-blue, > blue, white-with-orange, orange, white-with-brown, brown" where there > are four color-matched pairs of wires next to one another, you've got a > bad cable. > > The same error can occur when building wiring is "punched down" to the > RJ-45 jacks. > > A good Ethernet cable-pair tester can spot such things pretty quickly. >--- This email has been checked for viruses by AVG. http://www.avg.com WARNING-FRAUDULENT FUNDING INSTRUCTIONS Email hacking and fraud are on the rise to fraudulently misdirect funds. Please call your escrow officer immediately using contract information found from an independent source, such as the sales contract or internet, to verify any funding instructions received. We are not responsible for any wires sent by you to an incorrect bank account. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20180404/ab66141c/attachment-0001.html>
Been doing some more troubleshooting on this issue.? Started up TCPDump and let it run for a whole day, then loaded the file into Wireshark.? What is interesting is that there doesn't seem to be any lost packets.? The RTP sequence numbers are always contiguous. However, if I output the streams as .au files and listen to them there are obvious points where the audio just goes silent in the middle of the person speaking, and it effects both directions. Doesn't make any sense. On 4/4/2018 10:33 AM, Brent Davidson wrote:> At the first office, I replaced all the wiring except the in-wall > stuff.? Checked all the cables to make sure they were correct. I've > done cabling for the last 20+ years, so I've usually got a good feel > for that.? Make all my own cables and do all my own wiring.? I still > make a habit of checking that first because you never know when > somebody is going to decide to swap out a cable with one they just > pulled out of hammerspace for one reason or another. > > All of the duplex and flow control settings are set to > auto-negotiate.? The switch logs don't show any unexpected amount of > collisions, and no receive or transmit errors. > > I might add that I have the same setup in 8 offices.? Right now, only > two of the offices are reporting problems.? All of these offices were > previously operating fine with Asterisk 1.4 installations.? Over the > past year all offices were upgraded to new phone/fax servers running > Asterisk 13.? All offices ran fine for several months until the one > problem office started having the audio drop-outs, and then a few > weeks later the second office started having the same issue. > > Is there anything in the pjsip code that might cause RTP latency if > reverse DNS lookup timed out for one reason or another? > > > On 4/3/2018 5:20 PM, Dave Platt wrote: >>> I looked at your network diagram. Try checking the configuration of the >>> Ethernet ports on the firewall and the Asterisk box. Make sure they are >>> set to auto-negotiate and not set to a fixed speed and fixed duplex. >>> I have found in the past that if one end of a link is expecting auto- >>> negotiation (as the switches probably are) and the other end is expecting >>> a fixed configuration, things can degrade to half-duplex trying to talk >>> to full-duplex, resulting in lots of collisions and packet loss when there >>> is any kind of significant traffic. >>> >>> Your description would be consistent with the firewall introducing lots of >>> LAN collisions when busy, in the central gigabit switch, even if the VoIP >>> traffic isn't passing through the firewall. >> Also, check the wiring. Check each individual RJ-45 jumper, *and* the >> in-house wiring, with a proper tester that can verify that the >> individual pairs are hooked up correctly. >> >> I've seen all kinds of hell occur, in situations where somebody used >> telco-type RJ-45 connecting cables, in place of proper Ethernet >> connecting cables. >> >> The problem is this: in a telco RJ-45 cable (such as was/is often used >> for proprietary telephone systems) the individual wires are either not >> in twisted pairs, or are twisted-pairs in a 1-2 3-4 5-6 7-8 arrangement. >> These work fine for analog connections. They're latent-death-on-wheels >> for Ethernet. >> >> Ethernet only works well if you connect the pairs as a 1-2, 3-6, 4-5, >> 7-8 arrangement, because this is how the signals are sent electrically. >> Using the correct connections ensures that the signals on each pair are >> "balanced" electrically - that is, the two wires in each twisted pair >> are carrying equal-but-opposite currents for the two sides of an >> individual signal. This minimizes electrical coupling between pairs, >> and thus minimizes crosstalk. >> >> If you use a telco-style cable (these are often black, and flat), or if >> you use what looks like an Ethernet cable but which had its wires >> "punched down" to the connector in the wrong pairing, things go very >> badly indeed. One twisted pair might be carrying one TX signal and one >> RX signal. This pretty much *guarantees* terrible cross-talk between >> the two. >> >> The symptoms of this can be as was related... the connection appears to >> work OK under light load, when there's usually traffic flowing in only >> one direction at a time. However, when you put a bidirectional load on >> the connection, the signals going from A to B and from B to A cross-talk >> with one another, leading to a very high rate of corrupted/dropped >> packets on the network. >> >> This will often show up in the end device's Ethernet packet statistics, >> if you can get to them... look for a high rate of dropped or "bad" >> packets, FCS (frame sequence check) errors, etc. >> >> I've seen a fair number of cheap "Ethernet" cables that had been >> manufactured wrong. You should see a color pairing such as >> >> http://www.hardwaresecrets.com/how-gigabit-ethernet-works/ >> >> indicates - pins 4 and 5 are a pair (blue, and white-and-blue), and the >> next-outer pins are also a pair (orange, and white-with-orange). >> >> If you see a pattern such as "white-with-green, green, white-with-blue, >> blue, white-with-orange, orange, white-with-brown, brown" where there >> are four color-matched pairs of wires next to one another, you've got a >> bad cable. >> >> The same error can occur when building wiring is "punched down" to the >> RJ-45 jacks. >> >> A good Ethernet cable-pair tester can spot such things pretty quickly. >> > > > > ------------------------------------------------------------------------ > AVG logo <http://www.avg.com/internet-security> > > This email has been checked for viruses by AVG antivirus software. > www.avg.com <http://www.avg.com/internet-security> > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > *WARNING-FRAUDULENT FUNDING INSTRUCTIONS* > > Email hacking and fraud are on the rise to fraudulently misdirect > funds. Please call your escrow officer immediately using contract > information found from an independent source, such as the sales > contract or internet, to verify any funding instructions received. We > are not responsible for any wires sent by you to an incorrect bank > account.--- This email has been checked for viruses by AVG. http://www.avg.com WARNING-FRAUDULENT FUNDING INSTRUCTIONS Email hacking and fraud are on the rise to fraudulently misdirect funds. Please call your escrow officer immediately using contract information found from an independent source, such as the sales contract or internet, to verify any funding instructions received. We are not responsible for any wires sent by you to an incorrect bank account. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20180405/d69a076d/attachment.html>