ALERT: NTP-Based Distributed Denial of Service Attacks
Prevent your institution from being an unwitting partner in these attacks
This is the TECH version of the Alert.
There is also a CIO version.
- - - - - - - - -
March 10, 2014 To: IT Security Staff and Network and System Administrators (A CIO version of this Alert is available at ) REN-ISAC ALERT: NTP-Based Distributed Denial of Service Attacks - Prevent your institution from being an unwitting partner in these attacks The REN-ISAC  wants to raise awareness and drive change concerning common network time protocol (NTP) and network configurations that fall short of best practices and which, if left uncorrected, open the door for your institution to be exploited as an unwitting partner in delivering crippling distributed denial of service (DDoS) attacks against third parties. In a companion note to CIOs, the REN-ISAC recommends the following: === ACTIONS === 1. Distribute a copy of this message to your network administrators, information security staff, system administrators, and other relevant personnel. 2. Identify hosts on your network with ntpd installed and running. Disable "monlist" capabilities on those NTP servers. Or, for systems which cannot be updated or configured to eliminate the monlist risk, apply network filters to prevent external requests to these insecure systems. 3. Ensure your institutional networks are unable to originate Internet traffic with spoofed source addresses as described elsewhere in this document. ========== Although DDoS attacks exploiting the NTP and network configuration weaknesses have been around for a long time, the frequency and impact of these attacks have grown over the past year. The traffic sourced by any single system may be small enough to go unnoticed at an institution, however, the aggregate traffic from many sources, as experienced at the target, can be crippling. Recent attacks [2,3,4] average around 7.3 gigabits per second of traffic aimed at the victim organization - more than the full Internet bandwidth of most small and medium-sized colleges and universities. Given the history and success of recent attacks, we expect these attacks will rise in frequency and magnitude in the months ahead. There are two issues: 1. NTP is a protocol used to synchronize the clocks of networked devices. A deprecated command, "monlist", permits a requesting computer to receive information regarding the last 600 connections made to the NTP server. A small input request can generate a large response. When a malicious actor spoofs the source IP address of a victim targeted for attack, and repeatedly sends monlist requests to thousands of insecure NTP servers located all over the Internet, an avalanche of traffic is directed at the victim, overwhelming the victim's network capacity. Monlist has been removed from newer versions of NTP and monlist can (and should) be disabled in older versions. The NTP issue is complicated by the fact that not only is NTP running on upgradeable enterprise servers, NTP is also employed on infrastructure devices and embedded systems in your enterprise, some of which can be updated only with great difficulty and expense. Concerning these "difficult" systems, network firewalls may be one approach that can be used to mitigate the problem of these vulnerable-but-uncorrectable devices. 2. The second issue, the network configuration issue, involves the problem of traffic with forged apparent source addresses. Systems on your network should not be able to transmit packets that appear to be from a source IP address that doesn't belong to you -- your systems should only originate traffic with accurate source IP information. In addition to the problem of educational institutions being leveraged as the sources of attacks against third parties, there have been incidents of institutions being targeted as the victim of DDoS attacks. Best practices for mitigating DDoS attacks against your institution apply: know in advance how to partner with your ISP to mitigate the effects of an attack; concerning Internet2, refer to the "DDoS Attacks Policy" . DDoS mitigation services are also available from commercial organizations. ============================= TECHNICAL AND POLICY CONTROLS ============================= ===== Overview ===== While NTP is the DDoS amplification mechanism de jour, the problem encompasses a number of network service protocols , e.g. see our earlier Alert concerning open recursive DNS servers . Too many higher education institutions contribute to this avoidable problem. Unfortunately, this problem goes unsolved because organizations targeted in the attacks are not the same organizations that fail to follow best common practices (and which end up being exploited to conduct the attacks). The exploited organization often experiences little ill effect; therefore, motivation to solve the problem depends on each site's willingness to go the extra mile and be a good Internet citizen, even if doing so doesn't result in benefits to them directly. ===== Solutions ===== [ NTP ] Highly Recommended: Concerning systems on which NTP can be updated or configured: Upgrade NTP to version 4.2.7p26 or higher  If upgrade isn't possible, disable monlist in the ntp.conf file  Concerning all other (e.g. infrastructure devices and embedded systems): Manage NTP traffic (inbound destination port 123 tcp/udp), e.g. by using router ACLs, so that queries from outside the enterprise can only reach intentionally permitted enterprise time servers. Do NOT blindly block all incoming and outgoing NTP traffic, because NTP serves an important role in synchronizing system clocks. Review the Team Cymru Secure NTP template  for inclusion in your router template. Depending on your environment it may be difficult to limit queries from inside the enterprise to external servers (outbound) because many infrastructure devices and embedded systems rely on preconfigured NTP services. Recommended: Provide a means to monitor NTP for evidence of abuse. [ Network ] Highly Recommended: Apply BCP38/BCP84 filtering to prevent spoofed source address traffic from leaving your network.  Recommended: Collect and store network flow (NetFlow/Sflow/J-Flow) data. Real time network flow allows backtracking spoofed network traffic. Historical network flow facilitates incident response capabilities. ===== More In-Depth ===== [ NTP monlist ] NTP monlist is a remote command used in older versions of NTP for monitoring which hosts have connected to the server. Upon request, a list of the last 600 hosts will be sent. Monlist is a deprecated command, but in older versions of NTP, it is still enabled. New versions of NTP (after 4.2.7p26) have removed monlist capabilities in favor of ntpq mrulist . The "Most Recently Used" (MRU) functionality also includes the ability to rate limit and "Kiss-of- Death" packets, which explicitly request the client to stop sending and leaves a message for the system operator . You can request a free report of NTP servers on your network from the OpenNTP project group . [ Network Filtering to Prevent Source-Spoofed Packets ] Systems should not be permitted to send spoofed traffic to the Internet, pretending to be traffic from some other site's IP addresses. Roughly 80% of all networks have already installed filtering rules on their network routers to ensure any spoofed network traffic won't hit the Internet, but some networks -- including potentially yours -- have not yet done so. We need your help. Please ensure your institutional networks prevent traffic with spoofed source addresses from leaving your network. Blocking spoofed network traffic from leaving your network is an IETF Best Common Practice ("BCP"), see: http://tools.ietf.org/html/bcp38 and http://tools.ietf.org/html/bcp84 Filtering can be done at the subnet level, or at the institutional border, or both. ===== Additional Considerations ===== Unrelated to NTP amplification DDoS attacks, but of related concern to NTP, is the local denial of service (DoS) vulnerability involving NTP Mode 7 messages . Upgrading NTP to a current version or applying a network firewall rule to restrict inbound NTP traffic to intentionally permitted servers will eliminate/reduce this local DoS vulnerability. ========== Print-friendly copies of this Alert and the companion CIO version (with clobber-free long URLs) are available at the REN-ISAC web site: http://www.ren-isac.net/alerts.html. We'd appreciate your input on additional means to protect from this threat, and general feedback concerning this Alert. If you have any questions, please don't hesitate to e-mail us at firstname.lastname@example.org. Special thanks go to the members of the REN-ISAC Technical Advisory Group (TAG) . On behalf of the REN-ISAC team, Doug Pearson email@example.com Technical Director, REN-ISAC http://www.ren-isac.net 24x7 Watch Desk +1(317)274-7228 ========== References ==========  REN-ISAC http://www.ren-isac.net  Hackers Spend Christmas Break Launching Large Scale NTP-Reflection Attacks http://www.symantec.com/connect/blogs/hackers-spend-christmas-break-launching-large-scale-ntp-reflection-attacks  DoS attacks that took down big game sites abused Web's time-sync protocol http://arstechnica.com/security/2014/01/dos-attacks-that-took-down-big-game-sites-abused-webs-time-synch-protocol/  NTP reflection attack https://isc.sans.edu/diary/NTP+reflection+attack/17300  Internet2 DDoS Attacks Policy https://wiki.internet2.edu/confluence/display/network/Forms%2C+Maps%2C+Policies%2C+and+Procedures  TECH version of this alert http://www.ren-isac.net/alerts/REN-ISAC_Alert_NTP_Amp_DDoS_CIO_201403.html  REN-ISAC Technical Advisory Group http://www.ren-isac.net/about/advisory.html#technical  remove ntpd support for ntpdc's monlist (use ntpq's mrulist) http://bugs.ntp.org/show_bug.cgi?id=1532  DRDoS / Amplification Attack using ntpdc monlist command http://support.ntp.org/bin/view/Main/SecurityNotice#DRDoS_Amplification_Attack_using  Secure NTP Template https://www.team-cymru.org/ReadingRoom/Templates/secure-ntp-template.html  Network Ingress Filtering http://tools.ietf.org/html/bcp38  Ingress Filtering for Multihomed Networks http://tools.ietf.org/html/bcp84  Securing the Edge http://www.icann.org/en/groups/ssac/documents/sac-004-en.pdf  Access Control Options http://doc.ntp.org/4.2.2p2/accopt.html  For a list of open NTP by ASN, e-mail ntp-scan /at/puck.nether.net http://openntpproject.org/  Amplification Hell: Revisiting Network Protocols for DDoS Abuse http://www.internetsociety.org/sites/default/files/01_5.pdf  ALERT: Prevent your institution from being an unwitting partner in denial of service attacks http://www.ren-isac.net/alerts/dns_amp_ddos_tech_201305.html  NTP mode 7 denial-of-service vulnerability http://www.kb.cert.org/vuls/id/568372 [ Other Resources ] In addition to the references made in the text above, the following may be useful: Alternative to NTP - OpenNTPD http://www.openntpd.org/ NTP DoS reflection attacks https://cert.litnet.lt/en/docs/ntp-distributed-reflection-dos-attacks A Free Solution For DDoS Reflection Attacks: A Decade In Waiting http://blog.trendmicro.com/trendlabs-security-intelligence/a-free-solution-for-ddos-reflection-attacks-a-decade-in-waiting/ NTP Reflections https://labs.ripe.net/Members/mirjam/ntp-reflections -o0o-