Merged PR 13190: new topics for tcp/ip troubleshooting

windows/client-management/TOC.md

@@ -18,5 +18,10 @@
 ### [Advanced troubleshooting Wireless Network Connectivity](advanced-troubleshooting-wireless-network-connectivity.md)
 ### [Advanced troubleshooting for Windows-based computer freeze issues](troubleshoot-windows-freeze.md)
 ### [Advanced troubleshooting for Stop error or blue screen error issue](troubleshoot-stop-errors.md)
+### [Advanced troubleshooting for TCP/IP](troubleshoot-tcpip.md)
+#### [Collect data using Network Monitor](troubleshoot-tcpip-netmon.md)
+#### [Troubleshoot TCP/IP connectivity](troubleshoot-tcpip-connectivity.md)
+#### [Troubleshoot port exhaustion issues](troubleshoot-tcpip-port-exhaust.md)
+#### [Troubleshoot Remote Procedure Call (RPC) errors](troubleshoot-tcpip-rpc-errors.md)
 ## [Mobile device management for solution providers](mdm/index.md)
 ## [Change history for Client management](change-history-for-client-management.md)

windows/client-management/change-history-for-client-management.md

@@ -9,13 +9,23 @@ ms.pagetype: security
 ms.localizationpriority: medium
 author: jdeckerMS
 ms.author: jdecker
-ms.date: 11/30/2018
+ms.date: 12/06/2018
 ---
 
 # Change history for Client management
 
 This topic lists new and updated topics in the [Client management](index.md) documentation for Windows 10 and Windows 10 Mobile.
 
+## December 2018
+
+New or changed topic | Description
+--- | ---
+[Advanced troubleshooting for TCP/IP](troubleshoot-tcpip.md) | New
+[Collect data using Network Monitor](troubleshoot-tcpip-netmon.md) | New
+[Troubleshoot TCP/IP connectivity](troubleshoot-tcpip-connectivity.md) | New
+[Troubleshoot port exhaustion issues](troubleshoot-tcpip-port-exhaust.md) | New
+[Troubleshoot Remote Procedure Call (RPC) errors](troubleshoot-tcpip-rpc-errors.md) | New
+
 ## November 2018
 
 New or changed topic | Description

windows/client-management/troubleshoot-tcpip-connectivity.md

@@ -0,0 +1,109 @@
+---
+title: Troubleshoot TCP/IP connectivity
+description: Learn how to troubleshoot TCP/IP connectivity.
+ms.prod: w10
+ms.sitesec: library
+ms.topic: troubleshooting
+author: kaushika-msft
+ms.localizationpriority: medium
+ms.author: kaushika
+ms.date: 12/06/2018
+---
+
+# Troubleshoot TCP/IP connectivity
+
+You might come across connectivity errors on the application end or timeout errors. Most common scenarios would include application connectivity to a database server, SQL timeout errors, BizTalk application timeout errors, Remote Desktop Protocol (RDP) failures, file share access failures, or general connectivity.  
+ 
+When you suspect that the issue is on the network, you collect a network trace. The network trace would then be filtered. During troubleshooting connectivity errors, you might come across TCP reset in a network capture which could indicate a network issue.  
+ 
+* TCP is defined as connection-oriented and reliable protocol. One of the ways in which TCP ensures this is through the handshake process. Establishing a TCP session would begin with a 3-way handshake, followed by data transfer, and then a 4-way closure. The 4-way closure where both sender and receiver agree on closing the session is termed as *graceful closure*. After the 4-way closure, the server will allow 4 minutes of time (default), during which any pending packets on the network are to be processed, this is the TIME_WAIT state. Once the TIME_WAIT state is done, all the resources allocated for this connection are released.  
+ 
+* TCP reset is an abrupt closure of the session which causes the resources allocated to the connection to be immediately released and all other information about the connection is erased.  
+ 
+* TCP reset is identified by the RESET flag in the TCP header set to `1`.  
+ 
+A network trace on the source and the destination which will help you determine the flow of the traffic and see at what point the failure is observed.  
+ 
+The following sections describe some of the scenarios when you will see a RESET. 
+ 
+## Packet drops
+ 
+When one TCP peer is sending out TCP packets for which there is no response received from the other end, the TCP peer would end up re-transmitting the data and when there is no response received, it would end the session by sending an ACK RESET( meaning, application acknowledges whatever data exchanged so far, but due to packet drop closing the connection).  
+ 
+The simultaneous network traces on source and destination will help you verify this behavior where on the source side you would see the packets being retransmitted and on the destination none of these packets are seen. This would mean, the network device between the source and destination is dropping the packets. 
+ 
+If the initial TCP handshake is failing because of packet drops then you would see that the TCP SYN packet is retransmitted only 3 times. 
+    
+Source side connecting on port 445:
+
+![](images/tcp-ts-6.png)
+
+Destination side: applying the same filter, you do not see any packets.
+
+![](images/tcp-ts-7.png)
+
+For the rest of the data, TCP will retransmit the packets 5 times.
+
+**Source 192.168.1.62 side trace:**
+
+![](images/tcp-ts-8.png)
+
+**Destination 192.168.1.2 side trace:**
+     
+You would not see any of the above packets. Engage your network team to investigate with the different hops and see if any of them are potentially causing drops in the network.
+    
+If you are seeing that the SYN packets are reaching the destination, but the destination is still not responding, then verify if the port that you are trying to connect to is in the listening state. (Netstat output will help). If the port is listening and still there is no response, then there could be a wfp drop.  
+ 
+## Incorrect parameter in the TCP header
+ 
+You see this behavior when the packets are modified in the network by middle devices and TCP on the receiving end is unable to accept the packet, such as the sequence number being modified, or packets being re-played by middle device by changing the sequence number. Again, the simultaneous network trace on the source and destination will be able to tell you if any of the TCP headers are modified. Start by comparing the source trace and destination trace, you will be able to notice if there is a change in the packets itself or if any new packets are reaching the destination on behalf of the source.  
+ 
+In this case, you will again need help from the network team to identify any such device which is modifying packets or re-playing packets to the destination. The most common ones are RiverBed devices or WAN accelerators. 
+ 
+     
+## Application side reset
+ 
+When you have identified that the resets are not due to retransmits or incorrect parameter or packets being modified with the help of network trace, then you have narrowed it down to application level reset.
+    
+The application resets are the ones where you see the Acknowledgement flag set to `1` along with the reset flag. This would mean that the server is acknowledging the receipt of the packet but for some reason it will not accept the connection. This is when the application that received the packet did not like something it received.  
+ 
+In the below screenshots, you see that the packets seen on the source and the destination are the same without any modification or any drops, but you see an explicit reset sent by the destination to the source.
+    
+**Source Side**
+
+![](images/tcp-ts-9.png)
+
+**On the destination-side trace** 
+
+![](images/tcp-ts-10.png)
+
+You also see an ACK+RST flag packet in a case when the TCP establishment packet SYN is sent out. The TCP SYN packet is sent when the client wants to connect on a particular port, but if the destination/server for some reason does not want to accept the packet, it would send an ACK+RST packet.  
+
+![](images/tcp-ts-11.png)
+
+The application which is causing the reset (identified by port numbers) should be investigated to understand what is causing it to reset the connection. 
+ 
+>[!Note]
+>The above information is about resets from a TCP standpoint and not UDP. UDP is a connectionless protocol and the packets are sent unreliably. You would not see retransmission or resets when using UDP as a transport protocol. However, UDP makes use of ICMP as a error reporting protocol. When you have the UDP packet sent out on a port and the destination does not have port listed, you will see the destination sending out  **ICMP Destination host unreachable: Port unreachable** message immediately after the UDP packet
+ 
+ 
+```typescript
+10.10.10.1  10.10.10.2  UDP UDP:SrcPort=49875,DstPort=3343
+ 
+10.10.10.2  10.10.10.1  ICMP    ICMP:Destination Unreachable Message, Port Unreachable,10.10.10.2:3343
+```
+ 
+ 
+During the course of troubleshooting connectivity issue, you might also see in the network trace that a machine receives packets but does not respond to. In such cases, there could be a drop at the server level. You should enable firewall auditing on the machine to understand if the local firewall is dropping the packet.
+ 
+```typescript
+auditpol /set /subcategory:"Filtering Platform Packet Drop" /success:enable /failure:enable
+```
+ 
+You can then review the Security event logs to see for a packet drop on a particular port-IP and a filter ID associated with it.
+
+![](images/tcp-ts-12.png)
+
+Now, run the command `netsh wfp show state`, this will generate a wfpstate.xml file. Once you open this file and filter for the ID you find in the above event (2944008), you will be able to see a firewall rule name associated with this ID which is blocking the connection.
+
+![](images/tcp-ts-13.png)

windows/client-management/troubleshoot-tcpip-netmon.md

@@ -0,0 +1,60 @@
+---
+title: Collect data using Network Monitor
+description: Learn how to run Network Monitor to collect data for troubleshooting TCP/IP connectivity.
+ms.prod: w10
+ms.sitesec: library
+ms.topic: troubleshooting
+author: kaushika-msft
+ms.localizationpriority: medium
+ms.author: kaushika
+ms.date: 12/06/2018
+---
+
+# Collect data using Network Monitor
+
+In this topic, you will learn how to use Microsoft Network Monitor 3.4, which is a tool for capturing network traffic.
+
+To get started, [download and run NM34_x64.exe](https://www.microsoft.com/download/details.aspx?id=4865). When you install Network Monitor, it installs its driver and hooks it to all the network adapters installed on the device. You can see the same on the adapter properties, as shown in the following image.
+
+![A view of the properties for the adapter](images/tcp-ts-1.png)
+
+When the driver gets hooked to the network interface card (NIC) during installation, the NIC is reinitialized, which might cause a brief network glitch.
+
+**To capture traffic**
+
+1. Click **Start** and enter **Netmon**.
+
+2. For **netmon run command**,select **Run as administrator**.
+
+    ![Image of Start search results for Netmon](images/tcp-ts-3.png)
+
+3. Network Monitor opens with all network adapters displayed. Select **New Capture**, and then select **Start**.
+
+    ![Image of the New Capture option on menu](images/tcp-ts-4.png)
+
+4. Reproduce the issue, and you will see that Network Monitor grabs the packets on the wire.
+
+    ![Frame summary of network packets](images/tcp-ts-5.png)
+
+5. Select **Stop**, and go to **File > Save as** to save the results. By default, the file will be saved as a ".cap" file.
+
+The saved file has captured all the traffic that is flowing to and from the network adapters of this machine. However, your interest is only to look into the traffic/packets that are related to the specific connectivity problem you are facing. So you will need to filter the network capture to see only the related traffic. 
+ 
+**Commonly used filters**
+ 
+- Ipv4.address=="client ip" and ipv4.address=="server ip"
+- Tcp.port==
+- Udp.port==
+- Icmp 
+- Arp 
+- Property.tcpretranmits
+- Property.tcprequestfastretransmits
+- Tcp.flags.syn==1
+ 
+>[!TIP]
+>If you want to filter the capture for a specific field and do not know the syntax for that filter, just right-click that field and select **Add *the selected value* to Display Filter**. 
+ 
+Network traces which are collected using the **netsh** commands built in to Windows are of the extension "ETL". However, these ETL files can be opened using Network Monitor for further analysis. 
+
+
+

windows/client-management/troubleshoot-tcpip-port-exhaust.md

@@ -0,0 +1,196 @@
+---
+title: Troubleshoot port exhaustion issues
+description: Learn how to troubleshoot port exhaustion issues.
+ms.prod: w10
+ms.sitesec: library
+ms.topic: troubleshooting
+author: kaushika-msft
+ms.localizationpriority: medium
+ms.author: kaushika
+ms.date: 12/06/2018
+---
+
+# Troubleshoot port exhaustion issues
+
+TCP and UDP protocols work based on port numbers used for establishing connection. Any application or a service that needs to establish a TCP/UDP connection will require a port on its side. 
+ 
+There are two types of ports:
+
+- *Ephemeral ports*, which are usually dynamic ports, are the set of ports that every machine by default will have them to make an outbound connection.
+- *Well-known ports* are the defined port for a particular application or service. For example, file server service is on port 445, HTTPS is 443, HTTP is 80, and RPC is 135. Custom application will also have their defined port numbers.
+ 
+Clients when connecting to an application or service will make use of an ephemeral port from its machine to connect to a well-known port defined for that application or service. A browser on a client machine will use an ephemeral port to connect to https://www.microsoft.com on port 443.
+ 
+In a scenario where the same browser is creating a lot of connections to multiple website, for any new connection that the browser is attempting, an ephemeral port is used. After some time, you will notice that the connections will start to fail and one high possibility for this would be because the browser has used all the available ports to make connections outside and any new attempt to establish a connection will fail as there are no more ports available. When all the ports are on a machine are used, we term it as *port exhaustion*. 
+ 
+## Default dynamic port range for TCP/IP
+ 
+To comply with [Internet Assigned Numbers Authority (IANA)](http://www.iana.org/assignments/port-numbers) recommendations, Microsoft has increased the dynamic client port range for outgoing connections. The new default start port is **49152**, and the new default end port is **65535**. This is a change from the configuration of earlier versions of Windows that used a default port range of **1025** through **5000**. 
+ 
+You can view the dynamic port range on a computer by using the following netsh commands: 
+
+- `netsh int ipv4 show dynamicport tcp` 
+- `netsh int ipv4 show dynamicport udp` 
+- `netsh int ipv6 show dynamicport tcp` 
+- `netsh int ipv6 show dynamicport udp` 
+ 
+
+The range is set separately for each transport (TCP or UDP). The port range is now a range that has a starting point and an ending point. Microsoft customers who deploy servers that are running Windows Server may have problems that affect RPC communication between servers if firewalls are used on the internal network. In these situations, we recommend that you reconfigure the firewalls to allow traffic between servers in the dynamic port range of **49152** through **65535**. This range is in addition to well-known ports that are used by services and applications. Or, the port range that is used by the servers can be modified on each server. You adjust this range by using the netsh command, as follows. The above command sets the dynamic port range for TCP. 
+ 
+```cmd
+netsh int <ipv4|ipv6> set dynamic <tcp|udp> start=number num=range
+```
+ 
+The start port is number, and the total number of ports is range. The following are sample commands:
+ 
+- `netsh int ipv4 set dynamicport tcp start=10000 num=1000`
+- `netsh int ipv4 set dynamicport udp start=10000 num=1000`
+- `netsh int ipv6 set dynamicport tcp start=10000 num=1000`
+- `netsh int ipv6 set dynamicport udp start=10000 num=1000`
+ 
+These sample commands set the dynamic port range to start at port 10000 and to end at port 10999 (1000 ports). The minimum range of ports that can be set is 255. The minimum start port that can be set is 1025. The maximum end port (based on the range being configured) cannot exceed 65535. To duplicate the default behavior of Windows Server 2003, use 1025 as the start port, and then use 3976 as the range for both TCP and UDP. This results in a start port of 1025 and an end port of 5000.
+ 
+Specifically, about outbound connections as incoming connections will not require an Ephemeral port for accepting connections.
+ 
+Since outbound connections start to fail, you will see a lot of the below behaviors:
+ 
+- Unable to login to the machine with domain credentials, however login with local account works. Domain login will require you to contact the DC for authentication which is again an outbound connection. If you have cache credentials set, then domain login might still work.
+
+    ![](images/tcp-ts-14.png)
+
+- Group Policy update failures:
+
+    ![](images/tcp-ts-15.png)
+
+- File shares are inaccessible:
+
+    ![](images/tcp-ts-16.png)
+
+- RDP from the affected server fails:
+
+    ![](images/tcp-ts-17.png)
+
+- Any other application running on the machine will start to give out errors
+
+Reboot of the server will resolve the issue temporarily, but you would see all the symptoms come back after a period of time.
+
+If you suspect that the machine is in a state of port exhaustion: 
+ 
+1.	Try making an outbound connection. From the server/machine, access a remote share or try an RDP to another server or telnet to a server on a port. If the outbound connection fails for all of these, go to the next step.
+
+2.	Open event viewer and under the system logs, look for the events which clearly indicate the current state: 
+
+    a.	**Event ID 4227**
+
+    ![](images/tcp-ts-18.png)
+
+    b.	**Event ID 4231**
+
+    ![](images/tcp-ts-19.png)
+
+3. Collect a `netstat -anob output` from the server. The netstat output will show you a huge number of entries for TIME_WAIT state for a single PID.
+
+    ![](images/tcp-ts-20.png)
+
+After a graceful closure or an abrupt closure of a session, after a period of 4 minutes (default), the port used the process or application would be released back to the available pool. During this 4 minutes, the TCP connection state will be TIME_WAIT state. In a situation where you suspect port exhaustion, an application or process will not be able to release all the ports that it has consumed and will remain in the TIME_WAIT state.
+ 
+You may also see CLOSE_WAIT state connections in the same output, however CLOSE_WAIT state is a state when one side of the TCP peer has no more data to send (FIN sent) but is able to receive data from the other end. This state does not necessarily indicate port exhaustion.
+ 
+>[!Note]
+>Having huge connections in TIME_WAIT state does not always indicate that the server is currently out of ports unless the first two points are verified. Having lot of TIME_WAIT connections does indicate that the process is creating lot of TCP connections and may eventually lead to port exhaustion.
+>
+>Netstat has been updated in Windows 10 with the addition of the **-Q** switch to show ports that have transitioned out of time wait as in the BOUND state.  An update for Windows 8.1 and Windows Server 2012R2 has been released that contains this functionality. The PowerShell cmdlet `Get-NetTCPConnection` in Windows 10 also shows these BOUND ports.
+ 
+4.	Open a command prompt in admin mode and run the below command
+
+    ```cmd
+    Netsh trace start scenario=netconnection capture=yes tracefile=c:\Server.etl
+    ```
+
+5.	Open the server.etl file with [Network Monitor](troubleshoot-tcpip-netmon.md) and in the filter section, apply the filter **Wscore_MicrosoftWindowsWinsockAFD.AFD_EVENT_BIND.Status.LENTStatus.Code == 0x209**. You should see entries which say **STATUS_TOO_MANY_ADDRESSES**. If you do not find any entries, then the server is still not out of ports. If you find them, then you can confirm that the server is under port exhaustion.
+ 
+## Troubleshoot Port exhaustion
+ 
+The key is to identify which process or application is using all the ports. Below are some of the tools that you can use to isolate to one single process
+ 
+### Method 1
+
+Start by looking at the netstat output. If you are using Windows 10 or Windows Server 2016, then you can run the command `netstat -anobq` and check for the process ID which has maximum entries as BOUND. Alternately, you can also run the below Powershell command to identify the process:
+
+```Powershell
+Get-NetTCPConnection | Group-Object -Property State, OwningProcess | Select -Property Count, Name, @{Name="ProcessName";Expression={(Get-Process -PID ($_.Name.Split(',')[-1].Trim(' '))).Name}}, Group | Sort Count -Descending 
+```
+
+Most port leaks are caused by user-mode processes not correctly closing the ports when an error was encountered. At the user-mode level ports (actually sockets) are handles. Both **TaskManager** and **ProcessExplorer** are able to display handle counts which allows you to identify which process is consuming all of the ports.
+ 
+For Windows 7 and Windows Server 2008 R2, you can update your Powershell version to include the above cmdlet. 
+ 
+### Method 2
+
+If method 1 does not help you identify the process (prior to Windows 10 and Windows Server 2012 R2), then have a look at Task Manager: 
+
+1.	Add a column called “handles” under details/processes.
+2.	Sort the column handles to identify the process with the highest number of handles. Usually the process with handles greater than 3000 could be the culprit except for processes like System, lsass.exe, store.exe, sqlsvr.exe.
+
+    ![](images/tcp-ts-21.png)
+
+3.	If any other process than these has a higher number, stop that process and then try to login using domain credentials and see if it succeeds.
+ 
+### Method 3
+
+If Task Manager did not help you identify the process, then use Process Explorer to investigate the issue.
+ 
+Steps to use Process explorer: 
+
+1.	[Download Process Explorer](https://docs.microsoft.com/sysinternals/downloads/process-explorer) and run it **Elevated**. 
+2.	Alt + click the column header, select **Choose Columns**, and on the **Process Performance** tab, add **Handle Count**.
+3.	Select **View \ Show Lower Pane**.
+4.	Select **View \ Lower Pane View \ Handles**.
+5.	Click the **Handles** column to sort by that value.
+6.	Examine the processes with higher handle counts than the rest (will likely be over 10,000 if you can't make outbound connections).
+7.	Click to highlight one of the processes with a high handle count.
+8.	In the lower pane, the handles listed as below are sockets. (Sockets are technically file handles).
+    
+    File   \Device\AFD
+
+    ![](images/tcp-ts-22.png)
+
+10. Some are normal, but large numbers of them are not (hundreds to thousands). Close the process in question. If that restores outbound connectivity, then you have further proven that the app is the cause. Contact the vendor of that app.
+ 
+Finally, if the above methods did not help you isolate the process, we suggest you collect a complete memory dump of the machine in the issue state. The dump will tell you which process has the maximum handles.
+ 
+As a workaround, rebooting the computer will get the it back in normal state and would help you resolve the issue for the time being. However, when a reboot is impractical, you can also consider increasing the number of ports on the machine using the below commands:
+
+```cmd
+netsh int ipv4 set dynamicport tcp start=10000 num=1000
+```
+
+This will set the dynamic port range to start at port 10000 and to end at port 10999 (1000 ports). The minimum range of ports that can be set is 255. The minimum start port that can be set is 1025. The maximum end port (based on the range being configured) cannot exceed 65535.
+ 
+>[!NOTE]
+>Note that increasing the dynamic port range is not a permanent solution but only temporary. You will need to track down which process/processors are consuming max number of ports and troubleshoot from that process standpoint as to why its consuming such high number of ports. 
+
+For Windows 7 and Windows Server 2008 R2, you can use the below script to collect the netstat output at defined frequency. From the outputs, you can see the port usage trend.
+
+```
+@ECHO ON
+set v=%1
+:loop
+set /a v+=1
+ECHO %date% %time% >> netstat.txt
+netstat -ano >> netstat.txt
+ 
+PING 1.1.1.1 -n 1 -w 60000 >NUL
+ 
+goto loop
+```
+
+
+
+ 
+## Useful links
+
+- [Port Exhaustion and You!](https://blogs.technet.microsoft.com/askds/2008/10/29/port-exhaustion-and-you-or-why-the-netstat-tool-is-your-friend/) - this article gives a detail on netstat states and how you can use netstat output to determine the port status
+
+- [Detecting ephemeral port exhaustion](https://blogs.technet.microsoft.com/clinth/2013/08/09/detecting-ephemeral-port-exhaustion/): this article has a script which will run in a loop to report the port status. (Applicable for Windows 2012 R2, Windows 8, Windows 10)
+

windows/client-management/troubleshoot-tcpip-rpc-errors.md

@@ -0,0 +1,187 @@
+---
+title: Troubleshoot Remote Procedure Call (RPC) errors
+description: Learn how to troubleshoot Remote Procedure Call (RPC) errors
+ms.prod: w10
+ms.sitesec: library
+ms.topic: troubleshooting
+author: kaushika-msft
+ms.localizationpriority: medium
+ms.author: kaushika
+ms.date: 12/06/2018
+---
+
+# Troubleshoot Remote Procedure Call (RPC) errors
+
+You might encounter an **RPC server unavailable** error when connecting to Windows Management Instrumentation (WMI), SQL Server, during a remote connection, or for some Microsoft Management Console (MMC) snap-ins. The following image is an example of an RPC error.
+
+![The following error has occurred: the RPC server is unavailable](images/rpc-error.png)
+
+This is a commonly encountered error message in the networking world and one can lose hope very fast without trying to understand much, as to what is happening ‘under the hood’. 
+
+Before getting in to troubleshooting the **RPC server unavailable*- error, let’s first understand basics about the error. There are a few important terms to understand:
+
+- Endpoint mapper – a service listening on the server, which guides client apps to server apps by port and UUID.
+- Tower – describes the RPC protocol, to allow the client and server to negotiate a connection.
+- Floor – the contents of a tower with specific data like ports, IP addresses, and identifiers.
+- UUID – a well-known GUID that identifies the RPC application. The UUID is what you use to see a specific kind of RPC application conversation, as there are likely to be many.
+- Opnum – the identifier of a function that the client wants the server to execute. It’s just a hexadecimal number, but a good network analyzer will translate the function for you. If neither knows, your application vendor must tell you.
+- Port – the communication endpoints for the client and server applications.
+- Stub data – the information given to functions and data exchanged between the client and server. This is the payload, the important part.
+ 
+>[!Note]
+> A lot of the above information is used in troubleshooting, the most important is the Dynamic RPC port number you get while talking to EPM.
+ 
+## How the connection works
+
+Client A wants to execute some functions or wants to make use of a service running on the remote server, will first establish the connection with the Remote Server by doing a three-way handshake.  
+
+![Diagram illustrating connection to remote server](images/rpc-flow.png)
+
+RPC ports can be given from a specific range as well.
+### Configure RPC dynamic port allocation
+ 
+Remote Procedure Call (RPC) dynamic port allocation is used by server applications and remote administration applications such as Dynamic Host Configuration Protocol (DHCP) Manager, Windows Internet Name Service (WINS) Manager, and so on. RPC dynamic port allocation will instruct the RPC program to use a particular random port in the range configured for TCP and UDP, based on the implementation of the operating system used. 
+ 
+Customers using firewalls may want to control which ports RPC is using so that their firewall router can be configured to forward only these Transmission Control Protocol (UDP and TCP) ports. Many RPC servers in Windows let you specify the server port in custom configuration items such as registry entries. When you can specify a dedicated server port, you know what traffic flows between the hosts across the firewall, and you can define what traffic is allowed in a more directed manner.
+ 
+As a server port, please choose a port outside of the range you may want to specify below. You can find a comprehensive list of server ports that are used in Windows and major Microsoft products in the article [Service overview and network port requirements for Windows](https://support.microsoft.com/help/832017).
+The article also lists the RPC servers and which RPC servers can be configured to use custom server ports beyond the facilities the RPC runtime offers. 
+ 
+Some firewalls also allow for UUID filtering where it learns from a RPC Endpoint Mapper request for a RPC interface UUID. The response has the server port number, and a subsequent RPC Bind on this port is then allowed to pass.
+ 
+With Registry Editor, you can modify the following parameters for RPC. The RPC Port key values discussed below are all located in the following key in the registry:
+ 
+**HKEY_LOCAL_MACHINE\Software\Microsoft\Rpc\Internet\ Entry name Data Type**
+
+**Ports REG_MULTI_SZ**
+ 
+- Specifies a set of IP port ranges consisting of either all the ports available from the Internet or all the ports not available from the Internet. Each string represents a single port or an inclusive set of ports. For example, a single port may be represented by **5984**, and a set of ports may be represented by **5000-5100**. If any entries are outside the range of 0 to 65535, or if any string cannot be interpreted, the RPC runtime treats the entire configuration as invalid.
+ 
+**PortsInternetAvailable REG_SZ Y or N (not case-sensitive)**
+ 
+- If Y, the ports listed in the Ports key are all the Internet-available ports on that computer. If N, the ports listed in the Ports key are all those ports that are not Internet-available.
+ 
+**UseInternetPorts REG_SZ ) Y or N (not case-sensitive)**
+ 
+- Specifies the system default policy.
+- If Y, the processes using the default will be assigned ports from the set of Internet-available ports, as defined previously.
+- If N, the processes using the default will be assigned ports from the set of intranet-only ports.
+ 
+**Example:**
+
+In this example ports 5000 through 6000 inclusive have been arbitrarily selected to help illustrate how the new registry key can be configured. This is not a recommendation of a minimum number of ports needed for any particular system.
+
+1.	Add the Internet key under: HKEY_LOCAL_MACHINE\Software\Microsoft\Rpc
+
+2.	Under the Internet key, add the values "Ports" (MULTI_SZ), "PortsInternetAvailable" (REG_SZ), and "UseInternetPorts" (REG_SZ).
+
+    For example, the new registry key appears as follows:
+    Ports: REG_MULTI_SZ: 5000-6000
+    PortsInternetAvailable: REG_SZ: Y
+    UseInternetPorts: REG_SZ: Y
+
+3.	Restart the server. All applications that use RPC dynamic port allocation use ports 5000 through 6000, inclusive. 
+
+You should open up a range of ports above port 5000. Port numbers below 5000 may already be in use by other applications and could cause conflicts with your DCOM application(s). Furthermore, previous experience shows that a minimum of 100 ports should be opened, because several system services rely on these RPC ports to communicate with each other.
+ 
+>[!Note]
+>The minimum number of ports required may differ from computer to computer. Computers with higher traffic may run into a port exhaustion situation if the RPC dynamic ports are restricted. Take this into consideration when restricting the port range.
+ 
+>[!WARNING]
+>If there is an error in the port configuration or there are insufficient ports in the pool, the Endpoint Mapper Service will not be able to register RPC servers with dynamic endpoints. When there is a configuration error, the error code will be 87 (0x57) ERROR_INVALID_PARAMETER. This can affect Windows RPC servers as well, such as Netlogon. It will log event 5820 in this case:
+> 
+>Log Name: System
+>Source: NETLOGON
+>Event ID: 5820
+>Level: Error
+>Keywords: Classic
+>Description:
+>The Netlogon service could not add the AuthZ RPC interface. The service was terminated. The following error occurred: 'The parameter is incorrect.'
+ 
+If you would like to do a deep dive as to how it works, see [RPC over IT/Pro](https://blogs.technet.microsoft.com/askds/2012/01/24/rpc-over-itpro/).
+ 
+ 
+## Troubleshooting RPC error
+ 
+### PortQuery
+
+The best thing to always troubleshoot RPC issues before even getting in to traces is by making use of tools like **PortQry**. You can quickly determine if you are able to make a connection by running the command:
+
+```cmd
+Portqry.exe -n <ServerIP> -e 135
+``` 
+ 
+This would give you a lot of output to look for, but you should be looking for **ip_tcp*- and the port number in the brackets, which tells whether you were successfully able to get a dynamic port from EPM and also make a connection to it. If the above fails, you can typically start collecting simultaneous network traces. Something like this from the output of “PortQry”:
+
+```cmd 
+Portqry.exe -n 169.254.0.2 -e 135
+```
+Partial output below:
+ 
+>Querying target system called:
+>169.254.0.2
+>Attempting to resolve IP address to a name...
+>IP address resolved to RPCServer.contoso.com
+>querying...
+>TCP port 135 (epmap service): LISTENING
+>Using ephemeral source port
+>Querying Endpoint Mapper Database...
+>Server's response:
+>UUID: d95afe70-a6d5-4259-822e-2c84da1ddb0d
+>ncacn_ip_tcp:169.254.0.10**[49664]**
+
+ 
+The one in bold is the ephemeral port number that you made a connection to successfully. 
+ 
+### Netsh
+
+You can run the commands below to leverage Windows inbuilt netsh captures, to collect a simultaneous trace. Remember to execute the below on an “Admin CMD”, it requires elevation.
+ 
+- On the client
+```cmd
+Netsh trace start scenario=netconnection capture=yes tracefile=c:\client_nettrace.etl maxsize=512 overwrite=yes report=yes
+```
+ 
+- On the Server
+```cmd
+Netsh trace start scenario=netconnection capture=yes tracefile=c:\server_nettrace.etl maxsize=512 overwrite=yes report=yes
+```
+
+Now try to reproduce your issue from the client machine and as soon as you feel the issue has been reproduced, go ahead and stop the traces using the command
+```cmd
+Netsh trace stop
+```
+
+Open the traces in [Microsoft Network Monitor 3.4](troubleshoot-tcpip-netmon.md) or Message Analyzer and filter the trace for
+
+- Ipv4.address==<client-ip> and ipv4.address==<server-ip> and tcp.port==135 or just tcp.port==135 should help.
+
+- Look for the “EPM” Protocol Under the “Protocol” column.
+
+- Now check if you are getting a response from the server or not, if you get a response note the Dynamic Port number that you have been allocated to use.
+
+    ![](images/tcp-ts-23.png)
+
+- Check if we are connecting successfully to this Dynamic port successfully.
+
+- The filter should be something like this: tcp.port==<dynamic-port-allocated> and ipv4.address==<server-ip> 
+
+    ![](images/tcp-ts-24.png)
+
+This should help you verify the connectivity and isolate if any network issues are seen.
+
+
+### Port not reachable
+
+The most common reason why we would see the RPC server unavailable is when the dynamic port that the client tries to connect is not reachable. The client side trace would then show TCP SYN retransmits for the dynamic port.
+ 
+![](images/tcp-ts-25.png)
+ 
+The port cannot be reachable due to one of the following reasons:
+
+- The dynamic port range is blocked on the firewall in the environment. 
+- A middle device is dropping the packets. 
+- The destination server is dropping the packets (WFP drop / NIC drop/ Filter driver etc)
+
+
+

windows/client-management/troubleshoot-tcpip.md

@@ -0,0 +1,20 @@
+---
+title: Advanced troubleshooting for TCP/IP issues
+description: Learn how to troubleshoot TCP/IP issues.
+ms.prod: w10
+ms.sitesec: library
+ms.topic: troubleshooting
+author: kaushika-msft
+ms.localizationpriority: medium
+ms.author: kaushika
+ms.date: 12/06/2018
+---
+
+# Advanced troubleshooting for TCP/IP issues
+
+In these topics, you will learn how to troubleshoot common problems in a TCP/IP network environment.
+
+- [Collect data using Network Monitor](troubleshoot-tcpip-netmon.md)
+- [Troubleshoot TCP/IP connectivity](troubleshoot-tcpip-connectivity.md)
+- [Troubleshoot port exhaustion issues](troubleshoot-tcpip-port-exhaust.md)
+- [Troubleshoot Remote Procedure Call (RPC) errors](troubleshoot-tcpip-rpc-errors.md)