Troubleshoot Port exhaustion issues in Windows

 

Any network connection made between computers (TCP or UDP protocols), it is done through the ports. Imagine these as entry points or gateways which is used by a service or application. As more client connections are made, the ports shortfall in the count. In this post, we will share how to troubleshoot Port exhaustion issues.

There are two types of ports – Dynamic ports and Defined ports. Dynamic ports allow multiple clients to connect to defined ports. Websites are a good example. They usually have port 80 defined, but using an active port, they can serve multiple clients. Since there is a limit to Dynamic port, the connections will start to fail when all the ports are busy. It is termed as port exhaustion.

Troubleshoot Port exhaustion in Windows

The primary motive of this troubleshooting guide is to identify which process or application is exhausting the port. Once you have determined it, the next step is to fix the app.

Symptoms to identify Port Exhaustion:

1] Unable to sign in to the machine with domain credentials, however, sign-in with local account works. It is possible that an already used account may work, but the new one will fail. It happens because of caching.

2] Group Policy update will start failing. Every time you try making changes, you may receive an error saying “failed because of lack of network connectivity with domain controller” It could be temporary but is a sign.

3] File shares or network drives become inaccessible.

4] Remote desktop to the server where the application is hosted fails.

Other signs include event id 4227, 4231 in Event viewer for TCP with a message that allocation of dynamic port failed. NetStat command shows a massive number of entries for TIME_WAIT state for a particular application, and so on.

Use NetStat for Windows 10 and Windows Server 2016

• Open Command Prompt with elevated privileges.
• Run the command netstat -anobq
• Next, check for the process ID which has maximum entries as BOUND.

If you use PowerShell, you can use the below command to identify the process with maximum Bound.

Get-NetTCPConnection | Group-Object -Property State, OwningProcess | Select -Property Count, Name, @{Name="ProcessName";Expression={(Get-Process -PID ($_.Name.Split(',')[-1].Trim(' '))).Name}}, Group | Sort Count -Descendin

Many a time, the clients do not close the ports correctly. Even not in use, these ports are not free. It is one of the biggest reason for port exhaustion.

If the need is frequent, you can use Netstat command in a loop. The output can become available in a text file which can be used to monitor the trend. Here is what the script looks like:

@ECHO ON
set v=%1
:loop
set /a v+=1
ECHO %date% %time% >> netstat.txt
netstat -ano >> netstat.txt

PING 1.1.1.1 -n 1 -w 60000 >NUL

goto loop

Use Task Manager to find maximum handles

A slightly more natural method to find such applications is using Task Manager. While PowerShell and Command Prompts have their own merits, if you want to see the process quickly, this is a better method.

• Open Task Manager, and switch to the Details tab.
• Right click on any of the columns, and click “Select Columns.”
• Add “Handles” from the available options.
• Click on the handles column header to sort it by maximum number.

Microsoft suggests that if any the connections are failing, then check if the number of handles is higher than 3000. If that’s the case, then the application is the culprit. However, the OS services are an exception to this. For others, stop that process once, and then try to log in using domain credentials and see if it succeeds.

Process Explorer

You can use Process Explorer in case Task manager doesn’t help. It is useful for tracking down DLL-version problems or handle leaks and provide insight into rouge applications. Process Explorer should be downloaded from here and installed. Make sure to run it with elevated privileges.

1. Right-click on the column header, and then select “Choose Columns.”
2. Switch to Performance Tab, and add Handle Count.
3. From the menu, click on View > Show Lower Pane.
4. Again click on the menu, select View > Lower Pane View > Handles.
5. Sort the handles in descending order.
6. It will reveal the process(es) with the highest handle counts
7. Click to highlight one of the processes with the top handle count.
8. The lower panel will reveal type for all the handles. Ports or sockets are usually with “File DeviceAFD” labels.

Close the process with a high number of handles. If the application spawns back, it can be the cause, and you will have to fix the application or ask the developer of the OEM to fix it. If you cannot fix it because the application demands it, you should consider increasing the number of ports the computer can use. The below command (example) can change the range, and raise it.

netsh int ipv4 set dynamicport tcp start=10000 num=1000

The minimum start port that can be set is 1025. The maximum end port cannot exceed 65535.

That said the solution is still temporary. As an IT admin, you will have to find a better solution to troubleshoot port exhaustion. Sometimes, multiple servers can be used to increase ports, but that’s a different league altogether.

Original Article