In IT there are certain things that you don’t often “dig into”, either for lack of time or just the fact that something “just works”. Other than setting up necessary routes and gateways, usually through DHCP, you often don’t bother with routing on a user’s workstation, but hopefully a lesson I learned recently will stop someone else from making the same mistake.
I’ve been troubleshooting an odd network issue on a Windows 10 laptop, where the user simply lost connection; no errors, network still shows active link, just *drops*. I hate these issues, as they can really be anywhere, and you really have to wait for it to crop up in order to troubleshoot it while it’s happening.
So after exhausting many avenues, one angle I thought of was “well maybe it’s an IP layer thing and the routing table is getting corrupted somehow”. That would make sense; the laptop still shows a link, but IP traffic is just going nowhere. Thankfully, I grabbed a screen shot of the routing tables, then I opted to use a command of route -f to clear all the routing tables and “reset” things without actually rebooting or disconnecting anything. It still didn’t work, so I simply rebooted to get the user back online due to a time crunch. Emails were coming in so I walked away still perplexed about why the network keeps dropping.
Not much later, the user comes back and says they can’t get online again. Thinking the problem is happening AGAIN, I rush over to try some more troubleshooting, but realize emails are coming in, other internet-based products are fine, yet any web browser I open won’t go anywhere, almost like DNS isn’t resolving. That’s strange! I do see DNS IS resolving, pings are getting through, now what’s going on?!?! I backtracked my steps and realized I must have broke something in using route -f, so I brought up the routing tables and compared to my former screenshot, and definitely see a difference; many are missing, specifically the 127.0.0.1 routes.
Here’s where things get interesting; in most cases a reboot resets your routing tables and life is good. In this case, the reboot did NOT restore all the former routes, and now I’m into after hours and this user NEEDS to be able to use this computer.
Realizing it was only affecting browsers, my thought process went immediately to the Sophos web filtering, and realized the routes were likely related. With Sophos endpoint, there’s a level of web filtering baked into the product, and part of how that filtering works involves the software creating a virtual network within your computer and routing traffic through that first; as such it can block malicious sites and traffic. In doing a route print on a computer with Sophos installed (and perhaps other products), you’ll see a “loopback” configuration using routes like 127.0.0.1.
What I didn’t realize is that these route entries are deleted using route -f and not restored on reboot! So after going through the rigamarole of re-installing Sophos, thankfully that restored the missing routes and brought the user back to a working state; albeit they are STILL having the occasional network issue.
Today I swapped the dock, so we’ll see if that fixes the problem.