An alternative cause of VPN Error 809 in Windows (8.1)

I have not been using the integrated VPN interface in Windows for quite a long time. These days I tried to configure a L2TP VPN connection on Windows 8.1 as I did before, however I kept receiving Error 809 from time to time on my computer. On my phone, the connection goes pretty well, so it must be a configuration problem with my PC system.

The first suspect was network bridging. I use Hyper-V on my computer. As we know, the way Hyper-V handles networking with virtual systems is creating bridging on the host OS. You can google about “windows VPN error 809” and you will find the very famous problem of establishing L2TP/IPsec behind network address translation (NAT), a popular technology for rerouting traffic to an Ethernet, for example that linking your virtual machines. It is a security improvement since Windows XP SP2 to disable IPsec with NAT traversal. [1,2]

After trying to add the registry value as mention in the above knowledge base articles and rebooting, however, I still cannot establish the VPN connection with error 809. Finally I found a possible solution mentioned in a forum post. [3]

If you navigate to this key in regedit

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RasMan\Parameters

and find there is a value called ProhibitIpSec setting to 1, then this value may be preventing your connection. I deleted this entry and restarted the computer, then I can successfully establish the VPN connection! I also tried to delete the entry of the first solution about NAT then, and it turned out to have no effect on my problem.

The value ProhibitIpSec is used to disable the IPsec policy that Windows creates and it will prevent IPsec negotiation on L2TP calls, which will make the connection unsecured. [4,5] It is of course not safe at all to use it as an everyday configuration so I am curious about when and why I made this configuration. It turned out to be a requirement of the very broken VPN service from Zhejiang University [6] that I have been using when I was on campus there. I must say that the implementation of many IT services in ZJU is crappy with no concern on security or compatibility. Hope the service could possibly get better one day…

References

1. How to configure an L2TP/IPsec server behind a NAT-T device in Windows Vista and in Windows Server 2008, Microsoft Support, Article ID: 926179, https://support.microsoft.com/en-us/kb/926179.
2. The default behavior of IPsec NAT traversal (NAT-T) is changed in Windows XP Service Pack 2, Microsoft Support, Article ID: 885407, https://support.microsoft.com/en-us/kb/885407.
3. VPN Error 809, How-To Geek Forum, http://www.howtogeek.com/forum/topic/vpn-error-809, post #2.
4. Disabling IPSEC Policy Used with L2TP, Microsoft Support, Article ID: 258261, https://support.microsoft.com/en-us/kb/258261.
5. ProhibitIpSec, Microsoft TechNet, https://technet.microsoft.com/en-us/library/cc736542(v=ws.10).aspx.
6. Zhejiang University IT Service Guide, http://zuits.zju.edu.cn/attachments/2013-08/01-1376816310-168201.pdf (in Chinese), p 68.