Hey all. I’m hosting a Docmost server for myself and some friends. Now, before everyone shouts “VPN!” at me, I specifically want help with this problem. Think of it as a learning experience.
The problem I have is that the Docmost server is accessible over internet and everyone can log on and use it, it’s working fine. But when I try to access over LAN, it won’t let me log in and I am 99% sure it’s related to SSL certs over LAN from what I’ve read.
Here’s the point I’ve gotten to with my own reading on this and I’m just stumped now:
I’ve got an UNRAID server hosted at 192.186.1.80 - on this server, there’s a number of services running in docker containers. One of these services is Nginx Proxy Manager and it handles all my reverse proxying. This is all working correctly.
I could not for the life of me get Docmost working as a docker container on UNRAID, so instead I spun up a VM and installed it on there. That’s hosted at 192.168.1.85 and NPM points to it when you try to access from docmost.example.com - that’s all dandy.
Then, I installed Adguard Home in a docker container on my UNRAID server. I pointed my router at Adguard as a DNS server, and it seems to me that it’s working fine. Internet’s not broken and Adguard Home is reporting queries and blocks and all that good stuff. So that’s all still working as it should, as far as I’m aware.
So, in Adguard Home I make a DNS Rewrite entry. I tell it to point docmost.example.com to 192.168.1.80, where NPM should be listening for traffic and reverse proxy me to the Docmost server… at least I thought that’s what should happen, but actually nothing happens. I get a connection timed out error.
I’m still pretty new to a lot of this stuff and have tried to figure out a lot of things on my own, but at this point I feel stuck. Does anyone have advice or tips on how I can get this domain to resolve locally with certs?
I can provide more info if needed.
Cheers all!
Assuming NGINX is terminating SSL, I think the problem is ports.
Could you expound? My understanding of the goal here is that Adguard DNS catches my request for
docmost.example.comand redirects it to my UNRAID server, which has Nginx listening for traffic. Nginx then directs to the appropriate IP and port.Yes, just thought if you could check that the correct ports are opened. I.e. is port 443 open for NGINX on Unraid? Is NGINX forwarding traffic to the correct port to your backend? Is the backend configured to allow traffic on a certain domain/all domains if it is handling HTTPS?
Nginx forwards all traffic correctly outside of the local network, so accessing
docmost.example.comfrom outside local network works completely as expected, with certs and all.
I tell it to point docmost.example.com to 192.168.1.80, […] but actually nothing happens. I get a connection timed out error.
I suggest to collect more info about this “nothing”.
ping from your PC on the inside to the name docmost.example.com
ping from your PC on the inside to the number 192.168.1.80
traceroute from your PC on the inside to the name docmost.example.com
traceroute from your PC on the inside to the number 192.168.1.80
Then do the same in the reverse direction: from that docker container to your PC.
Maybe traceroute shows you some stations on the route. Then do the same from this station.
Write down the results thoroughly.
Thanks for the suggestions.
Ping from PC to
docmost.example.com: Pings fine, packet loss.Ping from PC to
192.168.1.80: Pings fine, no packet loss.Traceroute from PC to
docmost.example.com: 1 hop, all <1 ms, to192.168.1.80Traceroute from PC to
192.168.1.80: 1 hop, all <1 msPing from Nginx container to PC: Pings fine, no packet loss.
Traceroute from Nginx container to PC: Hops to
172.18.0.1in <1ms, and then it times out on subsequent hops.I decided to try to traceroute from my PC to
172.18.0.1and172.18.0.9which is the actual IP of the Nginx container according to UNRAID, and in both cases they hop to192.168.1.254which is my router, and then all subsequent hops time out.Do you know why pings would go through without any loss but traceroutes would fail? Any idea what’s going on here?
Never point your DNS at two different IP addresses like this. It will only cause you pain and unexpected behaviour.
What you are experiencing is solved by so-called “NAT reflection” or “NAT loopback”. It’s a setting that - in the optimal case - you should just be able to activate on the appropriate interface on your gateway.
If you do not have that setting or do not have access to the edge router, but only some intermediate router, you can do a nasty hack. You can point static routes to your public IP address to point at your local IP address instead. In that case, you also need to tell your server to accept packets with your public IP address as the destination.
I don’t think OP made two A records here. He simply configured the reverse proxy to point to the VM and the A record to point to the reverse proxy. In my mind, if NGINX is terminating SSL then the only problem could be ports.
Not two A records. From what I understand, OP has an A record pointing to their public IP address (which Nginx is listening on behind a NAT). Then, on the local network, OP uses their own DNS server to ignore that entry and instead always serve the local IP when a host on the LAN queries it.
Aside from OP’s devices potentially using a different DNS server (I was only able to solve it for my stock Android by dropping outgoing DNS in my firewall), this solution is a nightmare for roaming devices like mobile phones. Such a device might cache the DNS answer while on LAN or WAN respectively and then try to continue using that address when the device moves to the other network segment.
These are the most likely scenarios in my opinion - OP’s devices are ignoring the hacky DNS rewrite (either due to using a different DNS server or due to caching) and try to access the server via the public IP. This is supported by the connection timeout, which is exactly what you would see when your gateway doesn’t do loopback.
Couldn’t I troubleshoot this by using a different browser, or even incognito mode? Because when I do that, it still times out. I appreciate the explanation and advice. I’m not too worried about it at this stage only because my service I am trying to get working, Docmost, will really only be accessed from my desktop. Plus, as I said in OP, I am enjoying learning about this stuff and want to figure out why this specifically isn’t working for educational purposes, even if I switch to a different solution.
I didn’t think of that. Indeed, DNS caching/using different DNS servers for different devices will break it exactly like what OP is experiencing. Thanks.
I’ve got a similar problem at home, and I use a really straightforward solution: since the problem manifests on my pcs, I just add my services to the hosts file. It’s particularly good when I’m working on my next cloud, because sometimes you end up with a lot of data moving back and forth, and you don’t really want to hit your router to hit the outside world to hit your internal server when you don’t need it. I just sent it to resolve my main services to the internal IP address, and the best part is that even when the internet is down and DNS isn’t available, my services keep on humming away. I might not even realize.
Not all ISPs will support nat loopback (at least by default) so you could try calling them
For me, I turned their modem/router into bridge mode (just turned off wifi, it’s Rogers and they don’t have a bridge mode button on the Ignite routers) and use my own TP link router as a gateway and do my port forwarding to NPM there
I’ll start from the beginning.
You’re hosting a service on your LAN, and using port forwarding to expose this service to the internet on your public ip.
The problem is accessing your public ip from your private network. There are two ways to solve this.
a) NAT Loopback, which is a setting or rule you may be able to enable or create on your router to forward packets destined for your public ip (the ones from your private network) to your private server
b) Split horizon DNS, which is actually what you’re doing. Where you set it up so in one network (in this case the internet) you get one result, and on another network (your LAN) you get a different result.
If I had to guess, what’s happening is your dns isn’t resolving properly, and when your computer is trying to reach out to your public ip. The thing with dns is it’s a bit finicky, there are many different places to set your dns server.
First, you should check if it’s resolving correctly. It’ll show you the ip it resolves to when you ping it. If it resolves to your public ip, make sure your dns settings aren’t being overridden by your operating system, and try clearing the cache
Thanks for the ping suggestion. When I ping
docmost.example.com, looks like Adguard is correctly catching it and routing it to an internal IP192.168.1.80, which is exactly what I’ve told it to do. I tried to pinganothersub.example.comas well, and it was pinging my duckdns address and timing out. So when I ping, it looks like the packets get through but when I try to access it from a browser, it times out?https://puu.sh/Ks252/fa872908d9.png
(Also, I do not think NAT loopback will be possible with my router/ISP from some reading up I just did)
Right. Can you access your npm server via the ip in your browser? Even if it’s not docmost that it returns?
If you can, it’s probably your browser using its own dns so you’ll have to change that to adguard as well.
NAT Loopback can be a bit finicky but once you set it up there’s no tinkering, it’ll just work forever. The only problem (which really doesn’t matter a bit with a document sharing platform) is that packets first have to go through the router. If your server and client are on the same network then they can communicate directly with each other instead.
I also think it may be the browser not using the DNS provided by the router. This is often called Safe Browsing or Secure DNS in browser settings.
I agree with this comment, and would suggest going with the first solution (NAT loopback, aka NAT hairpin) rather than split-horizon DNS. I say this even though I have a strong dislike of NAT (and would prefer to see networks using flat IPv6 addresses, but that’s a different topic).
Specifically, problems arise when using DNS split-horizon where the same hostname might resolve to two different results, depending on which DNS nameserver is used. This is distinct from some corporate-esque DNS nameservers that refuse to answer for external requests but provide an answer to internal queries. Whereas by having no “single source of truth” (SSOT) for what a hostname should resolve to, this will inevitably make future debugging harder. And that’s on top of debugging NAT issues.
Plus, DNS isn’t a security feature unto itself: successful resolution of internal hostnames shouldn’t increase security exposure, since a competent firewall would block access. Some might suggest that DNS queries can reveal internal addresses to an attacker, but that’s the same faulty argument that suggests ICMP pings should be blocked; it shouldn’t.
To be clear, ad-blocking DNS servers don’t suffer from the ails of split-horizon described above, because they’re intentionally declining to give a DNS response for ad-hosting hostnames, rather than giving a different response. But even if they did, one could argue the point of ad-blocking is to block adware, so we don’t really care if SSOT is diminished for those hostnames.
If it was a certificate issue I’d expect youd just get an error from your browser saying the cert is invalid or expired.
If I had to guess though you’re running into a nat reflection issue: https://nordvpn.com/cybersecurity/glossary/nat-loopback/
Read up on that. But you may need to provide different DNS entries if you’re inside or outside your LAN or add a NAT hairpin rule to your router. But this is only applicable if you’re exposing the same service to the WWW.
Some other things to try though:
- Have you tried just pinging the address? Is the DNS resolution returning the address you expect?
- Whats in your nginx logs? Do you see anything when you try and connect?
- Within your nginx container can you ping your service directly? Is something blocking nginx from accessing the site?
It does not look like NAT loopback will be an option for me due to router/ISP.
Thanks for the ping suggestion. Copied from another comment of mine:
When I ping
docmost.example.com, looks like Adguard is correctly catching it and routing it to an internal IP192.168.1.80, which is exactly what I’ve told it to do. I tried to pinganothersub.example.comas well, and it was pinging my duckdns address and timing out. So when I ping, it looks like the packets get through but when I try to access it from a browser, it times out?When I try to connect to
docmost.example.com, I do not see anything come up on myproxy-host-14_access.log,proxy-host-14_error.log, orerror.log. Just nothing at all. When I access from outside the network, entries in the access log show up as expected from the IP I access it from (in this case, my phone off WIFI).I pinged
192.168.1.85from within my nginx container and looks like it communicated just fine. https://puu.sh/Ks29s/367c0b6144.png
