Hi, folks! I’ve learned a lot these past 8-9 months in self-hosting, but I’m not sure what’s going on with my reverse proxy setup. At this point, I’m only running Nginx Proxy Manager, Jellyfin, and Komga, separated from my home network via VLANs and firewall rules. My home network is 192.168.1.X and hosted services are on 192.168.10.X. Yet when I look at access logs for Jellyfin or Komga, both are reporting access from 192.168.65.1 when accessed from outside my home network. I don’t have a 65 subnet, so this is strange. The reverse proxy is run on a mini PC at .10.201, and Jellyfin and Komga are hosted on a NAS at .10.202; all of them run via Docker containers.

The instructions for Jellyfin for reverse proxy just say to list the address for the reverse proxy, which I did using IPv4 and the domain, and various configurations of only one or the other, as well as the 192.168.65.1 address that manifested out of nowhere. I haven’t observed any kind of change in IP address reporting after making changes here. I haven’t begun to try to configure Komga for the reverse proxy, because I figure until I get it working for Jellyfin and understand how to do it, it probably doesn’t matter; I only listed it here as evidence that two different services were reporting the same phantom IP address. I did ask for help on this issue on a Discord server a little while ago, and some of those folks suggested looking at the Docker networks. I left network settings on basically default for all three of these applications, and they all seem to generate something like a 172.17.0.X address, not the 192.168.65.1 that I’m seeing in logs.

I know there are supposed to be extra headers on network packets that these services can be aware of if they know it’s going through a reverse proxy first, and I figure this is important for proper monitoring of folks accessing my hosted services, so i want to get this figured out before I start running a few more apps. Any help anyone can offer would be appreciated. A lot of times, the documentation for this stuff is written up with an assumption of more understanding than the person reading it might have. Or maybe I just glanced right by something obvious and stupid.

  • Ooops@feddit.org
    link
    fedilink
    English
    arrow-up
    3
    ·
    edit-2
    2 hours ago

    the 192.168.65.1 address that manifested out of nowhere

    Or it’s indeed the internal virtual lan your docker config uses…

    Is that an actual problem or are you really just trying to fix the logs? That should be doable via headers so the real origin is preserved independent of proxying.

    • ampersandrew@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      edit-2
      2 hours ago

      Yeah, that appears to be the case from a link someone else linked here, but I was unable to trace that from anywhere I could find in my containers before.

      I am trying to fix the logs so I can see the origin of connections to my services, but I’ve been unsuccessful in getting additional configurations to resolve that.

      • Ooops@feddit.org
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        2 hours ago

        Well… it’s the default iirc (192.168.65.0/24). So that’s what docker uses if there is no explicit configuration you could find.

        • ampersandrew@lemmy.worldOP
          link
          fedilink
          English
          arrow-up
          1
          ·
          2 hours ago

          It appears to be, but I didn’t know that until today. And beyond that, I’ve been trying every permutation of both that IP and the IP of the server on my network in my Jellyfin proxy settings, and it still hasn’t helped to surface the origin IP address.

  • treadful@lemmy.zip
    link
    fedilink
    English
    arrow-up
    6
    ·
    5 hours ago

    You’re gonna want to configure nginx to send along an X-Forwarded-For HTTP header (there’s also Forwarded, but it’s not as well supported yet). An example in a location block:

    location / {
        proxy_pass http://backend/;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
    

    Looks like Jellyfin supports it.

    • ampersandrew@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      4 hours ago

      I’m not quite sure where this goes. It’s Nginx Proxy Manager (in case that affects your answer compared to vanilla nginx), and I’m using Docker Desktop for much the same reason: I’m doing as much configuration via GUI as possible. Is this a Docker configuration or an Nginx Proxy Manager configuration? There is a “Custom Nginx Configuration” screen for my Proxy Host; does it go there? In a nested location block?

      • treadful@lemmy.zip
        link
        fedilink
        English
        arrow-up
        2
        ·
        4 hours ago

        I really don’t know shit about the GUI tools, but presumably they overlay on the same configuration.

        The config I posted would be part of the nginx configuration. You’re instructing it to append the client IP to the X-Forwarded-For header.

        There is a “Custom Nginx Configuration” screen for my Proxy Host; does it go there? In a nested location block?

        I doubt it, but I’m not sure. The location block would normally be where your reverse proxy configuration would already be, and you’d need to just add that one line. But again, I don’t know how that translates to these GUI tools. Maybe if you search for proxy_add_x_forwarded_for and “Nginx Proxy Manager” you might find what you need.

        • ampersandrew@lemmy.worldOP
          link
          fedilink
          English
          arrow-up
          1
          ·
          edit-2
          3 hours ago

          I know you said you don’t know anything about the GUI equivalent, so I’m mostly responding to this comment to publicly post how far I’ve gotten off of your tips. There is a “Custom Locations” tab for my proxy host in NPM, but even after attempting to follow this guide I found, I can’t get my custom locations to persist after hitting the Save button.

          EDIT: I was leaving out the path on the location, so now that that’s corrected, it saves my changes, but I’m not seeing any progress in the IP address reporting.

  • airgapped@piefed.social
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    3 hours ago

    In addition to configuring x-forwaded-for in NPM as suggested by others, you will also need to add 192.168.65.0/24 under Dashboard > Advanced > Networking > Known Proxies in Jellyfin.

  • Pomal@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    1
    arrow-down
    1
    ·
    edit-2
    3 hours ago

    Draw a network map!!! It’s what I’ve done on every security engagement, ever. Mark out all your IP addresses, draw lines to denote the communicating RHP (random high port) against the well-known, e.g. 443/tcp HTTPs, for networks of hosts to servers. Then draw boxes around those IP spaces that are colocated on the same box. Then draw red lines perpendicular to every connection from a client to server that isn’t bounded on a single host.

    The red lines denote non-localhost traffic - places to be observed, and for the most part the only IP:port assignments you should be relying on for routing.

    My rec, draw the map and if that doesn’t help get back to us w it - or feel free to dm me.

    Edit: I’ll find and post an example as a response this thread later.

    • ampersandrew@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      1
      arrow-down
      1
      ·
      2 hours ago

      I’ve got a document labeling everything, but I’ve kept my circle small and only expanded out as I conquered the last challenge. The reason I haven’t installed a third service yet is because I don’t have the IPs properly reported yet. I know exactly which machine, IP address, and port each service can be found at.

        • ampersandrew@lemmy.worldOP
          link
          fedilink
          English
          arrow-up
          1
          arrow-down
          3
          ·
          2 hours ago

          I honestly don’t think I have to. I’ve only got two machines, and I don’t think it’s going to expand any farther than that. And I’m not sure how that helps me with this problem.

            • ampersandrew@lemmy.worldOP
              link
              fedilink
              English
              arrow-up
              1
              ·
              edit-2
              1 hour ago

              I’m not trying to be rude. I genuinely don’t see the path between that map and the solution to this problem. Do you need it for your benefit to assist me?

              EDIT: This is the best I can do against your instructions with no example. There’s a fair bit of your instructions I don’t understand. 443 is forwarded from the router to the mini PC in this diagram. I don’t know how to draw red lines perpendicular from the client PC to the mini PC, but that’s where it’s going from and to. I’m more concerned with traffic coming from the internet and making its way to Jellyfin.

  • Matt The Horwood@lemmy.horwood.cloud
    link
    fedilink
    English
    arrow-up
    1
    ·
    5 hours ago

    What your looking for is x-forwaded-for, that http tells an upstream server what the client IP is.

    My guess is your docker network might be 192.168.68, there for the client IP you see is in that network.

  • lemmyvore@feddit.nl
    link
    fedilink
    English
    arrow-up
    1
    ·
    5 hours ago

    Just to understand better:

    • When you say “when accessed from outside my home network”, what kind of access is that? Port forward? VPN?
    • Is 192.168.1.1 your router?
    • What’s your LAN netmask?
    • NPM listens on 192.168.10.201 ports and Jellyfin/Komga on 192.168.10.202 ports?
    • A connection from the outside comes through the router, gets forwarded to 192.168.10.201:443, the proxy sends it to 192.168.10.202:8096, and it shows up in the Jellyfin logs as coming from 192.168.65.1?

    If all of the above is correct I would look at the network interfaces that are up on the mini-PC and the NAS, as well as inside the containers of the proxy and of Jellyfin, and see if there isn’t a 192.168.65.1 up on there somewhere for whatever reason.

    You can also try making a manual request (wget, curl, nc) from the proxy container to Jellyfin and see what it shows up as.

    May also want to see what’s the DNS situation in your LAN and on those machines. You mentioned “names” being used, how are those defined and resolved?

    Last but not least are the containers you mentioned using docker bridge networks or ipvlan?

    • ampersandrew@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      4 hours ago

      When you say “when accessed from outside my home network”, what kind of access is that? Port forward? VPN?

      Accessing via 443 forwarding to my reverse proxy from the internet. My dad will hit my Jellyfin server from his house, and I can hit that server from mobile networks when I’m not on wi-fi.

      Is 192.168.1.1 your router?

      Yes. It’s also .10.1 on the 10 VLAN.

      What’s your LAN netmask?

      255.255.255.0 on both VLANs

      NPM listens on 192.168.10.201 ports and Jellyfin/Komga on 192.168.10.202 ports?

      Correct.

      A connection from the outside comes through the router, gets forwarded to 192.168.10.201:443, the proxy sends it to 192.168.10.202:8096, and it shows up in the Jellyfin logs as coming from 192.168.65.1?

      Correct. Jellyfin is actually listening on a different, auto generated, arbitrary port that I just stuck with, but other than that, you got it.

      I did check every IP address I could find anywhere on my network and couldn’t find the 192.168.65.X anywhere, but @anamethatisnt@sopuli.xyz here in this thread seems to have identified this as some NAT setting built in to Docker. Running on default settings for network, because I had no idea why I would want to modify any of those settings, they are running as bridge networks.