There is a setting in Plex that will allow it to work without authentication for any devices connecting from a defined subnet. So I just set this to my home network's setup and can use it without issue whether I have an Internet connection or not.

I confirmed this when I tested my Plex server on a Raspberry Pi out in a cabin in the middle of nowhere and it streamed fine without an Internet connection.