You should probably start with the FAQ: https://www.gnu.org/licenses/gpl-faq.en.html#combining

Swapping out one library for another compatible one is a modification of the program.

There is no reason to start with the FAQ. The FAQ is irrelevant. The FAQ is not part of the GPL and has no legal force whatsoever.

What part of the GPL brings swapping dynamically linked libraries within the definition of "modify"?

Because on first read, not only is such an activity not explicitly included, it is explicitly excluded — since you do not need copyright permission for program X to swap out library Y that it relies upon, see 17 U.S.C. § 117(a)(1) (excluding copies made in memory at runtime from the need for copyright permission), it can never be a modification to do so.

I don’t know why you bring up 17 U.S.C. § 117(a)(1); it is completely irrelevant to your question. It is an important thing to understand, but it doesn’t have any effect on the answer to the question you asked. I can’t even fathom why you would think that it does. Maybe it is some kind of motivated reasoning?

The FAQ is, however, quite relevant. It’s certainly not part of the text of the GPL, but your Question is very Frequently Asked, and thus the answer to it is right there (<https://www.gnu.org/licenses/gpl-faq.html#MoneyGuzzlerInc>).

    I'd like to modify GPL-covered programs and link them with the
    portability libraries from Money Guzzler Inc. I cannot distribute
    the source code for these libraries, so any user who wanted to
    change these versions would have to obtain those libraries
    separately. Why doesn't the GPL permit this? (#MoneyGuzzlerInc)
    
        There are two reasons for this. First, a general one. If we
        permitted company A to make a proprietary file, and company B
        to distribute GPL-covered software linked with that file, the
        effect would be to make a hole in the GPL big enough to drive
        a truck through. This would be carte blanche for withholding
        the source code for all sorts of modifications and extensions
        to GPL-covered software.
    
        Giving all users access to the source code is one of our main
        goals, so this consequence is definitely something we want to
        avoid.
    
        More concretely, the versions of the programs linked with the
        Money Guzzler libraries would not really be free software as
        we understand the term—they would not come with full source
        code that enables users to change and recompile the program.

This is a legal question. The statute, which is a legal authority, is absolutely relevant. The FAQ, which is not a legal authority and not part of the license and not even published by any relevant party, is absolutely irrelevant.