I disagree. Their point is that the user should always be able to share, modify and run the software they have, not that they should have access to every line of code written. Again, from the GNU project:

    In general we do not believe it is wrong to develop a program and not release
    it. There are occasions when a program is so useful that withholding it from
    release is treating humanity badly. However, most programs are not that important,
    so not releasing them is not particularly harmful. Thus, there is no conflict
    between the development of private or custom software and the principles of the
    free software movement.
    
    Nearly all employment for programmers is in development of custom software;
    therefore most programming jobs are, or could be, done in a way compatible 
    with the free software movement.

I doubt their modifications to the kernel (remember we're not talking about userspace stuff here) are "so useful that not releasing them is treating humanity badly".