This was actually something that was tried with music in the early 90s, by Philips and Sony with the CD-i. The musician/producer Todd Rundgren made an album specifically for this format called "No World Order" where the songs were all broken up into "modules", so to speak, and the user could configure them however they'd like.

https://en.wikipedia.org/wiki/CD-i

https://en.wikipedia.org/wiki/No_World_Order