100ms is slightly arbitrary, but the real point of saying "100ms" is that it's lower than the multiple seconds typically required for page load, and higher than the 16ms required for smooth animation. An appropriate target for input response time (the "R" in RAIL) is somewhere in between those two.
If 100ms strikes you as too high, then by all means target a lower number like 50ms. But it's still not a disaster if your 95th percentile is 100ms. Also, if your A time is above 16ms or your L time is above 5 seconds, your limited development time might be better spent improving those rather than bringing R down even further.
If 100ms strikes you as too high, then by all means target a lower number like 50ms. But it's still not a disaster if your 95th percentile is 100ms. Also, if your A time is above 16ms or your L time is above 5 seconds, your limited development time might be better spent improving those rather than bringing R down even further.