Only one way to find out... float some ice cubes in a glass of water and observe.

(Edit: I'm back to report the results. There was either no change in the water level, or a change below my measurement tolerance ;)

(Edit2: Here is a more serious take of that experiment: https://skepticalscience.com/Sea-level-rise-due-to-floating-...)