Yes, I see your point (no pun intended), but I would still maintain that this issue didn't necessarily bother mathematicians in the days before formalized set theory. The issue you mention is a reason that any definition of volume should not be based only on set cardinality, since there are those one-to-one correspondences with proper subsets (for infinite sets of points), yet volume should not be preserved by cutting something in half, or doubling something, in an ordinary way.
That is, if X is the unit line segment and Y is a pair of unit line segments joined end-to-end, we want the measure function µ to obey µ(Y) = 2µ(X), even though |X|=|Y| in set theory. And even though that's weird, formalized mathematics didn't get existentially ambitious enough to make anything truly bizarre out of it, I suppose, from Euclid all the way up until Vitali!
Yeah for me it simply doesn't. It isn't false, it just doesn't even make sense.
Define X and Y as you did but make one of the segments joined to make Y /be/ X.
Now if it is ever meaningful to have a one to one correspondence when infinity is lurking about surely the points on X correspond exactly, one to one, each with itself. Once all the points of X are accounted for, clearly half of Y remains, there is no other way. There is simply no point you can select in X that is free to correspond to the second half of Y if you select corresponding to itself in preference. You can never, ever select a point corresponding to the second half of Y if you admit preference to corresponding to itself in selection.
But X has an inexhaustible number of points to chose from by definition and so does Y. So if you start picking points at random from Y and matching them up to a previously unused point from X you can do that forever and never exhaust either of the sets of points. Thus X and Y correspond one to one and no points remain of either X or Y. And there are the same number of integers as there are integers that are even.
1=2 QED The walls are different measured heights but that doesn't matter because we've proved it. Yeah ok, but maybe it does to the poor sap who needs a useful roof?
In that case, I would say you're a finitist (which, as the name seems to suggest, is fine).
The set theory approach gives seemingly clear answers to all of these questions by talking about domains and ranges of functions, rather than talking about what is or isn't an "inexhaustible number". There are potentially different correspondences available, represented by different functions; some possible correspondences may follow the "if you select corresponding to itself in preference" pattern, while others don't. = and ≤ for cardinalities are defined using existence of certain functions between sets.
However, you don't have to believe that any infinite sets exist or that we should be allowed to quantify over them, or that we should attempt to define cardinality for infinite sets at all. Still, as Dana Scott said in a related context, "if you want more, you have to assume more".
That is, if X is the unit line segment and Y is a pair of unit line segments joined end-to-end, we want the measure function µ to obey µ(Y) = 2µ(X), even though |X|=|Y| in set theory. And even though that's weird, formalized mathematics didn't get existentially ambitious enough to make anything truly bizarre out of it, I suppose, from Euclid all the way up until Vitali!