It feels to me like you're redefining what a number is to be very different to what anyone with a maths background would say a number is. Essentially you're saying that neither e nor pi are numbers?
The only numbers that may exist in nature are the non-negative rationals. Negative numbers are widely useful, but they don't exist in any meaningful way: they're an abstraction over reality. Irrational numbers definitely don't exist, and don't get me started on so-called "imaginary" numbers -- they're no less present in nature than the negative numbers are.
But the concepts are useful, and the symbols have meaning. The article mentions normal words having different meanings in mathematical language, and that impacts even every-day use of maths.