@Ren it's been a while
There are a few things going on. (First, to reiterate since it's been a while, I obviously don't think mathematics suffices to describe our world --- I'd give the apparent particularity of the laws of physics weight, until I have really good reason to suppose all possible mathematically formulated laws also correspond to something real.)
First main point:
In ontology, there always has to be some kind of distinction drawn between the merely-linguistic properties and the properties that we really treat as existing out there. People have different takes on whether this distinction is just a practical one, or if it itself is genuinely 'out there'.... but at least practically, to get the discipline going, the distinction has to be made.
How this is relevant to your passage is that people who say mathematical properties (like set-theoretic ones) really exist are clearly saying they're not merely linguistic artifacts (the kind of arbitrariness found in how British and French ways of referring to the same ontological property may be distinct).
Rather, they're saying the property of being so and so number is expressing something truly real. I'd say the British vs French distinction corresponds more to different formalizations of mathematical truths than the truths themselves -- i.e. even with mathematics, one might write the symbol for '2' differently in different languages, but still be pointing to the same property.
I do agree that saying 'reality is just mathematical-like' is missing the point, in the sense that I think the real mystery is that there are these two possible sorts of things in reality --- a world of abstract sets and a world of physical things, and the question is what's the connection between them (even if one doesn't think one or both exist, they're still apparently different sorts of beasts). Why do mathematical properties appear so intricately in both? It's a question similar to the mind-body problem, i.e. how do we reconcile qualia with the physical.
In any case, my own personal beef with orthodox physicalism is just the incredible lack of any sort of a priori grasp (or anywhere close) of
any non-mathematical truth.
The view of perception we both discussed holds that when we see things, what happens is some external entity causes an experience in us -- so we couldn't possibly have an apriori grasp of the thing that caused the experience, at most of the experience itself. We then have various such experiences repeatedly and discover laws of nature.
The main issue is that in either neutral monism/panpsychism/etc i.e. not orthodox physicalism, there's room to say that we learn something about the metaphysical nature of the world through those experiences -- they would always have to be dismissed as relatively arbitrary ways we conceptualize the world that tell us more or less nothing about its ontological nature (this appeals to the merely-linguistic vs ontological distinction from above that's pretty ubiquitous in metaphysics -- I think orthodox physicalists are happy to say we learn some kinda linguistic truth i.e. we learn to use the concept 'red apple' in a sentence competently). That there's some direct grasp the mind has of at least some of the concrete features of the world seems to be pretty hugely at stake.
Whereas, if we had some a priori grasp of the nature of experience, even if we can't directly grasp the apples, we can grasp the experiences of apples.
Basically, when I say "I perceive an apple/the apple caused the experience of an apple in me," I think the reason I think the relation b/w me and the apple is
causal is that the experience itself seems concrete --- I note a concrete state in me that I didn't before. If there's nothing concrete I really learn by the experience/at most I'm learning the use of a linguistic device, it's kind of weird to me that I feel justified in inferring causal relations from it.
I think what orthodox physicalists might say to this is to accept that in some sense the mind is blind to the concreteness of the world. Maybe we can grasp the idea of laws of nature a priori, so we do grasp what it IS to BE a law of nature a priori at most, but we certainly can't
discover laws of nature a priori, which means we in some sense just have to
assume that the things we're describing are concrete.