What's wrong with probability?

In everyday English, we say ‘probably’ to leave room for error and communicate some uncertainty. That’s fine because everyone knows we’re not assigning actual probabilities in the sense of the probability calculus.

In math, we use the probability calculus to describe the frequency of outcomes for underlying processes that look random. Like a coin toss. That’s also fine because we know all possible outcomes and we have a measure for each.

Things go wrong when people use probability even though they don’t know the outcomes (because of the growth of knowledge, say, as you write in #4762) or they have no measure for them or the underlying phenomena don’t behave randomly (again because of the growth of knowledge). Like Elon Musk tweeting we’re 90% likely to see AGI in 2026. (Not a literal quote but he says stuff like that sometimes.)

Some people try to steal the prestige of math and hide their ignorance by using the probability calculus illegitimately.

See also https://www.youtube.com/watch?v=wfzSE4Hoxbc. It’s been years since I watched it but it’s bound to have related ideas.