Random thoughts about Randomness; the math of random numbers

@maxdancona,

maxdancona wrote:

(I am thumbing you back up even though I disagree with you. I don't know why anyone is thumbing you down).

Haters gonna hate.

Quote:

I have been pretty clear all along. I have defined the term "meaningful" to mean that you test for it; I can use my mathematical definition distinguish "random" events from non-random events.

I keep trying to explain the two different levels of qualitative-cause vs. quantitative outcome. What causes a coin to flip and land in a certain way, or dice to role and stop, or a computer to measure a certain quantity; those are all extremely complex qualitative processes.

However, when you find something regular to count, you can derive numbers from the complexity and do various things with those numbers. Then, you invent explanations that connect the numbers you have processed back with you understanding of the realities from which data were derived.

The randomness appears at the level of counting and probability of outcomes, but the outcome is a qualitative observation of a complex system of causation. E.g. you observe the number on the horizontal surface of the dice facing upward, not the sides or bottom. You choose something regular about the symmetrical system to count and tally, and that is what makes it appear that there's something random going on; but in reality the coin or dice or whatever is actually moving through a predictable series of physically-mechanical steps to arrive in its resting state where the observation takes place.

Quote:

I use randomness professionally, it is important for my job. I need to understand what is random and what is not random in a mathematical sense, and I need to know the measurable consequences of this distinction. Concepts that are "meaningful" (i.e. measurable and testable) are useful to me as an engineer and equally useful to statisticians and cryptographers and scientists.

If you would analyze why it's important, you would find that there are human/artificial reasons that make it important in the ways they have made it important. Animals don't gamble on whether a falling acorn bounces down through the branches of a tree on one side or the other. It could be that if the acorn lands on the shade side of the tree it sprouts and grows better, worse, or just differently than on the sunny side; but nature ultimately spreads her bets to both sides of the tree and sometimes some of the offspring die, and other times they survive and work together to help each other survive. None of it is random, but humans can classify and count and do math, so we can reduce complex deterministic system to the appearance of randomness by focusing on the counting and math instead of paying attention to what's really going on to cause it all to happen as it does.

Quote:

I think you are trying to set up some idea of "underlying reality" as a form of absolute truth. I can't test this "reality". I can't study it. I can't measure it. I can't take advantage of it or use it to make technology or a simulation or win a poker game. I don't see the point.

Reality is reality. It only appears to be 'underlying' to you because you are fixated on the representation of it using quantification and math. You can see for yourself that a coin or dice exist beyond the outcomes that you count for statistical purposes. You don't count the position about where the dice land on the table or how far they land from each other, even though those are even quantifiable properties that you could gamble on if the casino allowed it. You also count the hue or brightness of the dice or coin from various vantage points depending on the lighting, even though that could also hypothetically be quantified.

There are many ways to quantify reality, but reality isn't fundamentally quantitative. Even when you get to the level of the orbital quanta of the atoms, where electrons jump back and forth between states without any continuously-variable in-between state, they don't do that to be counted. We can count orbitals and cooper pairs or whatever because we have the mental capacity to represent aspects of systems in quantitative terms, but that is something we overlay on top of reality, which doesn't operate by counting itself and processing itself using mathematical algorithms?

Or do you want to argue that God is a mathematician who makes everything happen as it does by counting and quantifying and processing outcomes, which He then enacts because of how His calculations came out?

1 Reply

@livinglava,

(thumbing you up again)

I have no idea about whether your ideas on "underlying reality" are right or not. There is no way to test them.

I do not believe that your philosophizing is important. I can use the mathematics to do useful things. I can use random numbers to secure a computer network. I can do load balancing, and testing. I can use the mathematics to detect cheating in a casino, or to accurately predict profits. I can use the mathematics to explain all sorts of things from conspiracy theories to stock market trends to radioactive decay.

The mathematical properties of random numbers are well understood. And they are useful. And they are testable.

Maybe your conjectures of some underlying reality are correct. Maybe they are not. I have have no way of knowing. I fail to see the importance.

1 Reply

Human beings have the tendency to see patterns in random events. We see constellations in the stars. For centuries we have used these constellations to make life decisions, as if the patterns in the stars had meaning beyond being a random arrangement of unrelated objects that could only be viewed from a very specific perspective.

That is the point of Ramsey theory. Order in randomness is often mathematically certain (by that I mean there is an 100% chance of it happening). It can still never be used to make specific predictions.

0 Replies

@maxdancona,

maxdancona wrote:

I have no idea about whether your ideas on "underlying reality" are right or not. There is no way to test them.

Why do you think the self-evidence of empirical observation requires 'testing?'

If you look outside and it's raining, does your empirical observation require 'testing' to be valid? Isn't direct empirical observation itself a test for hypotheses about what can be observed?

What you are implying is that mathematical representations are the fundamental level of reality and that empirically-observable reality is a claim about what 'underlies' reality. Are you just trying to make some kind of clever, Baudrillardian, claim about simulation being fundamental and reality having disappeared, or do you really not understand that when you count things, there are causal mechanics that manifest the regularities that you identify in order to count and do quantitative analyses?

Quote:

I do not believe that your philosophizing is important. I can use the mathematics to do useful things. I can use random numbers to secure a computer network. I can do load balancing, and testing. I can use the mathematics to detect cheating in a casino, or to accurately predict profits. I can use the mathematics to explain all sorts of things from conspiracy theories to stock market trends to radioactive decay.

But you don't understand that all these things you are talking about doing are just a subset of the broader reality that exists?

You think it takes abstract philosophy to recognize that there's a reality outside of counting things and using the numbers to do math?

Quote:

Maybe your conjectures of some underlying reality are correct. Maybe they are not. I have have no way of knowing. I fail to see the importance.

So to you reality has no point outside of quantification and math?

1 Reply

@livinglava,

Whether it is raining or not is testable. I can easily distinguish raining and not raining, and rain has direct impact on what I am doing.

Whether it is raining or not is important. Whether the rain is because of some unknowable deity, or quantum fluctuation or some arrangement of stars is neither knowable nor important.

1 Reply

@maxdancona,

maxdancona wrote:

Whether it is raining or not is testable. I can easily distinguish raining and not raining, and rain has direct impact on what I am doing.

Testable by empirical observation. What do you think 'testing' is except configuring various forms of empirical observation?

Quote:

Whether it is raining or not is important. Whether the rain is because of some unknowable deity, or quantum fluctuation or some arrangement of stars is neither knowable nor important.

Attributing relative importance is subjective. Cause and effect are not.

0 Replies

I want to talk about Bayes Theory. This important for understanding random results, and it is also the basis for things like email filters (this mathematics is at the core of the technology to decide what emails go into your spam filter), and it is obviously related to current events.

Let's say you have a drug test (or a corona virus test) with a 5% false positive rate and a 5% false negative rate. A large over-controlling employer will 50,000 employees wants to test all of their employees and fire anyone who uses drugs. Let's say that 2000 of them are actually drug users.

Alice tests positive for drugs and is dragged into the room of people who are about to get fired. Given the error rate for the test, what is the chance that Alice actually uses drugs (if she doesn't, it can be argued she is being fired unjustly).

Let's do the math.

1. Out of the 2000 people who do drugs we will catch 95% of them. That means that 1,900 actual drug users will be in this group.

2. Of the 48,000 employees who don't do drugs the tests will erroneously implicate 5% of them. That means that 2,400 people who don't do drugs will be wrongfully accused.

3. That means that of the 4,300 people who test positive for the drugs, only 1,900 of them will actually be guilty.

That means even though the error rate is just 5%, anyone who tests positive will be more than likely innocent. There is a 56% chance that although Alice tested positive, that she doesn't use drugs.

1 Reply

@maxdancona,

maxdancona wrote:

I want to talk about Bayes Theory. This important for understanding random results, and it is also the basis for things like email filters (this mathematics is at the core of the technology to decide what emails go into your spam filter), and it is obviously related to current events.

Let's say you have a drug test (or a corona virus test) with a 5% false positive rate and a 5% false negative rate. A large over-controlling employer will 50,000 employees wants to test all of their employees and fire anyone who uses drugs. Let's say that 2000 of them are actually drug users.

Alice tests positive for drugs and is dragged into the room of people who are about to get fired. Given the error rate for the test, what is the chance that Alice actually uses drugs (if she doesn't, it can be argued she is being fired unjustly).

Let's do the math.

1. Out of the 2000 people who do drugs we will catch 95% of them. That means that 1,900 actual drug users will be in this group.

2. Of the 48,000 employees who don't do drugs the tests will erroneously implicate 5% of them. That means that 2,400 people who don't do drugs will be wrongfully accused.

3. That means that of the 4,300 people who test positive for the drugs, only 1,900 of them will actually be guilty.

That means even though the error rate is just 5%, anyone who tests positive will be more than likely innocent. There is a 56% chance that although Alice tested positive, that she doesn't use drugs.

Look at how abstract the logic is here: You can manipulate 'the chance Alice does drugs' using math logic, when the reality is that Alice does exactly what she does and/or gets caught or doesn't.

Statistics manipulate logic at an abstract level to distract from actual causation. If you understand what actually causes people to use drugs or not, you have a better way of predicting drug use in an individual than looking at categories and correlations.

For every category that correlates with a crime and thus makes it seem more likely that an individual is involved in that crime, there is a window of opportunity for people who don't fall into that category to get away with the crime without suspicion.

That's why racism/sexism/classism are extra bad beyond what they do to the people in the high-correlation category who nevertheless don't correlated at the individual level. E.g. let's say Alice comes from a poor family and so her likelihood of drug use is statistically lower, but she gets seduced into drug use by a richer colleague who buys drugs for her and tells her using them at parties will help her network and become more successful. Then, the same category that causes Alice to be more of a suspect also exempts her richer colleague from suspicion.

As a result, Alice could get caught and fired while her richer colleague doesn't and goes on to seduce whoever replaces Alice into the same situation that got Alice fired.

1 Reply

@livinglava,

You are slinging nonsense, Lava. The mathematics provides the answers to two different questions. If you understand the question, you understand the answer.

If you ask the question... what percentage of the people who are accused of using drugs are actually a false positive? The answer is 56%. That is an specific answer. That is the correct answer. It is not abstract at all.

In this circumstance, 56% of the people who test positive will not actually have drugs in their system. If you care about that statistic (and it seems pretty important to me), then an understanding of the mathematics is pretty important.

1 Reply

@maxdancona,

maxdancona wrote:

If you ask the question... what percentage of the people who are accused of using drugs are actually a false positive? The answer is 56%. That is an specific answer. That is the correct answer. It is not abstract at all.

It's an abstraction. It's like asking what percentage of the ocean consists of wave-crests without thinking about why waves form, why they have crests, etc.

I keep trying to show you how correlation isn't causation, but you just want to see a number and think, "hey I can work with that and it's simpler than thinking about complexities." so you ignore the real complexities of causation that really explain things instead of just providing statistical snapshots of outcomes.

Quote:

In this circumstance, 56% of the people who test positive will not actually have drugs in their system. If you care about that statistic (and it seems pretty important to me), then an understanding of the mathematics is pretty important.

It's an estimate based on an algorithm. Whether the estimate actually turns out to be accurate will depend on testing it, but even if it tests accurate, it doesn't explain anything about the bigger picture of drug economics and why people use drugs, how they get them, how they avoid testing positive for them, etc.

1 Reply

@livinglava,

I am not interested in another argument whether mathematics are useful. Ironically you are using a technology based on mathematics right now, probably without any knowledge of Demorgan's theorem, or encryption hashes, or any number of the mathematics need for you to read this. So, I will chuckle and move on.

The statistic that coronavirus test has an error rate is a useful statistic. It helps you understand, and it helps people with mathematical knowledge make further useful calculations. I get your point that there is value in understanding what is behind the error rate. That doesn't change the usefulness of the statistic. The point of Bayes' theorem is that there are two questions.

1. What percentage of the people who have the virus will test positive?

2. What percentage of the people who test positive actually have the virus? (in other words; if I test positive what are the odds I actually have the virus?)

These are two different questions. The answer to each of them interests me.

The value of mathematics is that you can ask precise, specific questions... and when you get answers, they are well-understood and useful.

Bayes' theory defines the relationship between these two questions, and allows you to calculate each.

1 Reply

Bayes' theory is also used to filter email. It has gotten a lot more sophisticated since the early days, but Google and others still use "Bayesian filtering". It is the same principle.

1. You know what percentage of spam emails have a certain combinations of words.

2. You want to know what percentage of emails with this certain combination of words is spam.

So Google collects data to figure out #1. Then they use Bayes' theorem to calculate the value for #2. If 99% or higher of emails with this combination of words is spam, you probably want it put in your spam folder.

Notice that Google doesn't have to care about "why" something with a certain set of words is spam. It doesn't matter at all. They just do the math and apply the results.

Bayesian filtering has proved to be very useful for email filtering. And this is just one example. It is used for error detection, image processing and even face recognition. If you want to know the odds that someone with the facial features in this photo is Max, you probably want to use Bayes' theorem.

0 Replies

@maxdancona,

maxdancona wrote:

I am not interested in another argument whether mathematics are useful. Ironically you are using a technology based on mathematics right now, probably without any knowledge of Demorgan's theorem, or encryption hashes, or any number of the mathematics need for you to read this. So, I will chuckle and move on.

This thread is not about that. It's about randomness and probability.

I'm addressing how probability, statistics, and math more broadly are related to the causal reality that generates countable outcomes in various ways but is not fundamentally caused by any kind of numerical relationships.

Quote:

I get your point[/b] that there is value in understanding what is behind the error rate. That doesn't change the usefulness of the statistic. The point of Bayes' theorem is that there are two questions.

Not just the error rate. You're talking about the usefulness of statistics, but what I am trying to tell you is that you have to put statistics and other math in the context of how reality actually works at the causal level, otherwise you make the fundamental error of confusing correlation with causation.

'Correlation & causation' are only two words, and repeating them over and over doesn't emphasize enough how important it is to distinguish the fact the human mind can correlate various observation with the fact that reality is actually caused be sequences of events that occur outside the mind.

Quote:

1. What percentage of the people who have the virus will test positive?

2. What percentage of the people who test positive actually have the virus? (in other words; if I test positive what are the odds I actually have the virus?)

These are two different questions. The answer to each of them interests me.

Understanding why they are two different questions and how to make sense of them is something that goes outside and beyond any math.

Quote:

The value of mathematics is that you can ask precise, specific questions... and when you get answers, they are well-understood and useful.

You don't have to defend the value of mathematics. When you understand reality beyond the mathematics, the value of math is self-evident in the fact that you understand the relevance of specific quantitative questions to a broader/deeper context in which they are meaningful.

It's because we teach kids math before/without teaching them the broader context of causal reality that math seems like some fundamental level of reality. There are lots of kids who can learn to crunch numbers but they have a really hard time with word problems and they grow up to rarely if ever apply math unless they are specifically required to do so for some reason, and they simply don't think in a broad/deep and critical enough way to arrive on their own at questions that evoke the need for quantification and math.

Probably money-management is the only way most people engage in math, or maybe nutrition; but how many people really think about the relationship between different quantities at a deeper level, e.g. how calories and grams in food are related/connected to the larger ecology of agricultural land use, etc.? People live more in a mindset of submission to authority where they think the purpose of math is to pass a test and get a better job, not to actually apply math to their own personal everyday life-theorizing.

1 Reply

@livinglava,

You seem fixated on correlation and causation. These are important concepts... but they have nothing to with the topics I am discussing.

If you make a measurement, it is just a measurement. You can do that just fine without having any understanding of the "reality" underneath.

Do you know how fast your computer is, or how much RAM it has?

1 Reply

@maxdancona,

maxdancona wrote:

You seem fixated on correlation and causation. These are important concepts... but they have nothing to with the topics I am discussing.

Probability of outcomes is just a form of statistical correlation. You are correlating the number of coin-tosses or dice-throws or whatever with the number of times you count a certain outcome.

You are ignoring what causes the outcome.

Quote:

If you make a measurement, it is just a measurement. You can do that just fine without having any understanding of the "reality" underneath.

No measurement is meaningful except as it relates to the reality that is being examined. You don't understand context because you just take it for granted without consciously/intentionally examining it.

Quote:

Do you know how fast your computer is, or how much RAM it has?

The fact that you can pose a question does make it relevant.

0 Replies

Bayes theory allows us to analyze situations where we have incomplete information.

You have probably taken Tylenol (i.e. acetaminophen). We know that Tylenol lowers fevers and reduces inflammation. We don't know how it works; scientists have not determined the biological mechanism behind these properties of Tylenol.

If we don't know why Tylenol works (i.e. what biological mechanism gives it its properties), how do we know it works at all? After all, correlation isn't causation. Just because when lots of people have taken Tylenol, their fevers have gone down doesn't prove anything.

We prove causation by randomized studies. You can divide 5,000 people into two groups randomly. Then give half of them tylenol and have of them some random bitter tasting chemical that isn't Tylenol. If the random group that has tylenol has significantly better results at lowering fever than the other group... you can say that you have shown a causal link between Tylenol and lowering a fever. The fact that we don't know how it works no longer matters.

Bayes law is at the core of this. When you run a randomized study you need to calculate the "error rate". You want to calculate the probability that your study results could have happened by pure random chance. If this possibility is on the order of 1 in 10 billion, you can be confident that your study is statistically valid.

1 Reply

@maxdancona,

maxdancona wrote:

We prove causation by randomized studies. You can divide 5,000 people into two groups randomly. Then give half of them tylenol and have of them some random bitter tasting chemical that isn't Tylenol. If the random group that has tylenol has significantly better results at lowering fever than the other group... you can say that you have shown a causal link between Tylenol and lowering a fever. The fact that we don't know how it works no longer matters.

You don't know what causes it to work. All you know is that you've compared it with something else that doesn't work.

Causation is mechanical. You can correlate that a billiard ball only moves when a player hits the cue ball and that it never moves when no player hits the cue ball, but the mechanical cause is that the player uses a stick to hit the cue ball into the other ball, and that's what causes it to move.

Causation is mechanical. If you assume causation because of correlation, all you really have is a hypothesis that is supported by statistics. It might be strongly supported, but you still haven't figured out what the causal connection is between the drug and its effect.

1 Reply

@livinglava,

You don't have to reject something just because you don't understand it.

1 Reply

@maxdancona,

maxdancona wrote:

You don't have to reject something just because you don't understand it.

Reject what? What are you implying that I'm rejecting?

0 Replies

We don't know why some people get leukemia and other people don't. There are risk factors, but even with people who have the same risk factors one may get the disease, and the other not. It is random. About 1 in 6,000 will contract it each year, many with no risk factors and no reason at all.

Let's say you have a town of 12,000 and one year 5 people contract leukemia.

This could be random chance, meaning that this town was just unlucky. Or this, could be a trend indicating that something is unusual that is elevating the rate of leukemia.

Most people would consider this question important. Given the number of towns in the US, what are the odds that 5 people would contract leukemia in one year just by random chance?

If this number is very small, then the possibility there a specific problem causing leukemia becomes likely.

Bayes' law can calculate the possibility that an event could happen by random chance. That's very useful.

1 Reply

Random thoughts about Randomness; the math of random numbers

Quick Links

My Account

able2know