Mothering a child with autism is something I never expected when my husband and I decided to start a family. Our families are both pretty much neurotypical, but we had concerns about our son’s development from an early age. I’m a research scientist working with cancer data, but immediately after my son was diagnosed I didn’t want data and figures. I needed emotional support and a community, and so I turned to Facebook groups and other online support networks to find parents on this journey too.
Instead of that emotional support and a community, I found companies and individuals, some of them medical professionals, hawking dangerous miracle cures, expensive supplements and fad diets, completely unsupported by science, to treat children with autism.
I was genuinely stunned to see so many parents being taken in by the false claims, but I was even angrier at people using this community for their own purposes to manipulate parents. These hucksters use language such as “supported by scientific evidence”.
I discovered another problem when I turned to my community, the science-based community, to help support these parents make evidence-based decisions for their children. Instead of offering support, what I discovered was very little empathy for parents who had been taken in by these dangerous claims. “These parents don’t trust us, so they won’t hear what we have to say,” I frequently heard, “I don’t understand how or why anyone could buy into this nonsense.”
On my quest for support, I was left feeling more afraid and isolated than ever, and I knew I wasn’t alone. I honestly felt like there was nowhere I could have a reasonable conversation. I realized that parents of children with autism were stranded on an information island, in desperate need of actionable science-based information to help their children, and that many in the science-based community weren’t willing to build a bridge to reach them. This leaves plenty of room for quacks to find a niche, and for falsehoods to spread like wildfire, fueling the frenzy of misinformation.
This is where I come in, if you’ll let me.
I’m more than happy to help build that actionable bridge that is so desperately needed, and I think we need to build it together. Otherwise, how do you tell me from a quack? Something that became clear to me in those autism support groups is that we’re speaking different languages. So today I’d like to talk about that phrase, “scientific evidence,” and when you can tell what is and isn’t evidence.
Evidence is a clearly defined concept in science, but it’s often used rather loosely outside of scientific circles. Scientific evidence is the product of testable and measurable factors with repeatable outcomes in multiple individuals.
I’m going to spend the rest of this post explaining that one statement, because it’s so crucial to understanding science. Let’s start off with the testable portion of that statement.
Scientific inquiry often starts when somebody notices something and says, “huh, that’s interesting.” That “huh” moment can come from an everyday observation, an unexpected outcome from an experiment, or by reading someone else’s research and noticing something that might not have been investigated. I’ve had that “huh” moment in my home life when my son was younger and I noticed he was less aggressive after he had been playing at the playground. In science, we need to break that “huh” down into its component parts so that we can test it.
But wait, if I’ve observed something in my home life, why can’t I just say that what I’ve observed is true, why do I have to test it? After all, I’m seeing it with my own eyes! Well, sometimes what we think we’re seeing isn’t actually what we’re seeing. More often than not, there are other things influencing our observation.
In the case of my son’s behavior what if it’s that we tend to go to the playground on the days when I have more time, and it’s actually the other, more tightly packed days that heighten his aggression? What if I’m then also more patient and relaxed on looser days, and it’s actually my behavior that he’s responding to? What if it’s not the playground at all, just being out of the house with no demands to have a bath or brush his teeth, or sit at the dinner table? In my mind, I’ve already decided it’s the exercise, but he’s also just a kid so it really could be anything affecting his behavior. So we need to test to see if what I think I’m seeing, is actually what’s happening.
Before we can test we need to identify what we’re testing. In science we call the things that we’re testing variables. The word “variable” literally means something that varies (or changes), and in scientific research there are two types; the independent variables and dependent variables. An independent variable is something that we observe, control or manipulate in our research. A dependent variable is what we measure as a response to that observation or manipulation. Another way to think of these variables is input and output. Your independent variables are your input, and your dependent variables are your output, outcome or consequence of that input. What we do in research is try to find these matching pairs by asking: if I do this, what happens?
Let’s go back to the example of the impact of playground time on my son’s behavior. I could control the amount of time he was on the playground, and even remove it from his day, and then see what his aggression levels are like. As I adjust the time on the playground, I would expect his aggression levels to be dependent on the independent variable “playground time”.
Don’t forget, the reason I’m testing this (in my imaginary experiment) at all is because I made a prediction about how the two variables, playground time and aggression, may be related. That prediction is called a hypothesis, and it’s what we’re testing in a scientific experiment.
A hypothesis can simply be defined as “what you think is going to happen.” Before I commit to my playground/aggression hypothesis, I need to conduct a literature review. I’ll need to read everything that has been written about the two variables and the relationship between them. If that sounds boring, don’t worry, science gets much more boring than a literature review. I’m going to skip over the data collection, entry and analysis portion of scientific research (which are arguably the most boring parts of science). I’ll address those in a later post.
In the case of my son’s behavior after playground time, let’s say I feel pretty confident about my hypothesis, after all, I live it every day. The literature review allows me to carefully define what it is I’m measuring, and provides ideas for the plausible relationship between the two variables. I may have a great prediction, but if the literature doesn’t give me any hint that it’s plausible, I may need to rethink my hypothesis. Hypothesis testing will establish if my observations are true associations. In the case of my imaginary study, it looks like a lot of work has already been done on this relationship, so it looks like I might be on to something here.
Once I have read literature on it and summarized it, I have my hypothesis and I’ve determined it’s plausible. Surely there is no way I will conclude anything other than a link between playground time and aggression. I can probably stop thinking about it too much at this point, right?
As I mentioned earlier there could be a whole host of other factors contributing to this behavior, and maybe not everyone who has studied this took all of those factors into consideration. I need to keep thinking about this, and every possible influence that could be tainting my observation to take the next step of designing a study that can measure and control for all of these issues.
It may seem obvious but in order to test something, we need to be able to measure it. The independent and dependent variables must be observable in some way to the people measuring it. We also need to have a way of quantifying what we’re seeing.
Think of a desk in a room. If I asked you to measure it, we would all agree that the desk is observable because we can all see the desk. Then, we would likely select a tape measure to report back, in inches, the dimensions of the table. Now, even if we place many desks in the room, we have a way of distinguishing them from one another by their dimensions. If I then start cutting the tables, we know which ones I have cut, because you can see the saw marks, and what their new size is because we have a “before” and “after” measurement.
Measurement is also what defines good research from bad research (I’ll address this in another post as well). In my desk example, I selected a good tool in the tape measure, and we can clearly see what it is I am measuring. But how do we measure emotions, beliefs and behaviors? There’s no tape measure for that. While these are more difficult to measure, science has come up with some pretty good ways to quantify them.
Back to the playground. To create evidence that playground play impacts aggression, I would have to be clear on what it is I am measuring. I do this through defining my observations. If autism is my desk, what does it look like? What is my tape measure replacement? What does aggression look like? It is difficult to measure aggression, because it can seem subjective, but there are scales that help us measure aggressive behavior. In the case of a child, we might count the number of times they throw a toy, yell or hit to measure aggression.
It doesn’t stop there. I then have to think about where the measurement is taking place. When we use the term “control for” in science, what we mean is to “remove” or “rule out”. So, if I want to rule out the influence of my mood on my son’s aggression, I have to remove myself from the situation. We have to be able to single out my son, the playground and his aggression as the only things “in the room”, or as close as we can get to this, before we can reach any possible conclusions about playground play and aggression in my child with autism.
That last part of that sentence is key in our evidence journey. We only have one subject (or data point), and that’s my son, coupled with my anecdotal observation, in an uncontrolled environment. I haven’t produced any actual evidence yet. Playground time may be helpful on a personal level for managing my home life. But it is not evidence in any way that all children with autism need playground time to reduce aggression. I know this seems a little confusing. It’s a beneficial observation that I have seen with my own eyes. I have a degree and background in science as well. Those are all helpful things. But I haven’t offered you any evidence.
Even if I was able to design my own little study, remove myself and single out my son, the playground and his aggression, it’s not enough. I would need to study a lot more children; hundreds more, like 1200 or so. The more children we are able to observe and measure, the more likely it is that we will see the truth in this link, if it is true. If you’ve read a scientific study, you’ve likely seen the letter “N”, “n” or “n=”. This stands for “number”, and usually shows up somewhere in the methods and results. Always look for this number. It really matters, because the smaller your sample size (the smaller the “n”), the less able you are to see a true link.
A good way to think about this is to consider who’s in your house and your community. Now think about how your family, your friends, and your community compares to a community across the country or even internationally. You immediately start to see that you have a lot in common with the people closest to you, and less things in common with those further away from you. This is important when you think about how your study is going to impact a broad audience. We call these “sampling errors” or “biases”, and they occur when factors we are not measuring impact the associations we see. These biases or sampling errors are more likely to occur in studies with low “n” numbers, or poor sampling techniques. If you only ask the people in your neighborhood a question, how can you know how people in another community or around the world would respond to the same question?
So, let’s say I have what I think is a really good study. I would then get it approved by my institutional review board (IRB) or a human subjects review board (HSRB). Researchers are heavily monitored by a board of members including representatives from law, religion, science and the general public. Look for these boards when you see social media claims of “research”. Who’s overseeing the research? If it doesn’t have an IRB, government authority like the FDA, or an ethics committee that’s holding them accountable, it’s not research. Also look at who is sponsoring the study. NIH funding adds credibility, for example. Let’s say I pass a review board, I get an NIH grant, I complete the study and it was published in a peer-reviewed journal, meaning other colleagues who research this field have looked at my methods and outcomes and signed off. I can close my computer and say that I’ve definitively determined that playground time reduces aggressive behavior in children with autism, right?
Not even close.
One study does not science make.
The next phase of creating scientific evidence is the repetition. Now I get put under the public microscope to see if I created a good study. Other scientists and researchers should be able to recreate my findings using the same or slightly altered methods. Most scientists will even go a step further and try to improve upon my study to further prove the link. This means someone may take 2400 children in their repetition of my study. Another investigator may use 1200 kids from the UK instead of the US. They are taking a similar approach, but they are changing something like the place or the sample size to really try and figure out if what I saw is true and valid. They’re taking my study and seeing if it holds true in their community. Then someone will do it again. Results should be repeatable in order to be true.
Over the course of several years, my study will be reproduced in various forms, and we may not all come to the same conclusion. It’s going to happen for a variety of reasons, because no study is free from error. But what we’re always seeking in science is a consensus; an overall agreement within the scientific community that the facts are pointing in the same direction, and those that don’t can probably be chalked up to error. Someone may even compile all the studies with similar methods on the same topic, and perform what’s called a meta-analysis. This is a statistical way of compiling all the information across similar studies and filtering out the errors. It’s quicker and cheaper than doing a completely new study, and is highly beneficial because it draws out the facts that hold true across all of the studies on a topic.
Now let’s say I have repeatable evidence that playground time reduces aggression in children with autism (quick reminder that I have not done this actual study, it is for illustrative purposes only).
Finally done, right?
Nope. Not even close.
One study does not create conclusive scientific evidence. What it creates is a big “huh,” moment for the scientific community. In order to have conclusive scientific evidence of something we need dozens of studies.
This brings us to the hierarchy of evidence. If we were to think of evidence as being a pyramid, the least trustworthy information would be on the bottom. Think of the widest base of the pyramid as a little bit of controlled chaos. Lots of personal stories, editorials, personal observations. The bottom rung isn’t considered evidence at all, but the next step up, is when we start to see the scientific method applied, but with low Ns. To get higher up in the pyramid, the evidence needs to be more concrete and absolute. The studies become more rigorous, but the number of studies with that level of rigidity are fewer. At the very top are the studies with the most evidence. Systematic reviews are at the top, randomized clinical control trials are then second, and our well thought out study with our 1200 children, clearly defined autism behaviors and proper tools for measuring aggression would be somewhere in the middle.
At the very top are systematic reviews or meta analyses. These are reviews done by researchers that take dozens of studies on the same subject, sift through the ones that had low Ns, used poor measurement tools, or controlled poorly for sampling biases and sees if these studies have come to any sort of consensus about the relationship between two variables. Because a systematic review includes many more studies, it also has many more data points (or Ns) than any one of those studies did. As researchers, we put much more weight into the studies at the top. And so should you.
The problem is, this is so not how our brains are naturally wired! In everyday life, our brains give personal experience much more weight than large swaths of statistical information. This is simple biology. We are programmed to trust our own experiences and the experiences of others for survival. As hunters and gatherers, we would watch a friend eat a certain berry and then die. Lesson learned, never eat that berry! Our brains think this way for our own immediate survival. We are simply wired to take anecdotal evidence, which doesn’t even make the pyramid, as fact. As researchers, we have to be specially trained not to believe what we see.
I really understand how parents get fooled by claims of miracle cures and treatments, because honestly, there is absolutely nothing natural about the scientific process. We, as parents, are emotionally invested in the outcomes. We believe them because we desperately want to. I also understand why researchers get frustrated and scoff at these claims and the people who believe them, because researchers have devoted their careers to seeking out and understanding evidence in the way I have just spent the last 2,300 words defining.
I feel like I have my feet in both worlds, and I want to close the gap. I understand how and why science moves so slowly. We need this level of caution and precision, but as a parent I also know that I need resources for my child now, not ten years from now!
If I can achieve anything here, it’s my hope to open the lines of communication. We are all in this together, and we need this bond to shut down the dangerous claims that frankly have too much room to breathe. As a researcher I need to remember I am a parent, one that needs support and help navigating the world of autism, not an eye roll from a colleague because maybe for a second I thought a special diet would help my son. As a parent, I need to remind myself that I am not better than the evidence, and neither is that quack peddling a cure on Facebook.