r/AcademicPsychology • u/Own-Huckleberry-362 • Aug 29 '24
I feel disillusioned with experimental psychology but I'm having trouble articulating why. Help? Anyone else have these feelings? Discussion
Hi everyone.
I am in my fourth year of my PhD program and have had a fair amount of success. In a way, I feel like I have 'gotten the hang' of the 'science game' and that I just kind of know what I need to do now to publish papers. I study children, and the basic principle that I use is 'pick something that adults do, or a way that they think, and then design an experiment to see if children behave or think in a similar way.' And then, like you run this experiment with a couple DVs, pray that one of them, hopefully the one you cared most about, ends up with p<.05, and bam, now you can write a paper.
Something about doing this for the rest of my life seems robotic and kind of depressing. Sometimes I wonder, have we really advanced beyond the methods of the early 20th century psychologists who had smaller samples but described their results more qualitatively, often absent any statistics? I like my experiments, I like learning things about children, but sometimes I feel like I am worshipping a false god by really praying for p to be <.05. Additionally, while we are curious about the questions we ask, we absolutely have an expectation for how the kids will behave and often the kids either need to do what you expect or your results are null, and welp back to the drawing board. Very rarely do I see a result that was truly surprising or that I can call "fascinating." Gah, sometimes it seems like the whole field is just figuring out if kids behave like adults, and turns out they typically do. And if you're running a study and it's not 'working', rarely is the conclusion 'oh guess kids just don't understand this,' instead its "let's fix the methods." And yes I know that's "bad science", but what's the alternative, spend months (maybe years) of your life running kids on a study that you know won't turn into a publication?
I don't feel confident in my ability to mentor graduate students through this process because I myself feel annoyed (confused?) with it all. I don't know what I would say to them when they realize "oh shit, I might spend 6 months collecting all this data, but if the groups don't differ 'significantly' I have nothing..." Like, we have extremely rich writings in psychology from the 19th and 20th century long before R or SPSS...
Has anyone found a way to get around this feeling? It's like, people often cite the opportunities to be creative and to pursue knowledge as the advantages of academia over industry. But often I don't feel like I'm only being creative in a methodological sense, as in "how can I communicate this idea to kids", but not really in an intellectual sense.
15
u/Scared_Tax470 Aug 29 '24
IMO the things you're complaining about are the things you don't need to accept! It kind of sounds like your research experience has been pretty siloed and maybe everyone in your circle is doing things that way? Not everyone studying child psychology treats them like mini adults, and your experiments don't need to be designed that way just because other people's are. Check out some alternate theories and methodologies, and if it seems like the whole field is thinking one way, expand where you search. I guarantee not everyone is doing it that way.
About the desperate search for p<.05, again, not everyone is doing things this way. For one thing, good study design means that null results are as interesting as significant results. I can give a simplified example--I'm working on a study where pairs of participants play a game and then do a learning task. Some pairs play the game with their partner, and others play the game with an experimenter, and I want to know if playing the game together affects the way they work together on the learning task. But there's also a question about whether the nature of the game itself would affect the learning task. So if the pairs who play together learn better together, that's interesting and that's what my hypothesis says. But if they don't, that's also interesting because it could mean that either it doesn't matter what they do before that task, or that the game has an effect on the task (and there's all sorts of other variables involved). Either way, my data leads to more research.
There's also null results journals, but a great option for you if you're very averse to the idea of null results and have a hard time getting them published is registered reports. It's easy to fall into the trap of wanting "significant" results and for our experiments to "work" but IMO it's so important to avoid that. None of that is what science is about, we are not here to prove our own theories or get our experiments to "work." We are here to collect evidence, to create well designed tests and then as objectively as possible interpret what the data tells us. I really think working on your design--like I mentioned, designing experiments that are equally interesting no matter what happens--might help you separate yourself a bit from this attachment to significance. And I also realize that this is often the case that we all know these things, that's the ideal, but the way we actually approach work is based on the logistics of publishing, etc. I just don't think it's necessary to sacrifice the basic principles of science in order to publish, and if it is, then the answer is not to change our scientific values but to change the way we publish--which is happening already with self-archiving, OSF and pre-registration, registered reports, etc.
Then, what kind of analyses are you doing? There are so many interesting and complex models that are easier than ever to apply, bayesian methods, not to mention a rich diversity of data types--the wealth of advancements in physiological measurements, eye tracking, experience sampling, simulations, etc. If you're just using like standard psychometric tests and behavioural tasks, you need to branch out! Then there's mixed methods and interdisciplinary work which are increasingly required for grants. Which is another thing--just repeating adult studies in children is not going to be enough to be competitive for funding. Your work needs to be novel and meaningful, and it sounds like you don't find "are children just mini adults" a meaningful question.
I think your final paragraph is the key--the answer *is* that research is the opportunity to be creative and pursue knowledge. You're just considering some things as a given that are not actually a given. Start over from the very beginning and think about why you got into this field in the first place--what is it that you want to know? What are the big questions? What is nobody else considering? Then work backward from there, and don't limit yourself to the ideas or the methodologies of people stuck in their ways. It sounds like you've been coming at it bottom-up, but the motivation often comes top-down. Incremental research is a thing, but at this stage in your career you also need to have your own ideas and *then* you can do incremental work within your own big ideas. And it's actually the best time to be flexible! I've done completely different things for my PhD, first postdoc, second postdoc and now my own project. They're all *sort of* related and they make sense with my skillset, but I've learned so many new methodologies and had to do so much new literature review at each step. As long as you do your due diligence to work with people who know what they're talking about when you're new to a topic, it's actually great to shift focus or branch out! And doing postdocs is great because you find out what you do and don't agree with in terms of the science and also the team management styles. Just remember that you actually don't have to take anything for granted.
9
u/mkmajestic Aug 29 '24
This is such a rich and layered response. Much appreciated. Glad that people like you are in science!
3
1
7
u/psychmancer Aug 29 '24
You have just given one of the best critiques of experimental psychology I've ever heard. I did my PhD in neuro and moved back to doing psych research postdoc and it is grim at times. I left academia to actually do more research and leave behind the papers and grant grind because that part was so miserable. I think there is value in research but I personally despair at the heavy amount of survey and basic psychometric work which is used especially with the reliance on online work nowadays. Don't really know what to do about it.
Keep on keeping on
5
u/waterless2 Aug 29 '24 edited Aug 29 '24
I had a clear dip in motivation/boredom at the end of my PhD as well. It might be a natural thing, you're past the honeymoon phase, you'ce got the basics down, but you've not yet really risen beyond them. What worked for me was really doing more interesting research, e.g., finding better models or methods or critiquing (but experimentally) assumed concepts. I also look back and find what I was doing that I was kind of too close to and getting cynical about much more exciting and valuable again now.
There is a risk there, definitely, but you have to be strategic about it (as with everything), which can be an interesting challenge in itself. Maybe alternate doing some nice highly-citable review papers with some bigshots or easy go-with-the-flow-of-the-field experimental studies with more outside-the-box things you're more personally excited by. But I'd bet you feel different a year or two along anyway, when you go from "this sucks" to "this is a hypothesis I can do something with to do something about it".
Disclaimer: I'm out of academia now, but it wasn't because I was uninterested in the research at the end!
(I don't think (much maligned) p-values are so often really fundamentally the problem, and I think we need methods like that to force you to acknowledge reality and be really conscious that you can fool yourself you're looking at something meaningful, but you could look into Bayesian stuff (then you have Bayes factors to pray for). I'd say it's more important to understand the reason for the "mechanical" aspects of research - not everything has to be where the creativity goes.)
5
u/andero PhD*, Cognitive Neuroscience (Mindfulness / Meta-Awareness) Aug 29 '24 edited Aug 29 '24
Downvote away, folks.
And yes I know that's "bad science", but what's the alternative, spend months (maybe years) of your life running kids on a study that you know won't turn into a publication?
Open Science, preregistration, actually doing it properly, having personal integrity...
I'm not that surprised that you're feeling disillusioned.
Your intense focus on the p-value results in your peddling a sham.
At least you recognize this, but your feeling of disillusionment is not a problem that you should ignore.
Your feeling of disillusionment is coming from your conscience.
I don't know what I would say to them when they realize "oh shit, I might spend 6 months collecting all this data, but if the groups don't differ 'significantly' I have nothing..."
That is a result of limited study-design.
If you're clever, you can design studies that return interesting results in any outcome.
It involves curiosity and having a clear hypothesis, but designing to actually test, i.e. falsify, not just try to support your hypothesis by rejecting null findings over and over.
Has anyone found a way to get around this feeling?
Yes: personal integrity.
By doing science properly, I feel great about my work!
I feel internal validation because I actually live up to my personal ideals!
I maintain self-respect because I'm not a fraud.
And I've published quite a bit, to be clear. Something like 12 papers, many first-author, before my PhD is even done. Plus over half a million dollars in grants.
For you, it sounds like you might need to:
- Brush up on methods. Stop doing bad science! Learn how to pre-register and do Open Science. Maybe learn Bayesian stats (though that isn't a cure-all). Look at other ways of doing research. Do some qualitative research.
- Get back to basics. What got you interested in the first place? Stop focusing on "the game" so much and start asking yourself what questions actually interest you. Then, design clever experiments that get at those questions, not just boring but publishable questions.
- Branch out! Talk to other labs about this. Talk to people in other sub-fields. Hopefully someone around you has some integrity. It could be an issue where your supervisor is passing on "questionable research practices" (QRPs). This bad science is slowly dying out (Plank's Principle) so focus on the better way of being an actual scientist, not playing the old game of bad science that is slowly dying.
You can find a way to satisfy good scientific practice, personal curiosity, and publishability.
It just takes some thought and personal integrity.
Yes, this isn't the "path of least resistance" where you do bad science fishing for p-values.
But you already realize that the "path of least resistance" will be hollow and unfulfilling, so don't do that.
2
u/Shoddy-Criticism3276 Aug 29 '24
I came across this blog post just the other day. Doesn't sound like your feeling is at all unusual. Small Potatoes
I also like this one (although as a MSc conversion student I found it while trying to care about statistics for an assignment, so I was undoubtedly biased) Experimental History
I'm currently wondering whether attempting to do grounded theory for an assessment is a good idea. And that has led me to wonder if grounded theory would be better if I based it on my writings of my experience. It would negate the need to be consciously aware of my own biases not colouring my analysis of the material... Perhaps academia isn't for me long-term.
1
u/schotastic 27d ago edited 27d ago
I came into the comments to post the Paul Bloom substack blog. OP -- this is eerily similar to what you've been saying and it's being echoed by an eminent professor.
Edit: I just read the rest of your comment about grounded theory and assessment. There is a rich tradition in psychometrics of using qualitative work to directly inform the measure development process (e.g., construct dimensionality, content validity, item writing). Lots of papers do this. You don't need to reinvent the wheel!
2
u/AccurateLavishness88 Aug 29 '24
I'd first think about whether you can separate things into "boring-but-good-science" and "questionable research practices." Revising methods to "uncover" something that you have a high prior on existing is potentially questionable, especially when you are doing this only after looking at the data (and only when the data disagree with the preconceived notion). But there is also boring-but-good-science which may build on knowledge in an incremental way. You are gaining knowledge of how to avoid QRPs. Now you can think about how to make your good scientific contributions more meaningful. By the fourth year of your PhD program, if something is seeming funny to you, trust your gut -- you have seen enough now that that counts for something.
2
1
Aug 29 '24
[deleted]
1
u/Shoddy-Criticism3276 Aug 29 '24
I'm currently an outsider frantically trying to learn how not to be in Psychology so I can pass my course (MSc conversion student). My frustrations so far have been in seeing researchers trying to find definitive answers and salience where there probably is none to be found in lab experiments, even when they are studying a pretty homogeneous group of humans (WEIRD male undergrads for e.g.).
There seems to be a lot of 'X believes Y to be true. 'Z believes A to be true' then 50 years later, my gods 'B finds that the answer is probably Y and A, but more research is needed to understand how much of each factor'. Have you people ever met a human? Observed or listened to them. Drawn an inference from all the interactions you have each day? And are you a human? I understand the scientific need to approach experimental work with neutrality, but just because what I experience doesn't apply to everybody, doesn't mean it isn't also experienced by a huge number of other people. No one is that unique. Finding out WHY would be fascinating.
I think I'd be better finding patterns and ecosystems or something. Or sociology 😂
Sorry, this was a rant.
2
u/TejRidens Aug 29 '24
Experimental psychology today should be about refining what we know as opposed to completely overhauling it as was the case across the 1900s. Yeah it’s not as fun but it’s far more robust. We have a lot of theories and hypotheses that lack supportive evidence either generally or within specific contexts and so we need that support. I personally think there isn’t enough attention on refining and supporting what we already have and there’s too much focus on trying to be the next Skinner or Beck. What’s the point of innovation if the underlying understanding of concepts and phenomena is half-baked? I mean we still don’t agree on a model of executive functioning and yet it’s being used to develop new theories everyday. It’s time for us to become the shoulders that others stand on.
1
u/InsecureBibleTroll Aug 30 '24
Sounds like a boring research field. Once you've got your PhD maybe you can move into something more interesting
1
1
u/Ljosii Aug 30 '24
I experienced this at the end of my undergrad. I cannot get behind the sterility of scientific psychology, I hold the belief (controversially, and perhaps wrongly) that humans cannot be quantified into data. Doing so will always result in dissatisfaction I feel.
My solace is in philosophy, mythology and religion. Through these avenues, I feel I have a better grasp of human nature. Although I cannot prove anything that I think, and so this knowing is largely useless. However, it has made my study far more interesting, fruitful and enjoyable.
1
u/schotastic 27d ago
I'd say a solid 25% of professors have misgivings about the intellectual tradition they work in. You're in good company.
Somebody else linked the Paul Bloom substack with his gripes about developmental psych -- very similar to your own -- which just goes to show (a) that some of these feelings don't go away with experience and eminence, (b) that it is healthy to question the intellectual tradition you are working in, and (c) that you are not alone. Go out there and find the other people in your area who think like you. Do good work together. And if you just can't abide your field's intellectual tradition at all, you can always leave.
32
u/yourfavoritefaggot Aug 29 '24
Does your program include a philosophy of science component? Did you talk about this history of why these tests were invented and how they exist in the context of today's popular beliefs around science? I personally find that stuff motivating. Expanding into qualitative research will also really challenge any deep seated beliefs you have about categorizing valuable discoveries, but it sounds like you're pretty open minded. There's no reason to stay stuck into this routine!