I have always found that the most challenging part of working in academia is finding the right team. The right team can mean many things, but one of those things is having the same way of doing science. While it may seem obvious, it really isn’t. Like finding the right coach for you at sports, or finding the right partner in life, finding a good match is the difference between a successful long-lasting partnership and a painful experience with lasting harm to your self confidence. This is especially true in Economics where papers can take years until seeing the light of day in a peer-reviewed journal: being committed to a bad team for 5 years or more is 5 years or more of (unnecessary) headaches.
After several bad matches, I’ve become obsessed with finding ways to screen out bad matches as early as possible and avoid unnecessary struggle. It’s hard enough to do research as it is, that you don’t want to drag a bad team on top of that. You may want to screen teams on many things, but to me a central point is honesty. Honesty in your work, and honesty in how you communicate with and treat your partners.
As it turns out, honesty is a central aspect of good research practice in the emerging consensus from the science reform movement. There are as many ways of doing bad science–from p-hacking to hypothesis searching after knowing the results (HARKing)–, but we are now starting to see emerge a common framework for good research practice in quantitative fields, for example the FAIR data principles (see for example). These best practices essentially revolve around transparency, reproducibility and ethical conduct. There are many layers to each of these themes and people have different ways of applying those. To me, an important part of ethics includes the way we communicate with and treat each other in a team, especially those in positions of vulnerability–research assistants, Ph.D. students, postdocs, or anybody in a vulnerable position in the team. And sadly, academia is full of stories of horrible (male and female) supervisors and senior co-authors who treat their more junior peers like carpets.
One thing that I have found increasingly useful over the years is using pre-analysis plans as a screening device. What is a pre-analysis plan? A Pre-Analysis Plan is a document you write at the beginning of a project before producing any results in which you specify as many details as possible about the study. There are many guidelines and templates out there, here are a few good examples:
a template from the Berkeley Initiative for Transparency in Social Sciences (BITSS)
a page with list of advice and checklist from J-Pal and an official guide here
a World Bank blog post from David McKenzie with a checklist
Even though most of the discussion around using Pre-Analysis Plans focuses on their use in experiments to reduce p-hacking and HARKing (see this excellent paper by Brodeur, Cook, Hartley and Heyes, 2024), my experience has been that they are especially useful for team work. Whether you are working on your own or with a team, writing down a Pre-Analysis Plan forces you to clarify the key elements of your project, the research question, the estimation strategy, the key outcomes, how you will measure them and handle multiple hypothesis testing, and the potential channels underlying the effects of interest. These are all things that you will inevitably have to write about later in drafting the paper, so it’s not a waste of time. On the contrary, this can form the basis of the eventual draft. More importantly, it saves you a ton of time when preparing your data because you open up your dataset knowing what is actually relevant for your project, and when estimating your effects of interest because you don’t get lost in specification searching which in any case is in the dangerous territory of p-hacking.
Beyond the clear advantage of bringing clarity, I see two lesser known advantages to Pre-Analysis Plans for team projects, whether experimental or not.
The first advantage is that they force the team to sit down and discuss all the details of how exactly they are going to conduct the research. In that sense, it reveals early disagreements that are bound to arise later in approaching data. For example, you have to specify a sequence such as not estimating any results until you have tested the assumptions underlying your research design, agreed on a specification and released the Pre-Analysis Plan.
The second advantage is that by asking the team to do this together, you immediately reveal who will be a problem on the team, either because of low engagement or because of questionable research practices. In other words, a Pre-Analysis Plan can be used as a screening device for team members, and can be a good way to signal that you care about your own research code of conduct.