Posts

, ,

Musings about statistics by a statistics-phobe

By B. Staffan Lindgren, Professor Emeritus

A while back, a paper accepted by The American Statistician entitled “The ASA’s statement on p-values: context, process, and purpose” was posted to the American Statistical Association website. The gist of the paper was that many disciplines rely too much on the p-value as the sole indicator of research importance. Not surprisingly, the paper received considerable attention.

Over my career, I had a love-hate relationship with statistics, knowing just enough to be dangerous, but not enough to really understand what I was doing. Consequently I relied on packaged software and/or colleagues or students who were more quantitatively minded than myself. For example, I generally made sure that a graduate student committee had at least one member with some strength in statistics to make sure I would not leave the candidate stranded or led astray. So if you read my thoughts below, keep in mind that I tread on very thin ice here. I fully expect some disagreement on this, but that is the way it is supposed to be. Ultimately it is your responsibility to understand what you are doing.

The approaches and tools for statistical analysis have changed a lot since my student days, which was at the dawn of mainframe computers for general use, on which we could use a software package called Textform rather than typing the thesis on a type writer as I (read “a secretary I hired and almost drove to depression”) did for my masters. My first visit to a statistical consultant at Simon Fraser University ended with the advice that “This data set can’t be analyzed, it contains zero values.” The software of choice was SPSS, which did not allow for any complexity, so I did a fair bit by hand (which might have been a good thing since it forced me to think about what I was doing, but certainly did not prevent errors). Later in my career it was sometimes a struggle to decide among differing opinions of statisticians what was and was not appropriate to use, but with a little help from my friends I think I managed to negotiate most of the pitfalls (no pun intended) fairly well.

The author with his eponymous insect trap, sometime after struggles doing statistics with room-sized computers. Photo: Ron Long

The statistic-phobic author with his eponymous insect trap, preparing to gather data and test hypotheses. Photo: Ron Long.

One of the issues with our reliance on p-values is that it is tempting to do post-hoc “significance-hunting” by using a variety of approaches, rather than deciding a priori how to analyze the data. Data that show no significance often remains unpublished, leading to potential “publication bias”. In part this may be the result of journal policies (or reviewer bias), which tends to lead to rejection of papers reporting ‘negative’ results. We have also been trained to use an experiment-wise alpha of 0.05 or less, i.e., a significant result would be declared if the p-value is lower than 0.05. There are two problems with this. First, it is an arbitrary value in a sense, e.g., there really is no meaningful difference between p=0.049 and 0.051. Furthermore, the p-value does not really tell you anything about the importance of the result. All it can do is give some guidance regarding the interpretation of the results relative to the hypothesis. I have tried to make students put their research in context, because I believe the objective of the research may dictate whether or not a significant p-value is important or not. I used to work in industry, and one of the reasons I left was that recommendations I made based on research were not always acted upon. For example, pheromones of bark beetles are often synergized by various host volatiles. But whether or not they are may depend on environmental factors. For example, just after clear cutting the air is likely to have high levels of host volatiles, thus making any host volatile added to a trap ineffective. However, a company may make money by selling such volatiles, and hence they would tend to ignore any results that would lead to a loss of revenue. On the other hand, one could argue that they have the customers’ best interest in mind, because if host volatiles are important under some circumstances, it would be detrimental to remove them from the product.

This leads to my thoughts about the power of an analysis. The way I think of power is that it is a measure of the likelihood of finding a difference if it is there. There are two ways of increasing power that I can think of. One is to increase the number of replications, and the other is to use a higher alpha value. It is important to think about the consequence of an error. A Type I error is when significance is declared when there is none, while a Type II error is when no significance is found when in fact there is one. Which of these is most important is something we need to think about. For example, if you worked in conservation of a threatened species, and you found that a particular action to enhance survival resulted in a p-value of 0.07, would you be prepared to declare that action ineffective assuming that it wasn’t prohibitively expensive? If you have committed a Type II error, and discontinue the action, it could result in extinction of the threatened species. On the other hand, if you test a pesticide, would a significant value of 0.049 be enough to decide to pursue the expensive testing required for registration? If you have committed a Type I error, the product is not likely to succeed in the market place. If the potential market is small, which tends to be the case for behavioural chemicals, it may not be feasible to use this product because of the high cost, which has nothing to do with statistical analysis, but could be the overriding concern in determining the importance of the finding.

One area where the sole use of p-values can become very problematic is for regressions. The p-value only tells us whether or not the slope of the line is significantly different from zero, and therefore it becomes really important to look at how the data are distributed. An outlier can have a huge impact, for example (see figure). As an editor I saw many questionable regressions, e.g., with single points driving much of the effect, but which in the text were described as highly significant.

Fig. 1. An example of where a single point is driving a linear regression. Take it away and there is no apparent relationship at all. Figure from http://www.stat.yale.edu/Courses/1997-98/101/linreg.htm

Finally, we need to keep in mind that a significant p-value does not indicate certainty, but probability, i.e., at p=0.05, you would expect to get the same result 19 of 20 times, but that still means that the result could be the result of chance if you only ran the experiment once. (If you run a biological experiment that yields a p-value close to 0.05 a number of times, you would soon discover that it can be difficult to get the same outcome every time). Depending on the context, that may not be all that confidence inspiring. For example, if someone told you that there was only a 5% probability that you would be get seriously sick by eating a particular mushroom, wouldn’t that make you think twice about eating it?? On the other hand many of us will gladly shell out money to buy a 6/49 ticket even though the probability of winning anything at all is very low, let alone winning the jackpot, because in the end we are buying the dream of winning, and a loss is not that taxing (unless you gamble excessively of course). I consider odds of 1:8000 in a lottery really good, which they aren’t of course, evidenced by the fact that I have never won anything of substance! So relatively speaking, 1:20 is astronomically high if you think about it!

Why am I bothering to write this as a self-confessed statistics phobe? I have mainly to emphasize that you (and by “you” I primarily mean students engaged in independent research) need to think of statistics as a valuable tool, but not as the only, or even primary tool for interpreting results. Ultimately, it is the biological information that is important.

/by
, , , ,

The W5 of Deciding about Graduate School

B. Staffan Lindgren is a professor of entomology at the University of Northern British Columbia, and 1st Vice-President of the Entomological Society of Canada. He has been the senior supervisor of 11 M.Sc. students and one Ph.D. student, co-supervisor of two M.Sc. students, and participated on more than 20 supervisory committees.

—————-

Recently I have been approached by several students asking about how to go about applying for graduate school. Furthermore, I and a colleague are doing a brownbag lunch discussion for the local student chapter of The Wildlife Society on this topic this week, and this got me thinking about what considerations a student should have. My conclusion is that you can break down the approach into a consideration of W5 (Why, Where, Who, What, and When) to optimize the chances of being successful.

In this post I will go over my thoughts on these W’s, and relate some of my own experiences, both as student and supervisor. I have not consulted the literature, but base this on personal experience alone, so you have to bear that in mind. For the record, I have not supervised a large number of graduate students, and all but one have been at the Master’s level. On the other hand, I have only “failed” as a supervisor once, which just means that I blame myself for the student’s failure to complete. On the other hand, I have also failed as a graduate student once, so I feel I have some relevant qualifications for writing this.

Why?

This question may seem somewhat redundant, but I believe it is an important first step. It is surprising how many students go into graduate school “to get a better job”. In my opinion, that is not a good reason at all. It is very possible, or in fact likely, that you can land a better job after completing a graduate degree, but there is no guarantee for advanced degrees automatically leading to better jobs. I have two examples. One of my more successful graduate students told me long after she graduated that she went into graduate school for this very reason. Somewhere along the way, she realized that she loved research, and her passion for it grew as a result. She subsequently carried on with a PhD, and now holds a very good research position. So in her case doing a graduate degree led to exactly what she set out to do to begin with, but it wasn’t graduate school per se that lead to her success, but rather her passion for what she was doing, along with some very hard work. My second example relates to my first, and failed attempt at graduate school. I was more worried about funding than topic, and opted to do a PhD in Endocrinology. I had really enjoyed my coursework in zoophysiology, so it seemed like a logical choice at the time. I was in a good lab, had a great colleague (who is now a professor with more than 300 authored or co-authored publications). As it turned out, it was not for me, however. The reasons were many, but a lack of passion for the subject area certainly contributed (see below).

Where?

Different institutions have varying reputations, and particularly if the ultimate goal is an academic position, it may make a difference whether you hold a degree from a major research university or primarily undergraduate teaching institution. However, there may be pros and cons with joining big labs. An obvious benefit is that a large institution is likely to have lots of infrastructure and resources. On the other hand, you may end up in a lab where your supervisor plays only a limited role in your actual supervision, i.e., you may be viewed more as a small cog in a large wheel than as an important individual. To avoid this, you have to ask the next question.

Who?

The supervisor is of critical importance in my opinion. All supervisors are not made equal, and they often have their own agendas and biases! Some may expect you to work things out for yourself, while others like to treat you like an employee. Depending on your personality, you may like one or the other, or somewhere in between. Highly productive, “big name” researchers are not necessarily the best supervisors! Moderately productive scientists at small institutions may provide a much better environment, particularly for graduate students lacking prior experience, e.g., Master’s students. I went into my first two graduate degrees (including the initial failed PhD in Sweden) pretty much blind. The endocrinology attempt was uncomfortable because of an internal schism between my supervisor and the head of the department, but other than that I was fortunate to get a very approachable and helpful supervisor. My supervisor for my Master of Pest Management and PhD degrees at Simon Fraser University was as good as they come; I learned an enormous amount from him, and model my own approach to supervision on that experience.  However, he did not suit everybody. The problem is matching your own needs and preferences with a suitable supervisor. I recommend all prospective graduate students to contact both former and current students of potential supervisors and ask what it is like to be a graduate student. I even recommend students expressing interest in me as a supervisor to do the same – I think of myself as a good supervisor, but I am clearly biased, and in control of the situation, whereas a graduate student would be dependent on my actions. Raise up front issues of support (not just salary, but field assistant, transportation, accommodation in the field, expectations). Ask about how the supervisor deals with authorship – believe it or not, there are supervisors who are prone to self-promotion. A good supervisor promotes his/her students, not themselves. Once you are in a graduate position, it is much more difficult to adjust things, so do your homework up front. I also recommend students to be frank with a potential (or existing) supervisor if there are issues. If you can’t communicate with your prospective supervisor before you are his/her graduate student, it is likely that you won’t be able to later. Sometimes this is just due to personality incompatibility, but it really doesn’t matter what the reason is if you end up in a bad situation. You are never going to go into a graduate position with 100% confidence that it will be perfect, but you can optimize the chances that it will be by doing some basic research.

A successful supervisor-student relationship can turn into a lifetime relationship: Staffan Lindgren (PhD 1982), Lisa Poirier(PhD 1995) and Dezene Huber (PhD 2001), gave back to their supervisor John H. Borden by successfully nominating him for an honorary doctorate at UNBC in 2009 in recognition of his enormous impact on forest insect pest management in British Columbia. Photo by Edna Borden.

A successful supervisor-student relationship can turn into a lifetime relationship: Staffan Lindgren (PhD 1982), Lisa Poirier(PhD 1995) and Dezene Huber (PhD 2001), gave back to their supervisor John H. Borden by successfully nominating him for an honorary doctorate at UNBC in 2009 in recognition of his enormous impact on forest insect pest management in British Columbia. Photo by Edna Borden.

What?

This is perhaps the most important decision you have to make, and it is closely linked to the first W (Why?). In my experience, the most successful students are not those who come in with the highest GPA or with the most funding (although it is easier to get accepted with those qualifications as it relieves the supervisor of some obvious burdens). Rather, they are the students with a burning interest in a specific type of project, or specific organisms. A great way to find your bearings is to get involved in research as an undergraduate student. When I was a PhD student, I had three undergraduate research assistants over the years. All three went on to get a PhD, one is now a research scientist with Forestry Canada, one is a conservation biologist with a consulting company (after Environment Canada was brought to its knees by the current government), and the third is a professor at a large institution in the United States. A number of students I have hired as undergraduate summer research assistants have successfully pursued successful careers. Decisions you make as a young person can profoundly affect your future. I went to the United States as a high school exchange student – without that experience I may have lacked the confidence to come to Canada for graduate school. As an undergraduate student, I participated in annual vole surveys and spider research, which taught me something about what types of activities I enjoy. When I first wanted to pursue graduate school, I failed to use that experience. My primary interest was entomology, but funding was hard to come by, so I opted for endocrinology because that graduate position came with a stipend. This decision turned out to be a huge mistake, and after 1 ½ years I had to give up. Essentially, I selected what to do for the wrong reason. (Thanks to my brilliant graduate student colleagues, I still ended up with five publications, which probably helped me get accepted at Simon Fraser University, so it wasn’t a complete waste of time, however).  At SFU, my MPM supervisor offered me a funded project that would have been applicable to Sweden, and he gave me 8 months to think about it. I eventually made the decision to take that on, and I have never looked back. Thus, once I reset the career compass to my original goals, I ended up where I always wanted to be, which is in forest entomology!

When?

Strangely, this question relates to both “Why” and “What”, although there is considerable variation among students in terms of what is right for each individual. In my experience, however, the most successful graduate students tend to have a little bit of “real world” experience before they pursue a graduate degree. In part, this may be because they have more experience, and therefore are more confident about their abilities, and possibly more aware of their weaknesses than someone fresh out of an undergraduate degree would have. These individuals have also had time to formulate what they are really passionate about, and in my mind, passion is the most important ingredient in a successful graduate degree. Yes, you need some basic skills (communication (written and oral), quantitative skills), a modicum of intelligence, and lots of patience for endless tedium (most research is 90% tedium, 5% frustration, and 5% elation), but you don’t have to be an A+ student. As a graduate student, a passionate B student will do better than a moderately interested A+ student any day. You would be surprised how many professors and successful scientists were relatively average in high school. If the timing is wrong, you may not be happy. For example, when I first tried to pursue graduate school and ended up in the wrong program, I could have waited 2-3 years and I may have had perfect opportunities in Sweden as a huge project on insect pheromones was initiated a year after I went to Canada. I had in fact contacted several of the professors that led that project, but at the time they didn’t have the funds in place.

I mentioned at the beginning that I failed as a supervisor once. This was a combination of not matching the student with an appropriate topic, and personal incompatibility. Both resulted from inexperience, as it was one of my very first graduate students. Even supervisors learn from experience.

I hope these musings are helpful you decide to pursue a graduate degree. Good luck!

/5 Comments/by
, ,

Landing an entomological tenure track job: perfecting the practice of academic kung fu

By Chris Buddle (McGill University) & Dezene Huber (University of Northern BC)

————–

Last autumn there was quite an interesting discussion on twitter among some entomologists in Canada about the ‘job search’ – more specifically focused on the process of seeking tenure-track academic appointments.  Many of us shared our sob stories, and although the time, place and characters varied, the common element was REJECTION.  Those of us who currently are lucky enough to hold faculty appointments remember the rejection to success ratio, and some of us still have stacks of rejection letters.  While most of us really enjoyed the academic freedom that came with working as a postdoc, the job-search process was more often than not discouraging and deflating, and a really difficult time in our lives.

Towards the end of the PhD program, most of us are riding high – our papers are getting published, we are truly ‘experts’ in our fields of study, we are being congratulated, buoyed by our peers and mentors, and we are ready to take on the world.   We found ways to get a post-doc and perhaps traveled to a different country for additional experience, with a sense of hope, optimism, and enthusiasm for the next stage of our careers.

Then, like the world supply of helium, our hopes were quickly diminished.

“I will easily get a job interview at THAT University”.

Nope.  Not even an interview.

“Perfect – that job advertisement was MADE for me – they will hire me.  It’s a perfect fit”.

Nope. A mass e-mail rejection letter instead.

“I’m the GREATEST in my field of study.  Universities will be asking me to apply”

Nope.  That never happens.

”I’m sure that I’ll be seriously considered for this position”

Nope. The rejection letter came back saying that there were more than 400 applicants for the position.

“Even if I don’t get the job, I’ll be able to get feedback from someone on the committee.”

Nope. It’s highly unlikely that, among the 400 applicants, anyone on the committee even remembers you.

There are really two ways to look at this.  It is possible to get discouraged and frustrated, and give up hope OR it’s possible to see that persistence can pay off and eventually the right job will come along, and you will be competitive.  Sure, the opportunities have to be there, but that kind of timing and ‘luck’ isn’t something you can control.

Here are a few pointers that will hopefully help you think about that tenure-track job search, and give you a sense of optimism:

  • It will take a huge dose of patience and persistence, but there ARE tenure-track jobs out there for people with Entomological interests, even in Canada. Recently, Manitoba hired an entomologist, and University of Ottawa just hired an assistant professor on the evolution of plant-pollinator interactions.
  • University professors do eventually retire! (…Although it needs to be noted that the reality in the current economy is that their positions are not always replaced)
  • You don’t have to restrict your options to only University positions.  We know of faculty members who worked in private companies, or in government, and made a lateral transfer, eventually, to academia.  Your holy grail may be a tenure-track job, but other opportunities are equally rewarding and could eventually get you a tenure-track job. Or you may find that life “beyond the ivory tower” is much to your liking anyhow. In fact, you may be interested in the advice column at Chronicle.com by that very name.
  • Be creative with your CV.  There are relatively few jobs for entomologists, sensu stricto, but there are jobs for evolutionary biologists, ecologists, or other more ‘general’ disciplines (Look: Concordia recently held a competition for a community and ecosystem ecologist!)  Re-work your cover letters and CV to reflect your potential in these jobs, and that you use insects as ‘model organisms’. And always tailor your cover letter and CV to any job for which you apply. Don’t just send in the same material to every search committee. Search committees are looking for that elusive thing that we call “fit.”
  • Keep your eye on the ball:  to get that coveted university position, the peer-reviewed publication remains the MOST IMPORTANT item on your CV.  Publish, publish, publish. During this stage of your career, keep the focus on that part of the research process. In particular, enjoy the fact that, as a postdoc, you are relatively free to conduct research and publish without many of the other responsibilities (e.g., teaching, administration) that will come with a tenure-track post.
  • Be realistic. If a job ad states that the committee is looking for an acarologist specializing in the mites of toucans, and you are an acarologist who studies toucan mites, then you have a good chance of landing an interview. If the job ad asks for a “terrestrial ecologist working at any scale from microbial to landscape” and you fit somewhere in there, chances are so do a few hundred other recent graduates.
  • When you see something that looks potentially appropriate for you, apply. Rejection is painful but costs nothing; not applying to something that might have worked out is doubly painful.  People who have agreed to write you letters of recommendation will be patient with you (if they are not, perhaps they are not the right people to give you a letter…?)
  • Have another postdoc or your mentor read through your application material. Chances are your mentor has been on a few search committees and can give you useful tips.
  • Every time you apply for a job, consider it a chance to improve your application material.
  • When you do land an interview, prepare for it like there’s no tomorrow. You are a researcher, do your best to figure out everything that you possibly can about the department to which you are applying and, even more, the personalities that make up that department.  Once you get an interview, this means your CV is strong enough, and the job interview is about the ‘fit’.
  • OK, to be fair, there are other tricks to success in academia.
  • Landing an academic position is not always going to be in the cards for everyone. It is best to have alternate plans so that you don’t get stuck in the so-called postdoctoral holding pattern for years and years. At least one of us (DH) committed to himself to start to explore alternate options at the five year mark after walking the convocation stage. Have a plan B. Your Plan B might actually turn out better than your Plan A in the end.
  • Rejection in terms of tenure-track jobs is really just a warm-up to the continual sense of rejection you will feel if you do end up working as a Professor.  You might as well get used to it.  This is not a statement to bring on doom and gloom: it’s the reality.  You must develop broad shoulders.

Rejection is a fundamental and core part of the academic life: The publication process is becoming so difficult that you can pretty much assume that your paper will get rejected the first few times around (check out this paper about rejection rates…).  Funding agencies are cash-strapped, and it’s getting harder and harder to find ways to fund research projects.  High caliber graduate students will ‘shop around’ for the best graduate program, and will often reject your laboratory. Be a practitioner of academic kung fu – use the weight of rejection against rejection itself by learning from it and applying it to your next attempt.

Depressed yet?

Don’t be.  A tenure track has so many advantages, and these far outweigh the annoying stream of rejections. And the other options available to a bright, young researcher are often as appealing (and usually pay more) than being on the tenure track anyhow.  ..but that’s a topic for another post.

/3 Comments/by
, , ,

Dear Buggy: Where Do Research Ideas Come From?

Dear Buggy is the the alter-ego of Dr. Chris MacQuarrie, a research entomologist with the Canadian Forest Service. You can ask Buggy questions of your own on Twitter @CMacQuar.

—————————-

Hello!

Dear Buggy has lept out of the pages of the ESC Bulletin and landed in the new and exciting wilderness of the ESC blog. My loyal readers shouldn’t worry, I’ll still be writing my column, but between editions of the Bulletin I’ll be posting here.

I’m very excited to be contributing to the ESC blog, but I’ll admit I am a tad nervous. When Crystal and Morgan invited me to contribute I was worried that it would be hard to come up with interesting topics. Thankfully the ideas began to flow after a glass of good scotch and I think I’ve come up with a few ideas that should keep me busy. After that? Well I’m always open to suggestions.

While thinking about this first blog post for ‘Dear Buggy’ I recalled how I felt when I first signed on to write Dear Buggy for the Bulletin. Where was I going to get these ideas!? Fortunately, a lot of the suggestions for my early columns come from the then-editor, Kevin Floate. Kevin had the original idea for Dear Buggy and shared with me his collection of questions and ideas. Later on, the ideas began to flow and inspiration came from others around me. Although, when I’m stuck for an topic I still go back to the original list that Kevin gave me. Good ideas can be hard to come by when you’ve got writer’s block and a deadline is fast approaching

As I planned this blog post I began to muse over the source of all my ideas, in particular “Where do I get my research ideas?”.

For example, when I was a new MSc student many of my research questions were influenced by the ideas of my supervisors. This isn’t all that unusual.  I suspect that when most of us started in research we were given, or at the least influenced, by ideas of others. As we mature scientifically we eventually start to come up with our own ideas. In fact, a good part of becoming a successful, independent researcher is tied to coming up with good ideas (which we might also call hypotheses). So where do these ideas come from? And perhaps more importantly, what do we do with these ideas once we have them?

I find inspiration hits at the oddest times and in the oddest places . I think Jorge Cham at PhD comics captured it best in this series of comics. Like most, I’ve been inspired in the ‘usual’ places: reading papers, attending seminars, talking with colleagues, etc… But inspiration can happen in other places as well. My mind tends to wander on my bike-ride home, when I’m pushing my daughter in her stroller, and quite often when I’m sharing a glass of scotch with my wife (who, lucky for me, is also an entomologist). As it turns out, this ‘mind wandering’ actually helps you have those ‘eureka’ moments, especially if you have been banging your head against the wall for awhile. I wrote recently about figuring out when you are best at writing. I think that advice can be extended to figuring out when and where you are inspired and to make sure you go there often.

But what about capturing ideas? My mind is like the proverbial sieve, but with one annoying quirk. I often can remember that I had an idea, I just can’t remember what it was.

To combat this selective memory I try to capture my ideas in my work journal as soon as possible. I’m a bit old fashioned so my journal is still kept in a notebook. Since my journal is also where I keep track my current projects, I make sure I highlight any new ideas so they are easy to find later on. There are many, many web-tools out there that can do the same job. The trick, though is to find something that works for you and to use it. My wife, for example,  is also an artist and long-ago got in the habit of carrying a sketchbook with her. That sketchbook now contains just as many ideas for research projects as it does ideas for art projects.

Finally, I must make a confession. Most of my ideas are bad. Some are half baked, others were thought of by someone else and rejected 30 years ago, a lot are impractical, infeasible, or near-impossible to execute or fit into the research that I’m doing. These over time get filtered out. Those that survive this process of natural selection, I keep. I then draw from this storehouse when the right moment comes along. Not all of these ideas will pan out of course, but by hanging on to the good ones I always have the right idea at hand when opportunity presents itself.

I’d be curious to hear about where you find your inspiration and how you track your ideas. Leave them in the comments section and I’ll summarize the best ones in a later post.

Cheers and see you next month,

Buggy.

/by