2020 has been an incredibly challenging year, and on a personal note, has marked an important milestone — graduating with my PhD in computer science from Cornell University. This has been a six year journey, where my personal growth as a machine learning researcher (from thrills of first discoveries to the laborious grind through publication rejections to identifying a broader research vision) also took place against the backdrop of the rapid growth and change of the entire field (2014 NeurIPS: ~2k attendees, 2020 NeurIPS: ~20k attendees).

With this year coming to an end, I’ve put together some of my reflections and lessons learned from my (Machine Learning) PhD experiences. I discuss topics including expectations going in, common challenges during the PhD (and some strategies for helping with them), keeping up with papers, the community nature of research and developing a research vision. I hope that these topics are helpful in navigating the PhD and research in Machine Learning!

  1. Expectations going into the PhD
  2. Common Challenges through the Journey
    1. Feeling completely stuck
    2. Feeling overwhelmed with keeping up with ML progress
    3. Feeling Isolated
  3. Three Useful Personal Skills
  4. Keeping Notes on Papers and Ideas
  5. The Importance of Community
  6. Developing a Research Vision
  7. Summary

Expectations going into the PhD

In the post title, I’ve referred to the PhD as a “journey”, an aspect often underappreciated, particularly if one is coming straight out of undergrad (which was my experience). A typical Machine Learning PhD is going to be ~5-6 years of relatively unstructured time, and during this, not only will you learn research skills and knowledge about the field, but you’ll also develop personal preferences on how interesting specific problems are, the aesthetics of different subfields, and even perspectives on the type of work being done across academia/industry/policy/nonprofits.

These evolving personal preferences will influence the type of research you decide to work on, and even the post-PhD career path you pick. But particularly at the start of the PhD, it’s hard to predict how these personal perspectives will evolve. In my case, I started my PhD fully assuming I’d stay in industry, part way through began seriously considering academia, and at the end made the very difficult decision to turn down academic offers and stay on in industry. So going into the PhD program, it’s often helpful to take it step by step, and focus on getting the most out of the experience (learning/research/community participation), instead of a very specific desired outcome (which is prone to change, and may also add unnecessary pressure.)

Common Challenges through the Journey

Doing a PhD can be an immensely rewarding experience, and, particularly in Machine Learning, offers the chance to contribute to fundamental scientific understanding as well as impactful technology deployment. I’ve been grateful to my PhD for providing many opportunities to experience both of these! However, the duration and unstructured nature of the PhD can also make it challenging. My journey definitely consisted of ups and downs, and at various times I’ve struggled with feeling isolated, completely stuck, and even overwhelmed by trying to keep up with the rapid pace of progress. Looking back, and through discussion with peers, I now know these low points can unfortunately be quite common. But because these experiences are shared across many people, there can also be strategies for working through them. Below I discuss some of these experiences and strategies.

Feeling completely stuck

One very common challenge is feeling completely stuck, either on a specific project or with regards to the research process on the whole.

If the challenge is a specific project, where you’ve pushed hard and it’s still not quite working, then some strategies that might help are

  • Making a write up: Collect any partial experimental results, mathematical insights, jotted notes on motivation, etc and take time to put together a write up. This can help with providing a picture of where things stand and where the important gaps are.
  • Pivot: if there’s a specific part of the project that’s not working, is there the possibility to reframe the question (possibly taking inspiration from related work) to make it more tractable?
  • Forming connections: are there links between what the current project focuses on and other areas of study? Can that connection be explored in this project? This can both help progress and in making the project relevant to a broader community.
  • Feedback on writeup: It might also be helpful to get feedback on the project write up from peers, collaborators and friends in the research community. They may be able to offer new perspectives or suggest improvements.
  • Workshop submission: it can also be useful to make a workshop submission. This also provides a chance to help collect together all the research results and get useful feedback. (For some time now, I’ve gained the most out of the workshops at machine learning conferences, due to being able to discuss/get feedback on ongoing directions and meeting other researchers working on the same area.)
  • Wrap up and move on: Occasionally, there may be a project which sounded promising in the beginning, but has been difficult to make work, and is also inherently challenging to reframe or form connections to other areas. In this (difficult) situation, it may make most sense to wrap up the project quickly and move on. If you have partial results, it’s likely worthwhile to create a final writeup of those and share, so one option is to do this, get confirmation from collaborators and final feedback, and keep it as an arXiv preprint or workshop paper.

If the feeling of being stuck originates from the research process more broadly, one important point I’ve realised is that gaining research maturity can often be very hard to measure, especially when evaluating yourself! Midway through my PhD, I started working on some healthcare applications, and was only making slow headway on learning about the area/writing papers. This had me feeling stuck and somewhat frustrated at my slowdown in research progress. But when I re-read some papers that I’d first come across at the start of my PhD, the depth and context with which I could understand their results was strikingly different from earlier on.

Critical aspects of research maturity — understanding the broader context of results, being able to form connections between different areas, quickly narrowing in on novel key contributions in your subfield — don’t immediately translate to tangible outputs (more papers). But they’re central for becoming an independent researcher with a rich research vision — arguably the main research goal of the PhD. And if you’re reading papers, learning about the field, and working on research directions yourself, (and maybe even teaching/mentoring) most likely you’re making progress on all of these important aspects!

Feeling overwhelmed with keeping up with ML progress

Machine Learning is a vibrant, fast-paced field. But the flipside of this is drowning in the flood of new papers, new preprints, new blogposts, new implementations, new frameworks, etc, etc. (Fun statistic: NeurIPS this past year had ~10k submissions and ~2k accepted papers — no wonder we’re feeling overwhelmed!)

My strategy in dealing with this has been

  1. Have a number of go-to links to find references to related papers. For me, this has been a combination of (i) subscribing to the arxiv stat.ML cs.LG mailing lists, arXiv-sanity, Twitter, (occasionally) reddit/MachineLearning, paperswithcode and perusing Semantic Scholar/Google Scholar.
  2. Keep a reading list of papers If I come across an interesting paper but don’t have time to read it then (often the case), I make a note of it and try to return to it later.
  3. Have a paper reading strategy If a paper is very close to research directions I’m actively working on, I’ll read it in detail, otherwise I’ll skim the abstract to get a high level picture.
  4. Occasionally read up on different areas Occasionally (maybe once per year), I’ll look into a few interesting areas I’m not working on, and read several papers in each to get a sense of what is being worked on.

It’s also helpful to remember that (i) everyone feels overwhelmed with the rate of publishing and (ii) many papers may rely on the same underlying idea, and often being familiar with the idea is enough for keeping up with the field.

Feeling Isolated

Another common challenge in the PhD is struggling with feelings of isolation. In the first couple of years of my PhD, some projects required that I kept laser focus on very narrow, specific questions, which were also highly laborious and (felt) never-ending. During those times, it was hard not to feel completely cut-off from other researchers and the broader field, and I’m very grateful for all the support and guidance from my PhD advisor in pushing through that situation.

More broadly, this scenario can be common especially earlier on during the PhD, where you might simultaneously be learning how to see through a research project from start to finish, and at the same time have less context and connections to the broader research field/community. Maintaining connections to the field/community can be very helpful in making sure you don’t feel isolated. Some ways to do this could be: (i) setting up collaborations with (senior) students/postdocs (ii) getting feedback on your work in progress – this might be your advisor/lab, but could also be other peers/mentors working in the field (iii) actively participating in the broader community, whether that’s through simply attending conferences, mentoring or organizing workshops.

Three Useful Personal Skills

Having discussed some of the common challenges faced in the PhD and ways to help address them, the rest of this post will overview some useful considerations for research progress.

In particular, I’ll begin by discussing three personal skills I found to be very helpful throughout the PhD: initiative, focus and perseverance. This is of course drawing on my personal experiences, and there are varying opinions on useful personal skills! But for me, coming out of undergrad, a key difference I noticed in the PhD program was the need to take initiative — whether that meant reading important relevant papers, doing rapid preliminary studies of the feasibility of different approaches, talking to peers doing related research, or even attending and being an active participant in conferences. Because the PhD broadly consists of unstructured time, being productive largely relies on your initiative for learning and conducting research.

Two other skills that I’ve found very helpful are focus and perseverance. When getting started with a new research direction, focus is very helpful to peruse related work, distill the key points, quickly learn from initial exploration and determine the main project directions. Perseverance on the other hand is very useful (especially) for wrapping up the research project: there’s often a long tail of edits/additions for the paper to be submission ready, and post-submission, further edits to respond to peer-review and paper rejections. It can be hard to muster the motivation to make all of these edits (especially when preparing the paper for yet another resubmission, and having newer, more interesting projects also going on), but the variability of the peer-review process often means it’s worth persevering through.

Keeping Notes on Papers and Ideas

Through my PhD, there are two documents, one started in my first year, and the other in my third year, that I’ve continuously kept updated. The first keeps track of papers that I’ve read – every time I read a new paper, I add it into the doc, along with a short summary of my takeaways. The document is now over fifty pages (which maybe means I should switch to Mendeley or Paperpile), and has been a very useful way to quickly flick back to papers I’ve read years earlier and get key points. The other document keeps track of research ideas. Everytime I have a promising new idea, I make a note of it. Over time, this has helped inform my research directions and highlight key themes.

The Importance of Community

One important property of (Machine Learning) research that took me time to realise is that research is fundamentally a community endeavor. The problems that we aim to tackle are incredibly difficult, and progress relies on the cycle of you building off of others’ ideas and others building off of your ideas. This is a crucial factor to keep in mind when exploring research directions. What is the community excited about, and why? Are there shortcomings or gaps? Are there natural next steps to study? Taking the time to discuss these questions and others with peers in the community is vital to developing well-informed, germane research questions. And if you identify an exciting, new research direction of interest to the field, it’s often useful to build a community around that direction — this can happen through initiating collaborations, disseminating key open questions and organizing workshops. From very early on in my PhD, I was interested in understanding the key empirical phenomena exhibited by our modern deep learning systems. But working on this topic then was very challenging. The field was evolving rapidly, making the focus of any kind of analysis a moving target, and significantly adding to the challenge of building a new community around this topic. So publishing my first deep learning analysis papers was pretty difficult, and definitely an act of perseverance! But since then, it’s been wonderful to witness and contribute to the growth of this exciting research area!

Developing a Research Vision

While I described earlier that when getting started with the PhD, it may be better to take things one step at a time and focus on the experience instead of a specific goal, from the research maturity perspective, the PhD does have a specific goal: to make you an independent researcher, with a rich (articulable) research vision.

In current Machine Learning research, with the deluge of papers, it’s easy to feel stressed about the need to continuously churn out publications. But while paper writing is an important skill, I think the crucial test of research maturity is being able to have knowledgeable perspectives on your field which help identify key research questions, connected by overarching themes — a research vision.

Having a well developed research vision is enormously motivating. As an analogy, it’s a little like completing a “paint by numbers” kit: instead of just seeing the color of each individual square, you suddenly appreciate the full picture.

So how does one develop a research vision?

As a first note, from my PhD journey, I think it’s hard to develop a full-fledged research vision without some years of research experience. In my first couple of years of PhD, I remember reading papers and watching talks of senior researchers, and being frustrated that I couldn’t identify/articulate interesting research questions nearly as well. In the years since then, the compounding effects of all the papers I’ve read, projects worked on, seminars attended, have significantly improved my ability to do this. (There is of course room to improve! Going forwards, this ability will continue to develop, as I gain more context and understanding of larger subfields.)

Being more specific about the stages that led to a (better developed) research vision: it started off with exploration, with my first few projects giving me diverse exposures and helping me understand what I found intrinsically interesting. From there, there were natural followup projects to study, which finally also led to some related questions on applications/deployment. All of this started coming together under the broad theme of Machine Learning Design and Human-AI interaction at Deployment, and, as research visions are good at doing, also inspired new questions. (I am deeply grateful to my PhD advisor for his insights, guidance and encouragement through all of this!)

As a final point, I want to emphasise that the years of experience really do have a compounding effect. As you work on research projects, it becomes easier to identify the salient ideas in research papers, this informs your personal perspectives and promising questions for next projects, working on these next projects makes it easier to absorb/give talks, which then circles back to help with identifying new interesting research directions, which eventually coalesce to form a broader vision.

Summary

In summary, doing a PhD can be very fulfilling. It is however a journey, and has its ups and downs, personal discoveries, and evolution of (research) perspectives. I’m very grateful for the many rich experiences during my PhD, and hope this post might be helpful for others on the journey!