ACL 2020 just happened virtually this week. If not due to the COVID-19, I wouldn’t have to pre-record my presentation talk1, and I’d have enjoyed the cozy and bright summer in Seattle for a week. I’d like to share my adventure at this virtual but very fantastic conference.

Overall, I cannot enumerate more words like cool, fabulous, superb, and alike to describe my impression. Throughout days of diving in, I felt it being so inclusive when hearing voices from people across the globe with diverse backgrounds. It was so accessible when many big names in the field were providing enormous practical and considerate advice. And it was so eye-opening when listening to thought-provoking talks given by renowned speakers doing amazing neuroscience, cognitive science, and psychology research.

This was my first NLP (*CL) conference. Luckily, drawing lessons from my innocent virtual experience at ICLR 2020, I was able to navigate the scheduled activities more comfortably. I’ll start with several highlights that I loved most, and then record some tidbits that may be useful to future me.

Highlights

If I can only say 4 things that I love, I’d choose the watch party, keynote/invited talks, Q&A sessions, and mentoring sessions.

Watch Party

A group of people watch talks together and share what they think afterward. This kind of watch party helps you learn what others think and can generate interesting thoughts and ideas. Thanks to my great advisor and awesome labmates, we planned a watch party before the conference. The party format is simple; each of us picks papers we like, and then we watch the pre-recorded videos with most likes together. We did this through web meetings every day at a fixed time (usually before the Q&A sessions of the talks).

I’d like to see if we can extend this idea to the outside group or anyone with similar interests in specific topics or papers. I plan to do this for future conferences that I’m going to attend.

Keynotes and Invited Talks

I agree that, in principle, we should attend keynote talks as many as possible at a conference. The talks are most likely to be inspiring and thought-provoking and provide insights into significant problems. I watched the keynotes twice, one alone and another with our group at the watch party. Watching twice is not a waste of time. Instead, it helped me understand the ideas better. And I had different feelings and thoughts each pass. For example, in the first pass of Kathy’s keynote, I realized we should pay more attention to the history of NLP and simply enjoyed hearing from what the experts say in Kathy’s interviews. But the second time, I paid more attention to the issues/concerns she raised in the talk: (1) we should look at the data more carefully, address tasks that matters and learn the tasks not datasets; and (2) we should not ignore the interdisciplinary parts of language, and should put more effort into the interpretability and analysis of outputs. Josh’s Keynote was also very inspiring and thought-provoking. He explained things in such an accessible and articulate way, and no wonder many conferences and forums have frequently invited him.

At first glance, workshop invited talks may not sound eye-catching, but I did find it equivalently thought-provoking as keynotes. Evelina’s talk especially amplified this feeling, and I also found her previous talk at NeurIPS (Composition as the Core Driver of the Human Language System) very inspiring and fascinating. I probably will write another post describing what I learned from that.

Mentoring Sessions

Attending mentoring sessions is the most valuable part for students like me. We all have questions and troubles in our research or life in general, but not all of us can hear the wisdom from senior people. I was lucky to attend two sessions; the first one was “Navigating research problems/directions in NLP and Managing up” given by Lillian Lee, Kenton Murray and Amrith Krishna. And the second one was an additional session2 named “The effective communication in research” provided by Prof. Jason Eisner and Valia Kordoni. It was great to have more than one mentor, which helped cover a wide range of possible concerns. I was happy and grateful that they could share their unique experiences and practical and insightful advice. The mentors are very wise, considerate, and approachable. One concern frequently raised by juniors was how to approach people during a conference like this one.

I also heard there were other wonderful mentoring sessions that I was not assigned to or were not able to attend due to constraints. For example, topics like “Long-term career planning + Becoming a research leader: building your professional identity”, “Establishing collaborations and networking”, “Whether to do a PhD and how to apply for and choose a graduate school”, “Choosing between academia and industry”, “Changing career direction to NLP” etc are all covered. I’d like to suggest future conference organizers that it will be great to post session information beforehand so that we junior people could choose at will.

Q&A Sessions

I wish I could have attended more Q&A sessions. Each conference(including workshop) paper has two 1-hour time slots for Q&A sessions to accommodate people from different time zones. I love this format; people can go to sessions of their interests and not only ask questions but also discuss topics they are passionate about. For example, people came to my sessions to ask questions that they were not able to find the answers in my talk (my bad 🤔 ) or simply because they’d like to know more details they cared about, they were also happy to chat about things for next steps or just what might be relevant to their own work. I had a nice time talking to people at my session. One important thing I noticed was people may come without watching your talk in advance, so the preparation to give a 30-s ~ 1-minute pitch is necessary. The focus of the pitch could be the main take-home messages you want people to know. You could also walk through a few key slides with the audience. Take a glance at my version if you don’t mind 😂 .

I’d like more of this kind of Q&A sessions. Probably we could make each session shorter (say 30 minutes?) but more frequent (like 4 time slots spanning different days?).

Tidbits

Tutorials and Interesting Papers

Tutorials. I attended two tutorials “Interpretability and Analysis in Neural NLP”3 and “Open-Domain Question Answering”4 based on my interest. I’ve been doing question answering research for a while, and I’m very happy to see what the experts say. One emerging paradigm for open-domain QA is the “retrieve-free” approach where the models don’t need to retrieve evidence passages to answer questions. I’m pretty excited about doing research in this direction and have grown bigger interests in playing generative models like T5 or BART. I believe that interpretability is essential to language research, and it’s important to understand models and their outputs to make applications more robust and reliable. I’m a newcomer to this area, so taking a tutorial lesson was a good fit for me. I really liked the extensive examples the speakers gave in the tutorial, which helped me better understand the problems or arguments they wanted to convey.

Interesting Papers. This year we’ve got a new “theme” track of papers. As said on the official website:

A theme paper provides a better understanding of a class of related problems or approaches and provides a holistic analysis and view which may not be apparent by looking at each problem/solution individually. You do not need to present new solutions and results in a theme paper.

I enjoyed the two theme papers: “Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data” and “The Unstoppable Rise of Computational Linguistics in Deep Learning”. Out of curiosity, I found Prof. Emily M. Bender had given a tutorial at ACL 2018: 100 Things You Always Wanted to Know about Semantics & Pragmatics But Were Afraid to Ask*, this is a follow-up material of the “Climbing towards NLU” paper for people who want to know more about “meaning”, “form” and “understanding”. Interestingly, I found the theme paper “What Question Answering can Learn from Trivia Nerds” presented a new innovative video format where the two authors interactively ask-and-answer questions to explain their ideas.

As for the regular tracks, I wouldn’t say too much because I’m gonna read them in detail anyway, and that could take another post if I’d write down my takeaways. It’s worthy of note that authors in the field are so good at giving names to the papers, just name a few:

  • “Contextual Embeddings: When Are They Worth It?”
  • “What Does BERT with Vision Look At?”
  • “Two Birds, One Stone: A Simple, Unified Model for Text Generation from Structured and Unstructured Data”
  • “Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations”

Workshop papers were also worth to check out. I mainly attended RepL4NLP-2020. The paper “Exploring the Limits of Simple Learners in Knowledge Distillation for Document Classification with DocBER” shows that knowledge distillation provides a spectrum of effective and efficient baselines. One notable result they report is a single-layer distilled LSTM model achieves BERT-base parity on a majority of datasets, using at least 40× fewer FLOPS and only ∼3\% parameters. Since I work on efficient NLP research, I was able to attend the Q&A session of the paper Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning. It was very nice to talk to Mitchell Gordon about optimizing big models such as pruning and quantization.

Advice from Mentoring Sessions

In this part, I’ll try my best to put together a bunch of specific advice I’ve heard from the mentors or people who have given valuable suggestions. Before I start, I found there is a nice post already written (kudos to Zhijing!).

  • Opening the dialogue. People in the community are happy to chat in general. There is the evidence that people are happy to talk. But we do need to pay attention to the manner. One useful advice given by Prof. Graham Neubig is:

    To start a conversation, just tell me who you are (e.g. PhD student at XXX, working on YYY) and why you’d like to chat :) (quoted from his tweet)

    And also the nice piece of advice from Prof. Emily M. Bender:

    Better is to lead with a short intro (“Hi, I’m a student at X working with Y on Z”) and then jump into the common interest (“I really enjoyed your paper ____ and I’m interested in applying the ideas to ____”) and then a specific invitation/request. Especially in these covid times, an invitation to do a virtual talk + meeting might be a good way to go.

    As for emailing students before faculty: not looked down on at all!! That sounds like a great strategy. Also, the talk + meeting invitation will likely be more appealing to a student looking to build their network than to someone more established.

  • Prepare for meetings.

    i always bring a written document to meetings (normally weekly) (quoted from Kenton’s RocketChat msg) As an advisor who people have said the same thing about, what some of my students use to keep me on track is to preface their documentation with “[student name’s] key agenda items”, and a quick verbal agreement at the start of the meeting with time allocations. Also, if I get the documentation a day in advance, I can very easily be guilted into shutting up about a topic if the student says, “I already addressed that in the thing I sent you earlier.” (quoted from Lillian’s RocketChat msg)

  • Jason gave us fabulous pieces of advice for many things. You can find more at his advice webpage. For example, How to find research problems, Write the paper first, How to prepare a talk.

  • Compiled advice. “EffectiveCommunication, Long Term Career Planning in NLP, and Changing Career to NLP” from Ahmed Awadallah and Pradeep Dasigi: original google doc, archived pdf version(more accessible for those cannot use Google services). Amanda Stent pointed us to the CRA resource library. And Marija Stanojevic reminded us of The Tips for prospective and early-stage PhD students from ICLR 2020: original Google doc, archived pdf version(more accessible for those cannot use Google services).

  • Books. Amrith suggested Algorithms to Live By: The Computer Science of Human Decisions. Jason suggested twice Crucial Conversations: Tools for Talking When Stakes Are High(I’ve heard from other places that this book is a must-read if you want to learn effective communication, but Jason’s endorsement made me realize that I need to put it on top of my plates). I just bought the kindle version of the two books. This reminds me of another book that I love very much: “Why Buddhism Is True: The Science and Philosophy of Meditation and Enlightenment”, this is the only one (if I can’t choose two) I’d recommend (if I’m qualified) for literally anyone.

Many thanks to the above (and all at ACL 2020) amazing mentors. I feel greatly indebted to them. The above specifics are just the things that I’ve found useful myself. I believe there are many other specifics that are not covered. But if you have any tips that came across at ACL 2020, I’m happy to give credits and incorporate them here5. My collaborator Harsh has also compiled a list of PhD advice here, thank you Harsh.

Social

One integral part of attending a conference is to socializing with people. This year at ACL, we had many fantastic “Birds of a feather” sessions where people with similar interests on some topics hung out on Zoom rooms.

Birds of a feather meetups help junior researchers or researchers attending ACL for the first time strike up conversations with other researchers in their areas of interest. This can also be a good avenue for junior researchers to get feedback on ongoing ideas and learn about relevant ongoing projects at other groups/universities and facilitate more exchange of research ideas, and collaborative discussions. (quoted from virtual ACL website)

It was so refreshing to hear thoughts from people, I wish I could attend more.

As for direct message people, I was a bit hesitant and I think I could do better next time. But I was lucky to have a chance chatting with Roy Schwartz 1-1 since we share similar interests (efficient NLP). He has done tons of impressive work in the area of efficient NLP and interpretability. Thank you Roy for sharing your experiences at UW and AI2.

Off the main sessions, I also attended several so-called unofficial meetups. They were mostly created by junior students, it was fun chatting with them either just mentioning some random thoughts or not talking about research stuff at all. I hope we could connect in person next time.

Summary

In a word, this was an amazing adventure for me. Many thanks to the great organizers and all the people who contributed to this adventure! I feel like the community is so vigorous, lively, and inspiring. I definitely look forward to the next one!


Footnotes:

  1. Pre-recording was not a bad thing, though, because it took some energy to fix glitches, adjust the background sunlight, wait until environment noise went away, etc. 

  2. Junior people like me loved so much of the main mentoring sessions. So we ask for more additional sessions, big thanks to all the generous mentors. 

  3. slides and video 

  4. slides and video

  5. I plan to write another post (possibly together with friends) describing my compiled list of advice regarding research, writing, communication, etc.