Reviewing Conference Proposals is Like a Job Interview

April 17, 2011

I just got through my first experience as a reviewer for a couple of conferences and I feel like I learned a lot in the process. I made a lot of mistakes, some of which felt silly and others I still feel a bit bad about. You see I have been submitting proposals to conferences myself for a couple of years, and I know how heartbreaking it can be to get your proposal turned down for a conference. Especially when you know your material is really great. So, it was both revealing and initially a little bit scary to be on the other side of the process.

To begin with, often all you have to work with is a submission form from an automated system. This really severely constrains the amount of information that you have about a given proposal or the person who submitted it. It reminds me a lot of the job interview process. As the hiring manager, all you get for your initial input is a resume – perhaps the world’s most ineffective information communication tool. Somehow, using only the text on the page, you have to divine the personality of the applicant, their knowledge of the subject domain, and assess the overall merit of their application.

Now, if your experience with the job interview process has been anything like mine, you know all too painfully well that there is almost no way in hell to choose a decent job candidate solely based on their resume. The information is excessively sparse, there is no feedback, and you have no way to validate the assertions that are made. Oh, and you have many more candidates than you have jobs, so contrary to what they might tell you in HR, you are very likely looking to filter people out.

So when you are filtering resumes, desperately trying to find the good candidates, you usually adopt some criteria for assessing the quality of their resume. These criteria are usually things like:

  1. Spelling and Grammar
  2. Clarity of thought and presentation
  3. Attention getting words, thoughts or ideas
  4. Relevant experience
  5. Etc.

Of course, none of these criteria really translate into a guarantee of a superstar future employee, right? In fact, all of those criteria are pretty weak indicators of quality overall when you are looking for the next great programmer. However, initially they are really the only guides to you are given to assess whether or not a candidate is worth investigating further.

The same problem applies to reviewing submitted proposals to a typical conference. You don’t have anywhere near enough information, and the criteria that you apply are very likely inadequate to the task of identifying a quality proposal. Therefore, you end up with a set of criteria like this:

  1. Spelling and Grammar
  2. Clarity of thought and Presentation
  3. Attention Getting words…
  4. Relevant experience…

I hope that you can see where I’m going with this by now. It is an imperfect system at best. It can be further aggravated by submission systems that actually conceal information in the interests of fairness. Some conferences will “anonymize” the proposals so that you do not actually know who the submitter is. I think this is done in the interest of creating reviews that focus on the merit of the ideas alone and not the reputation of the presenter. This practice has an unintended consequence of further restricting the information that the reviewer has to work with. Imagine reviewing resumes where you cannot see names or work experience and you start to get a feeling for what working with anonymous proposals might be like.

Details

At its most basic, with some review systems I feel like you are really left with the following: Did the submitter care enough to provide a detailed description of the proposal and how it would be presented? Were they willing to invest the time and effort to provide me with as much information as possible? My experience is that all too often people, even very experienced presenters, will skip over entire sections of the submission form or provide only single sentence answers. Often, you can very quickly break the pile down into two stacks: people who bothered to fill in all the blanks with some decent detail and those who do not. I think many folks who submit to conferences would be stunned to see just how often people neglect to fill in the details.

As a reviewer, you are left with two stacks: those who did provide detail and those who did not. Which stack would you prefer? Now does that mean that people who left out information in the proposal had poorer presentations? No. It is very likely that there are some great proposals that get overlooked this way – in fact, I’m quite sure this happens all the time. However, let’s face it, getting your submission accepted to a conference is a competition. You need to do everything you can to understand what the reviewers are looking for. First, I can tell you that rich detail sells big. It tells a reviewer that you are willing to do the extra work to sell them on your proposal. Investing in the detail suggests that you may understand your topic and know how to deliver it. Even with rich detail, there is no guarantee that the presenter is any good, but what else do you have to work with? Often not very much.

I have seen especially impressive proposals where people provide links to video of themselves giving the presentation, links to the PowerPoint slides, and more. When someone is able to put additional material like this into a proposal, I find it very impressive. It tells me that they are very passionate about their topic and that they are willing to go out of their way to provide additional detail (that reviewers are starved for) in order to be considered. Very few people bother to do this, so when people actually bother to provide this kind of information, they *really* stand out. It is still not a guarantee you will be accepted, but believe me it puts you closer to the top of the list than the bottom. Just like in job hunting, you want to do anything you can to make yourself stand out.

The Problem with Themes

Even filling in all the blanks and providing significant detail often isn’t enough. I think it guarantees you are in the hunt, but there is more to consider. One of the toughest considerations as a reviewer is theme. Often there is some sort of conference or stage theme that you are responsible for satisfying as a reviewer. All too often I have seen terrific proposals that I was convinced would make compelling and interesting sessions, rejected because they didn’t appear to match the theme of the conference or the stage they were submitted to. For example, if you submit a proposal on “Writing Great User Stories” to a conference that has “Radical New Ideas” as a central theme, you are more than likely going to be rejected. No matter how great the material is, no matter how wonderful a presenter you are. If there is a perceived mismatch between your topic and the conference or stage theme, you are very likely out of the running.

Now I think that by their nature, themes are dreadfully subjective and vague and this is a bit of a tough nut for the submitter and reviewer to crack. I think conference organizers feel compelled to use themes to help give their attendees some sense of the value they intend to provide. It seems that most conference organizers do not feel compelled to just fill their agendas with any old good presentation that comes along. They also do not want repetition. From what little I’ve seen so far, it’s pretty easy to end up with three different proposals that all seem to boil down to, “Another Intro to Scrum”. Even if you have great presenters like: Mike Cohn, Jeff Sutherland, and Ken Schwaber – in the end, only one gets picked. In addition, if it is a Lean/Kanban conference, probably none of them gets picked. That is regardless of the quality of their stellar proposals or their godlike presentation skills.

Is that fair? I don’t know. Personally, I hate it when I feel I have material that is valuable, has been well received by my audiences, and supported by a solid proposal – and it is rejected. I am a competent presenter. I want to tell the reviewers what boneheads they are, that they missed a great opportunity! I want them to know they could not see value if it kissed them on the nose! But I don’t. That just does not seem like a very smart approach to me. So, I have a beer with my friends and go on a bit of a rant – fortunately they tolerate me, and I get over it.

Decision Time

Themes aside, sometimes it also just comes down to a matter of taste. As a reviewer, you are confronted with two great proposals and you only have room for one. You have to make a difficult choice where there is no obvious winner. In a case like that, it really will come down to some sort of gut instinct (often wrong) that you end up relying on to make the choice. You can put in place scoring systems and other mechanisms to make the decision appear more objective, but the bottom line is that it is a subjective judgment and you have to make a call. Of course, it does not make you feel any better when you are on the wrong end of the decision. It is hard to understand how your rocking proposal that you poured your heart and soul into could have been rejected.

Those feelings are natural enough and I understand all too well how they come about. The point is that you need to keep the bigger picture in mind. A rejection by a given conference may have little or nothing to do with your skill as a presenter or your mastery of the subject matter. That’s part of what makes this process, like job hunting, so bloody frustrating. We would all like to crack the code and have our genius recognized. However, the process, the information, and the people are imperfect. There are a depressing number of ways that great material can be overlooked.

If I haven’t put you completely off your feed by now, I’d recommend a couple of resources to follow up on if you are interested in improving your chances of getting your proposals accepted:

Mark Levinsons Blog:

http://agilepainrelief.com/notesfromatooluser/2011/04/how-we-reviewedagile-2011-coaching-stage.html

Mitch Lacey’s blog:

Part 1: http://mitchlacey.com/Blog/Agile-Conference-A-Stage-Producers-Story-Part-1.html

Part 2: http://mitchlacey.com/Blog/Getting-your-Session-Accepted-to-the-Agile-Alliance-Agile-Conference.html

Part 3: http://mitchlacey.com/Blog/Stage-Producer-101-Building-the-Best-Stage-Possible.html


Classifying Impediments

February 25, 2009

2009-02-26-220518

 

 

 

 

Categorizing things can be a very useful tool for understanding the world around us. I watch my 2 year old daughter doing it all the time. She points at a robin in a tree and chirps, “Birdy!” Then she sees an eagle in a book, points and says, “Birdy!” I guess Homo sapiens must be pretty good at drawing these kinds of associations, because it’s apparent that we start doing it at a very early age. We use categorization to discriminate the different things we encounter in the world.

Recently I have been trying to look at impediments with “new” eyes. If you keep an impediments log you can see patterns arise and common themes become apparent as you review impediments with similar causes. When I first started tracking my impediments I used a spreadsheet to keep track of them. I was trying to create a history of impediments that I could review and reflect on both by myself and with the team. At first, I just started out recording the name of the impediment and tracking whether or not it was resolved. Soon however, I started adding more categories to the list of impediments. I wanted to track when I first became aware of the impediment, and when it was resolved, I wanted to categorize the impediments so that I could track themes – impediments that might share a common origin. I also started to track the root cause of the impediment – what had occurred (or not occurred) that had led to the impediment in the first place?

As I tracked my impediments over time, the list of categories grew longer and longer. Here is the list of impediment categories that I track today:

Missing dependencies: This could be software, hardware, or people dependencies. All too often something needs to happen, but something required to make it happen is missing. I want to order a burger at the drive through, but there is no one to take my order at the window. I picked up the idea for this category from Matthew May’s excellent book, “The Elegant Solution”. He outlines a lot of the lean strategies used at Toyota and the principles that support them. In one chapter he categorizes many different kinds of waste that can be found in any process. It was reading this chapter where I had a “eureka!” moment: impediments = waste!

Defects: This one is pretty obvious most of the time. Something is broken. There is a bug in the system. Something isn’t working as intended. The heater in my car seat stopped working. The bugs assigned to the team in triage. The little plastic lid won’t snap on to the top of my latte and stay.

Not enough time: The story of my life. Usually this indicates a scheduling or commitment problem.

Interruption: Do you have your email client setup to alert you as soon as a new email arrives? That’s an interruption. Interruptions can be obvious, like when the phone rings, or they can be subtle, like the way we manage our email.

Incomplete Work: Half done work can be as bad, if not worse than work that was never started at all.

Waiting: This category is everywhere. Try keeping a notepad with you and noting all the times that you wait during the day – for anything. You may be surprised at how much waiting is going on in your day.

Miscommunication: Communication between team mates – between teams – between silos…

Decisions not made: Organizational dysfunctions are impediments – sometimes the toughest impediments to resolve. The good news is that they are also frequently the kinds of impediments that once resolved, deliver the most reward.

Poor Maintenance: OK, I admit it – I was thinking of my car when I came up with this category. Without revealing too much, I realized that there are a lot of things that need maintenance in my life, and they all need to run smoothly (my car, my house, the dishwasher, the servers that run our software, etc.)

Disorganization & Clutter: Anyone who has seen my desk knows where this category of impediments came from.

Over commitment: Sometimes impediments arise from the very fact that we are just trying to accomplish too much. The good news is that this category of impediment is easy to resolve – just back off the throttle.

Lack of control/Discipline: This encompasses those impediments that arise because of something that wasn’t done. Sometimes there is an established process that isn’t followed – that can be a problem. Other times the process itself can be the impediment.

Forgotten: Alright, so my memory isn’t what it used to be. OK, my memory was never that great. The point is, forgetting to do things can be a big impediment to getting things done.

Distractions: email, web browsing, and the three martini lunch – It’s amazing I get anything done at all. It pains me to think that many of the things that we get such pleasure from can also impede use from achieving our goals (such pleasurable impediments). This category of impediments is the hardest for me to see.

This is by no means an exhaustive set of impediment categories. I’m quite sure that there are many more.  The categories can overlap too. An impediment can fit into multiple categories at once (i.e. waiting and decisions not made seem a natural combination).

There is one other benefit of categorizing impediments that I am only now starting to realize – categories provide a language for talking about the impediments for your team. It’s a lot like patterns in the respect that simply giving a name to a class of impediment allows us to discuss the issue as a group without getting locked into specifics. It provides a level of abstraction for the discussion – and dealing with abstractions is where we make unexpected connections to other solutions.

I see working with categories as a tool that we can use to help remind ourselves of the things that we should be looking for when we are seeking to reveal the impediments around us. Categorizing is something that we do very well as human beings. Scientists have been doing it for a long time (Kings Play Chess On Fine Grain Sand). On a cognitive level, categorizing our environment helps to frame how we think about the problems we are facing. It allows us to better discern the similarities and differences in the objects under study.

We need to take a scientific, empirical approach to working with impediments. Categorize them any way you choose – the very act of making the categories will help you to discover new impediments. Uncovering impediments is uncovering problems and it is in the solutions to those problems where you find innovative ideas for yourself and your team.