Mistakes I’ve Made on Reviews

Recently I’ve been fortunate enough to be a reviewer for a couple of conferences: Agile2011 and the Seattle Scrum Gathering. It was an eye opening experience for someone like me who has been on the submitting side of the process to see and participate in how the reviews actually get made.

As a rookie reviewer, I wanted to share some of my experiences with the review process in the hope that others may learn from it. I do want to emphasize that overall, it was a wonderful experience that I enjoyed and hope to do more of in the future. That said, I also wanted to be unflinching in my reflections on what could be improved in my own performance.

Overall, the review process has been very interesting (You can read more about the process itself here and here). I’ve made the following kinds of mistakes:

1. I’ve been too self-conscious in my feedback – afraid to take a stand. As a reviewer, it’s my job to have an opinion and not water it down in the interests of appearing “balanced”. Balance will come from the other reviewers, not from me.
2. I’ve been a little snarky or rude. In general, I do not do this too much, but everyone has a bad day. As someone who goes through the proposal process as a submitter, I know how painful it can be to be flamed by some jerk reviewer having a bad day. As reviewers, we need to keep a lid on that. We have an obligation to be firm in how we express our opinion, but polite.
3. I’ve been too brief in my feedback. Sometimes just asking for more detail is not enough. When I’m lazy, I can just blow through reviews and simply ask for more detail without expanding on anything else. I’m coming to realize that’s a cop-out. I was provoked by someone once to provide more detail. Their proposal was actually fantastic and I didn’t have a whole lot to say about it other than, “Great job!” Well, rightly enough, they called me on that and asked if I could provide a little more feedback that is useful. I got a little angry with myself, sat down, and pounded out a critique that was nearly two pages long. I was surprised with myself. It was good. Afterward I had to ask myself, “Why don’t I give everyone that kind of quality feedback?”
4. Not understanding the audience for the feedback. It’s one thing to communicate with a private audience of reviewers, “Hey, this proposal is total crap. The guy didn’t even bother to give his full name…” it’s an altogether different thing to share that feedback with the submitter, “Dear so & so, thank you for your submission, while there were many great proposals, we couldn’t accept all of them. Yours was rejected due the brevity of its content.” You don’t want to get stuck re-writing the feedback for hundreds of reviews. Even more reason to write reviews really well the first time.

Fortunately the system is pretty robust and forgiving of error. Given that there are teams of reviewers who provide multiple reviews, it allows room for error (and learning!). I don’t have to provide stellar feedback for every review if there are 5 other reviewers also providing feedback – hopefully one of us is going to provide some valuable input.

For me, the conference submission and review process has been about learning. I’ve learned as someone who submits proposals that sometimes even the best material will go overlooked by a given set of reviewers for conference. I don’t take it too personally. There is a rich ecosystem of speaking opportunities available to me as a speaker, and if I can’t find an opportunity one place, then it’s very likely that I will find it in another. If I am passionate about the topic, I am usually pretty darn persistent about it too. This persistence has paid off. The more I get to know some of the more prominent speakers in our business, the more I realize and respect that they have been dealing with these sorts of vicissitudes in the process for many years.

As a reviewer, I have been lucky enough to participate in making the sausage that we call a review. It is a messy process with lots of flaws. I feel an obligation to improve my reviewing skills over time in order to provide folks who submit proposals the best possible experience with the proposal review process. But as groups separated by a screen of sorts, it’s easy to get frustrated with the process.

Working the Conference Ecosystem – More on the Review Process

I have submitted to some conferences for 3 years in a row without any success. It sucks. I figure I haven’t yet cracked the code for what they are looking for. That brings me to the subject of the conference review process. Here are some of the processes I’ve seen (and I’m sure there are more):

1. The proposal “black hole” – you toss in your submission, you get only one shot and get no feedback or questions. 6 weeks later you receive an accept or reject email.
2. The proposal “incubation” process – usually a high feedback process with a supporting submission system that provides lots of peer review and allows for some evolution of a proposal.
3. The “invitational” process – just invite the presenters you like. No chumps allowed.

The Black Hole

I think the first system, the “black hole”, is probably the most common. The proposals all are batched up and reviewed by a committee of some sort. As an outsider submitting to this system you have absolutely no insight into the decision making process. You really don’t know how they make their decision. All you know is that after a set period of time you receive an accept or a reject and that is pretty much the end of the story. Not much learning takes place on the part of either party in this system. There really is not much opportunity for learning to take place when there is no feedback. The nice thing is that the anxiety is actually kept to a minimum. I know that sounds crazy, but compared to some other processes, the “black hole” proposal process is relatively painless. You are in or you are out. No convoluted explanations, no bogus feedback by reviewers who don’t know what they are talking about, no agonizing over a million little revisions. You are in or you are out – period, full stop. Sometimes ignorance really is bliss. There, I actually said it.

Incubation

The second system, the “incubation” proposal process, is very high feedback. My personal experience has been that this is a bit of a mixed blessing. Proposals that are promising, but perhaps would not otherwise be considered, have a chance to be improved and matured with a rigorous feedback process. I find this possibility very exciting. I like the idea of taking a proposal, perhaps from someone relatively new, and helping them to develop it into something really great. I think there is a place for a peer review system that provides new talent with guidance and helps them to bring their ideas to a new audience. As potentially pompous as that may sound, I like it! And in the ideal world this is just how it works.

However there is a dark side to these “incubation” proposal systems: sometimes the feedback really does more harm than good. In these sorts of submission systems I think there is a very high bar that the reviewer has to meet. Poor feedback is almost worse than no feedback at all. Often times these systems allow public feedback from the general audience. I have mixed feelings about this sort of feedback. While I think it’s valuable in some regards, some of my worst, most caustic, useless feedback has come from these sorts of systems. People who are just venting their garbage. I guess as a someone who is proposing to a conference using an “incubation” system, you need to have thick skin. You can’t be too sensitive about your feedback. You need to be able to take the feedback and filter the wheat from the chaff. That’s probably true in any system, but certainly more so in one that allows unregulated public feedback.

Furthermore, as I mentioned before, there is a higher expectation for the quality of the feedback you will receive in a conference like this. As a reviewer, it’s a lot more work – a lot more dialog is necessary in order to help someone develop a proposal that needs significant changes in order to be approved. As a reviewer, that’s your job. To keep coming back and providing guidance and critique as the submitter makes changes. I’m always a little amazed when a submitter receives feedback and then doesn’t update their proposal. Feedback, even tough feedback, generally means that the reviewer is willing to continue the dialog. So go for it! Make the changes and then ask for more feedback! That’s what a healthy dialog looks like! Keep pushing until the reviewer gives in! After all, if you don’t respond to the feedback, you’re proposal is very likely dead.

Perhaps the worst case is when there is no dialog at all in the “incubation” systems. It happens. It’s the, “Great proposal, but not for this stage.” kind of feedback that will drive a submitter stark raving mad. This is a flaw in the reviewer, not the submitter. The reviewers need to work this stuff out and be able to give a coherent message to the submitter. Even worse, there have been times when there has been just a couple of, “great idea” comments and then your proposal is rejected. Again, this is a failure of the reviewers – reviewers really owe the submitters more than that.

Now I appreciate the fact that reviewers are human too. Therefore, I don’t expect miracles…often. But like in any herd there is safety in numbers. (Did I really just call a review team a herd?) As long as you can provide multiple reviews it is much more likely that at least one of you will come up with a cogent, intelligent set of critiques or feedback that resonate for the person who submitted the proposal. I’m not the most experienced reviewer, but I feel best when there are upwards of 5 reviews per proposal at minimum. Then I feel like a sufficient number of eyeballs have looked at the proposal and that there is a reasonable chance that the “wisdom of the crowd” will kick in and enable some useful dialog.

Invitation Only

Finally, there is the invitation only system. I really don’t have any experience with this, but I know of conferences that are run this way. I think on the one hand it offers a certain degree of reliability. As a conference organizer you are interested in keeping the quality high for your attendees and you aren’t interested in taking many risks. So, you stick to those you know and their friends and this system does seem to work. The flip side is that you aren’t necessarily going to get a lot of new voices and new ideas. Not every conference values innovation like that, but I suspect that for the conferences that do want to be on the cutting edge, you can’t afford to just invite those you already know. You need to take a few risks.

So Many Conferences, So Little Time…

One other thing that I try to keep in mind when submitting to a given conference is that there are a lot of conferences to choose from. Some are harder to get a submission into than others. A local open space conference is a great place to try out ideas and see if there is traction in the audience for them. The bar to entry is extremely low.

Then there are regional conferences where there is some review, but often they are quite easy to get into. The audience is still reasonably small, and there isn’t the intense competition to be a speaker. These regional conferences offer a great deal and can be a nice middle ground where you can continue to grow and nurture presentation ideas and delivery.

Finally, there are the big national and international conferences that garner a large audience and get lots of attention. There is a lot more competition to get submissions into these conferences. If you are coming up with an idea for the first time at one of these large conferences, you probably shouldn’t be too disappointed if it gets shot down for not being well developed enough. You will be competing against folks who have been developing their material at other venues and have refined things pretty well by this point.

I think the person submitting proposals needs to keep some perspective on the overall conference ecosystem in mind when submitting to a conference. A big national conference may not be the best place to float a new, untested idea for the first time. That’s not to say you can’t do it, but perhaps trying it out in a smaller venue would be well advised.

Reviewing Conference Proposals is Like a Job Interview

I just got through my first experience as a reviewer for a couple of conferences and I feel like I learned a lot in the process. I made a lot of mistakes, some of which felt silly and others I still feel a bit bad about. You see I have been submitting proposals to conferences myself for a couple of years, and I know how heartbreaking it can be to get your proposal turned down for a conference. Especially when you know your material is really great. So, it was both revealing and initially a little bit scary to be on the other side of the process.

To begin with, often all you have to work with is a submission form from an automated system. This really severely constrains the amount of information that you have about a given proposal or the person who submitted it. It reminds me a lot of the job interview process. As the hiring manager, all you get for your initial input is a resume – perhaps the world’s most ineffective information communication tool. Somehow, using only the text on the page, you have to divine the personality of the applicant, their knowledge of the subject domain, and assess the overall merit of their application.

Now, if your experience with the job interview process has been anything like mine, you know all too painfully well that there is almost no way in hell to choose a decent job candidate solely based on their resume. The information is excessively sparse, there is no feedback, and you have no way to validate the assertions that are made. Oh, and you have many more candidates than you have jobs, so contrary to what they might tell you in HR, you are very likely looking to filter people out.

So when you are filtering resumes, desperately trying to find the good candidates, you usually adopt some criteria for assessing the quality of their resume. These criteria are usually things like:

1. Spelling and Grammar
2. Clarity of thought and presentation
3. Attention getting words, thoughts or ideas
4. Relevant experience
5. Etc.

Of course, none of these criteria really translate into a guarantee of a superstar future employee, right? In fact, all of those criteria are pretty weak indicators of quality overall when you are looking for the next great programmer. However, initially they are really the only guides to you are given to assess whether or not a candidate is worth investigating further.

The same problem applies to reviewing submitted proposals to a typical conference. You don’t have anywhere near enough information, and the criteria that you apply are very likely inadequate to the task of identifying a quality proposal. Therefore, you end up with a set of criteria like this:

1. Spelling and Grammar
2. Clarity of thought and Presentation
3. Attention Getting words…
4. Relevant experience…

I hope that you can see where I’m going with this by now. It is an imperfect system at best. It can be further aggravated by submission systems that actually conceal information in the interests of fairness. Some conferences will “anonymize” the proposals so that you do not actually know who the submitter is. I think this is done in the interest of creating reviews that focus on the merit of the ideas alone and not the reputation of the presenter. This practice has an unintended consequence of further restricting the information that the reviewer has to work with. Imagine reviewing resumes where you cannot see names or work experience and you start to get a feeling for what working with anonymous proposals might be like.

Details

At its most basic, with some review systems I feel like you are really left with the following: Did the submitter care enough to provide a detailed description of the proposal and how it would be presented? Were they willing to invest the time and effort to provide me with as much information as possible? My experience is that all too often people, even very experienced presenters, will skip over entire sections of the submission form or provide only single sentence answers. Often, you can very quickly break the pile down into two stacks: people who bothered to fill in all the blanks with some decent detail and those who do not. I think many folks who submit to conferences would be stunned to see just how often people neglect to fill in the details.

As a reviewer, you are left with two stacks: those who did provide detail and those who did not. Which stack would you prefer? Now does that mean that people who left out information in the proposal had poorer presentations? No. It is very likely that there are some great proposals that get overlooked this way – in fact, I’m quite sure this happens all the time. However, let’s face it, getting your submission accepted to a conference is a competition. You need to do everything you can to understand what the reviewers are looking for. First, I can tell you that rich detail sells big. It tells a reviewer that you are willing to do the extra work to sell them on your proposal. Investing in the detail suggests that you may understand your topic and know how to deliver it. Even with rich detail, there is no guarantee that the presenter is any good, but what else do you have to work with? Often not very much.

I have seen especially impressive proposals where people provide links to video of themselves giving the presentation, links to the PowerPoint slides, and more. When someone is able to put additional material like this into a proposal, I find it very impressive. It tells me that they are very passionate about their topic and that they are willing to go out of their way to provide additional detail (that reviewers are starved for) in order to be considered. Very few people bother to do this, so when people actually bother to provide this kind of information, they *really* stand out. It is still not a guarantee you will be accepted, but believe me it puts you closer to the top of the list than the bottom. Just like in job hunting, you want to do anything you can to make yourself stand out.

The Problem with Themes

Even filling in all the blanks and providing significant detail often isn’t enough. I think it guarantees you are in the hunt, but there is more to consider. One of the toughest considerations as a reviewer is theme. Often there is some sort of conference or stage theme that you are responsible for satisfying as a reviewer. All too often I have seen terrific proposals that I was convinced would make compelling and interesting sessions, rejected because they didn’t appear to match the theme of the conference or the stage they were submitted to. For example, if you submit a proposal on “Writing Great User Stories” to a conference that has “Radical New Ideas” as a central theme, you are more than likely going to be rejected. No matter how great the material is, no matter how wonderful a presenter you are. If there is a perceived mismatch between your topic and the conference or stage theme, you are very likely out of the running.

Now I think that by their nature, themes are dreadfully subjective and vague and this is a bit of a tough nut for the submitter and reviewer to crack. I think conference organizers feel compelled to use themes to help give their attendees some sense of the value they intend to provide. It seems that most conference organizers do not feel compelled to just fill their agendas with any old good presentation that comes along. They also do not want repetition. From what little I’ve seen so far, it’s pretty easy to end up with three different proposals that all seem to boil down to, “Another Intro to Scrum”. Even if you have great presenters like: Mike Cohn, Jeff Sutherland, and Ken Schwaber – in the end, only one gets picked. In addition, if it is a Lean/Kanban conference, probably none of them gets picked. That is regardless of the quality of their stellar proposals or their godlike presentation skills.

Is that fair? I don’t know. Personally, I hate it when I feel I have material that is valuable, has been well received by my audiences, and supported by a solid proposal – and it is rejected. I am a competent presenter. I want to tell the reviewers what boneheads they are, that they missed a great opportunity! I want them to know they could not see value if it kissed them on the nose! But I don’t. That just does not seem like a very smart approach to me. So, I have a beer with my friends and go on a bit of a rant – fortunately they tolerate me, and I get over it.

Decision Time

Themes aside, sometimes it also just comes down to a matter of taste. As a reviewer, you are confronted with two great proposals and you only have room for one. You have to make a difficult choice where there is no obvious winner. In a case like that, it really will come down to some sort of gut instinct (often wrong) that you end up relying on to make the choice. You can put in place scoring systems and other mechanisms to make the decision appear more objective, but the bottom line is that it is a subjective judgment and you have to make a call. Of course, it does not make you feel any better when you are on the wrong end of the decision. It is hard to understand how your rocking proposal that you poured your heart and soul into could have been rejected.

Those feelings are natural enough and I understand all too well how they come about. The point is that you need to keep the bigger picture in mind. A rejection by a given conference may have little or nothing to do with your skill as a presenter or your mastery of the subject matter. That’s part of what makes this process, like job hunting, so bloody frustrating. We would all like to crack the code and have our genius recognized. However, the process, the information, and the people are imperfect. There are a depressing number of ways that great material can be overlooked.

If I haven’t put you completely off your feed by now, I’d recommend a couple of resources to follow up on if you are interested in improving your chances of getting your proposals accepted:

Mark Levinsons Blog:

http://agilepainrelief.com/notesfromatooluser/2011/04/how-we-reviewedagile-2011-coaching-stage.html

Mitch Lacey’s blog:

Part 1: http://mitchlacey.com/Blog/Agile-Conference-A-Stage-Producers-Story-Part-1.html

Part 2: http://mitchlacey.com/Blog/Getting-your-Session-Accepted-to-the-Agile-Alliance-Agile-Conference.html

Part 3: http://mitchlacey.com/Blog/Stage-Producer-101-Building-the-Best-Stage-Possible.html