Scaling Self-Organizing Systems

January 27, 2019

I was reading Geoffrey West’s book, Scaling: The surprising mathematics of life and civilization recently and something interesting jumped out at me. Where networks, whether they are biological or man-made, are concerned, there appear to be some efficiencies as the network grows. Generally speaking, fewer resources are required for the same amount of effort as scale increases two-fold or more. That applies when we are talking about networks. However, when West reviewed corporations he noticed that the same benefits of scaling did not apply. Surprise!

Hold my beer. I’ve got this…

The organizations that West examined were very likely hierarchies. Hierarchies are the worst performing sort of network. So, it is no surprise at all that scaling doesn’t work well for hierarchies. As hierarchies get larger, communication and the flow of resources tends to get less efficient. Living systems are self-organizing, cities are self-organizing, hierarchies are definitely not self-organizing.

If West had looked at organizations that were founded on self-organization like Morningstar, W.L. Gore, or Semco, I suspect he might have found a different result.

Hierarchy vs. Constellation

September 14, 2014


“Organizations must shift away from repetitive-function hierarchies with rules and enforcement and walls. Instead, we must migrate rapidly to becoming a global ‘team of teams’ that comes together in whatever combination necessary to add the greatest value to the changes underway.” – Bill Drayton

It’s easy to despair that people can not see social structures as anything other than dominance hierarchies. I suppose that makes sense given our primate origins. In college, I spent hundreds of hours observing chimpanzees at the local zoo as part of a research project. It doesn’t take very long to understand the hierarchy within a small colony of apes. The dominant ape spends a significant portion of this time strutting around making sure everyone knows he’s the boss. He uses every tool in the book, from the brutish power of dominance displays, all the way to the subtle selection of who gets to groom him first. He’s the big man on top.

However, the longer you watch, the more you start to detect other hierarchies within the primate social system. Female chimps have their own sub hierarchy that is just as much a power game as that of the dominant male. Within a very short time you are seeing different types of social hierarchy all over the place. I found myself in awe of the number and complexity of the hierarchies that chimpanzees display. I would maintain that the hierarchies and power games played by chimps rivals anything we have in even the most vicious office politics.

Of course, maybe the hierarchy is in the eye of the beholder. Perhaps we humans are wired to categorize things according to certain patterns. Maybe we are just inclined to see everything in terms of hierarchies as some sort of side effect of the way our perceptual systems are set up. I imagine there is some of that at work here. After all, as primates we have been fine tuning our hierarchical behavior for millennia, so it would be no surprise if we tend to see them everywhere we look.

That notion, that hierarchy is a sort of built in default, is a pretty depressing idea if, like me, you aren’t all that fond of hierarchies. There is no doubt that hierarchies have their advantages, but they have disadvantages too. Look how well the hierarchy has worked out for the Chimpanzees. They are well on their way toward extinction, so you could argue that whatever social strategy they are using, it’s not contributing sufficiently to help ensure their survival.

“Male chimpanzees have an extraordinarily strong drive for dominance. They’re constantly jockeying for position.” – Frans de Waal

It may be that because the chimpanzee is so geared toward hierarchy they are unable to utilize other social structures that would allow them to adapt to their changing environment. Perhaps it is their inability to change their behavior that spells their doom. On the one hand, blame the human for killing them. On the other, recognize that this is an adaptive challenge to which the chimpanzee has not found a winning strategy to counter.

Fortunately, humans appear to be capable of a bit more flexibility when it comes to our social and organizational structure. That is, while we still lean strongly toward the hierarchy (call it the default if you will), we seem to be capable of using other ways of organizing ourselves when the need arises.

That’s a good thing, because whether you are a business, or a society, we face a number of challenges as well. Recently a friend introduced me to Ashby’s Law of Requisite Variety,

If a system is to be stable the number of states of its control mechanism must be greater than or equal to the number of states in the system being controlled. Ashby states the Law as “variety can destroy variety”

In a nutshell, when a system faces a challenge, the complexity of that system must be equal to the complexity of the threat in order for the challenged system to survive. Let’s take that back to hierarchy now. Hierarchies consolidate decision making and rely on decisions made by the guy at the top. This helps to simplify the number of responses, which can be useful, but may not be the most adaptive strategy when under threat. In a complex environment, a hierarchy is a rather simplistic structure to use. It doesn’t have the requisite variety required to cope with a complex environment where challenges can arise in many different forms. fortunately, nature has provided us with many different models of social organization to choose from.

In a peer network, no one is officially in charge. It doesn’t have a command hierarchy. It doesn’t have a boss. So, all the decisions are somehow made collectively. The control of the system is in the hands of everyone who is a part of it. – Steven Johnson

Take insects, for example, they use a very different structure often described as a swarm. The swarm does not rely on a single individual or a subset of individuals to determine its response to a given challenge. This means that the swarm can use all of the members of its population to dynamically respond to a challenge. This leads to combinatorial explosion of different alternatives, which gives the swarm a huge arsenal of complex responses. It’s probably worth noting that on an evolutionary scale, the insects have been pretty successful using this strategy (and there are those who would argue they are the most likely to win).

Of course people can use these strategies too. Swarming and other social models are much more rarely used by people, but there are examples. Wikipedia is a great example of a swarm where there is no top down direction. Some organizations, like AA are also good examples. While there are not a lot of examples to choose from, with the increasing complexity of our social and business environments one might wonder if we may see an increasing diversity of swarms and other alternative social structures as a result.

Time will tell. I think the recent evolution of various and sundry Agile methods may be a hint that the underlying social structures are broken, and that people see a need for alternative structures to hierarchy in order to meet the challenges of today’s ever more complex challenges. If that is the case, then I expect to see many more of these Agile methods and frameworks arising in the not too distant future.

Breaking Down Silos

May 1, 2011

In the first two stages of our story of the Robbers Cave experiment we have explored how in-groups or silos are formed and how they can come into conflict. In the final stage of the experiment we learn how Sherif and his fellow researchers attempted to reconcile the conflict and defuse the tensions between the two groups. The third phase really gets at how we resolve conflicts between different groups. Everything else done in this experiment up to this point has been a setup for this phase.

The first thing to note is that the researchers considered a variety of different possible solutions that they could bring to bear in attempting to resolve the conflict between the two groups. As they outline it, they saw the following options:

  1. The appeal to a “common enemy” – when the study was first performed in 1949 they had used this appeal to try and bring the two groups together, but they were not satisfied with the results. They felt that using a common enemy as a tool to bring two groups together only serves to widen the conflict by introducing conflict with a third party. This sort of defeats the purpose of the study. The groups stop fighting each other only to start fighting with another.
  2. Disintegrating the two groups by focusing on the “shining” individual – This usually occurs at the expense of other individuals and the researchers felt that anything that brought about the disintegration of the groups again defeated the purpose of the study. The porpose of the study was to look for a way to defuse the tension between the two groups without destroying either group.
  3. Using “leadership” as a tool to bring the two groups together – The researchers felt that this approach would not work. They felt that leadership, while important in starting off a conflict, is really too weak a force in the social dynamic to have a significant impact on the direction of the group once the conflict really gets going. They felt that appeals to leadership would be ineffective very quickly after the groups came into conflict. In essence, leadership may influence the initial direction of a conflict, but once that steamroller gets going, leadership is too weak a social influence to stop it.
  4. Introducing common superordinate goals – goals that are important to and shared by the two groups but cannot be achieved by either group on its own. This is the option that they chose to test as the primary mechanism for resolving the conflict between the two groups.
  5. Contact as equals – this theory is that if the two groups can be brought together into contact with each other in situations where they are equals, that the contact alone will help to reduce the conflict between the groups.

It was the last two methods that the researchers resolved to put to the test. The Sherif and company didn’t seem to have a lot of faith in the Contact as Equals idea, but they felt that it was commonly held and practiced enough that it deserved some consideration. With that in mind, they setup multiple pre-arranged contact situations for the teams and looked to see if there was a significant change in the tensions between the two groups. Long story short – no. In fact, the researchers felt so strongly that this wasn’t the right approach that they set a hypothesis for this stage that stated:

“It is predicted that the contact phase in itself will not produce marked decrease in the state of tension between the two groups.” p.160

I guess that means that getting people from competing groups in the same room isn’t enough. Even if you do it a lot. The researchers in the Sherif experiment did, and apparently it had no effect – as predicted. So they didn’t have a lot of faith in this approach, but they gave it a shot.

The approach that they really liked was the solution that employed superordinate goals. As they put it in the study:

“When groups in a state of friction are brought into contact under conditions embodying superordinate goals, the attainment of which is compelling but which cannot be achieved by the efforts of one group alone, the groups will tend to cooperate toward the common goal.” p. 161

To test this hypothesis the researchers arranged for the following kinds of challenges:

  1. The Drinking Water Problem
  2. The Problem of Securing a Movie
  3. Campout at Cedar Lake
  4. Tug of War Against the Truck
  5. Meal Preparation
  6. Tent Pitching
  7. A Trip to the Border

I won’t go into the details of each story. A couple of things are apparent from this experiment. First, you can’t just do this once and have everybody walk away happy. Finding a single superordinate goal is not enough. I’ve seen this in practice too. I’ve been in situations where there were teams/silos in conflict and a situation arose where there was a superordinate goal. Everybody worked together to solve the problem, and then they went right back to their problem behaviors when the goal was resolved. What we see here is that their needs to be more than one superordinate goal in order for lasting changes to be made. At least that’s my hypothesis.

In the end, the two groups are reconciled. They become so tight that they start to blur the lines between the two groups. Rather than avoid each other, now they insist on including each other in activities. They even go so far as to sacrifice things so that they can include others from the previous “enemy” team. It’s a dramatic example of breaking down silos.

Dueling Silos

April 29, 2011

In Forming Silos, I reviewed the Robbers Cave Experiment and how they tried to create silos as part of their research on in-group and out-group conflict. In this post, I want to review stage two of the experiment, where the intent is to create conflict between the two newly created silos.

On the face of it, creating tension between two groups is so easy it’s almost embarrassing. All the researchers did was create a series of winner take all challenges for the two groups to compete in. Then, to aggravate the situation, they subtly manipulated the scoring so that each side would cheated out of a win. Of course, back in the 1950’s this was a relatively novel idea. Now we have reality TV and we call it ‘Survivor’. The difference the Robbers Cave experiment and ‘Survivor’ was that in the case of Robbers Cave, nobody is voted off the island.

The challenges were the sorts of games and other activities that you would commonly find in summer camp: baseball games, football games, tent pitching competitions, tug o’ War, cabin inspections, skits and songs, treasure hunts, and so forth. The emphasis was on keeping the competition as realistic as possible. I think the importance of keeping the setting as natural as possible was critical to the success of the experiment. They did not want the boys to feel like they were being manipulated or observed by the camp counselors/researchers (even though that is exactly what was happening). So rather than setting up artificial challenges, the researchers went to great lengths to make sure that the challenges were appropriate to the setting. In fact, the researchers were so concerned about this that they repeated the study on at least two different previous occasions before getting it right on the third try. That sort of persistence is particularly impressive, given the enormous investment in time and resources required to put together a large experiment like this.

So we have the two groups competing against one another for scarce resources (various and sundry trophies, flags and other rewards). In very short order the two groups not only hate each other, but they are getting into name calling and fist fights and refusing to go anywhere near each other. They go so far as to raid each other’s cabins, wreaking havoc, and stealing things from ‘the enemy’ team. By the end of this phase, we have two groups that well and truly hate each other. They exhibit all of the territoriality and biases that characterize a pair of badly dysfunctional silos. Furthermore, they manage to accomplish this in less than a week. Apparently, it does not take very long to create a dysfunctional group.

As in the previous stage of the experiment, the researchers had a few hypothesis about what might happen in stage two when the competition was introduced. Their first theory was that competition would increase the in-group solidarity of the two teams. This makes sense, when we are competing against some outside enemy, you would expect it to tighten the interpersonal bonds within your own group. In fact that’s exactly what happened in this case. The teams became more tightly knit in the face of competition from another group. To use a cliché, each group closed ranks in the face of danger.

Another hypothesis was that the functional relationships between the two groups would affect the relationships within each individual group. This is a little hard to sort out, but as I understand it, what this means is that if things are going poorly for your group in the competition between your two teams, it is very likely that there may be a change in leadership for your group. I’ve seen this happen before in competing teams. The tension and the friction that builds up between the two groups eventually leads to one group looking, at least temporarily, like the “winner” and the losing group may reorganize itself in the face of their own perceived leadership failure. I do not think this necessarily alleviates the problem in any manifest way, but it does follow that the intergroup relations reflect on the relations of people within each group.

The final hypothesis is the one that I find the most worrisome of the bunch. Basically, in a group conflict, the theory is that low status members of either group will be more likely to act out violently in deed and word than higher status members of the group in order to to improve their own status within the group. Therefore, not only do you have the inter-group conflict going on, but you also have people within each group who are trying to take advantage of that conflict in order to advance their own status within each group. How do they try to change their status? By being the loudest voices to demonize the other group. They do accomplish their ends by actively provoking conflict between the two groups. This of course just serves to further aggravate the tensions already present between the two groups. At this point, the conflict has becomes self-perpetuating.

If we stopped the experiment here, I think that there are many people who might just say, “You see? I told you so. People are just nasty and brutish.” There certainly were those who only seemed capable of reading to this point in the study and deciding that it confirmed all of their worst stereotypes of human behavior. However, the experiment doesn’t not stop here. This is still the setup for what is perhaps the most interesting phase of the experiment – the stage where they introduce the strategies they used to reconcile the two groups. The story of how they managed to re-integrate the two groups and achieve real collaboration is truly remarkable.

Forming Silos

April 26, 2011

As part of my research for our Silo Busting tutorial at XP2010, I’m reading “The Robbers Cave Experiment: Intergroup conflict and cooperation” by Muzafer Sherif et. al. I first heard about this experiment from Linda Rising (one of my all time favorite speakers and writers) who used it as the topic for a great presentation that she gave at the Agile2008 conference. Her presentation made a big impression on me, so much so that I found myself ordering the book about the study. The Robbers Cave Experiment is a classic experiment in social psychology from the 1950s that has profound implications for the way that organizations work together today.

[One tiny little caveat here: this is a social psychology study from the fifties. At the time, psychology was, and some would say still is, struggling to be taken seriously as a science. As a result, in general the published papers are God-awful dry and boring. I mean make-a-grown-man-cry-for-mercy boring. That way they seem more scientific! You have been warned.]

The purpose of the experiment was to explore how social groups form, how they come into conflict, and to experiment with means of resolving inter-group conflict. The subjects of this experiment were two groups of 12-year-old boys who were going to a summer camp at the Robbers Cave State park in Oklahoma. This study took place in the late forties and early 1950s, back in the day when there was a lot more latitude with selecting and experimenting with human subjects.

[OK, another digression: researchers got to do the coolest stuff to people in the late forties and early fifties! They got away with all sorts of crazy experiments back then (see Zimbardo’s Prisoner experiment). Ah the good old days…we can’t torture people in experimental psychology the way we used to. Amateur hour is over. Now we leave torture to the professionals: the military.]

The boys (or subjects – see how scientific that sounds?) were carefully screened for selection for this summer camp. They had to pass a battery of psychological tests and meet specific criteria in order to take part in the experiment. The goal was to select from a population that didn’t have a background of disturbed family histories, large differences in social background or other dramatic differences that might cause confounds in the experimental design.

The first phase of the experiment was an exercise in-group formation. The researchers needed to create some silos in order to test their hypothesis about breaking them down. The boys were taken to campsites and proceeded to play games, go exploring, and generally go about the process of forming, storming, and norming that all teams go through – even teams of 12-year-old boys.

There are some interesting hypotheses that the researchers had about this first phase of the experiment:

  1. That hierarchies will form
  2. That your place in the hierarchy affects your own assessment of your own performance as well as that of others
  3. That members of groups will adopt the “norms” of the group and doggedly stick to those norms in the face of conflicting evidence.

I find these notions very intriguing to us as Agile practitioners. First, I think at its heart many of the Agile methods are rooted in egalitarian notions of communal leadership and are fairly antithetical to the idea of command and control. So, it seems to me that hierarchies, at least the way that I’m used to thinking of them, are generally considered a bad thing from an Agile perspective. This experiment theorizes that given our natural inclinations, the hierarchy is the default organizational structure for people (well, for 12 year old boys anyway). The results support this theory. My gut reaction: that is a major bummer.

Maybe not all is lost though. Perhaps the hierarchy is a default place to start absent any other influences, but evolution can take place. Perhaps it is evolution toward a more communal, collaborative style of group? I don’t know. I’m certainly not an expert in this field, but I find it fascinating and somewhat frustrating that hierarchy seems to be the default choice. Of course, when talking about silos, it’s hard not to refer to hierarchies. They seem to go hand in hand.

The next theory was that your place in the hierarchy would affect how you perceive your own performance and the performance of others. It turns out that we tend to overestimate the performance of those who are higher than us in the hierarchy and to underestimate the performance of people who are lower than us in the hierarchy. So does this imply that we tend to think that the boss is a genius and that the people who work for us are idiots? Ouch! Sherif and his researchers tested this and found that indeed, we do tend to overestimate the abilities of those higher up in the pecking order and underestimate the abilities of those beneath us. Keep that in mind the next time you are talking to the boss!

Finally, the members of the group came to “normalize” their assessments of conditions to match those of the group they were in. So independently, you might tell me that you prefer green, but if the group prefers blue, then guess what? You are going to start reporting that you prefer blue too. It’s all part of fitting into the group. One interesting observation was that members of a group frequently reported themselves as “working harder” than outside groups – even when there was no evidence to support this claim. I’ve certainly seen plenty of that when working with high tech groups and teams.

The rest of the study is equally, if not even more fascinating in its theories and its conclusions. This research, whether or not you agree with it, has some profound things to say about the way that human beings work in teams – and the dramatic effect teams have on our individual judgement. I found many parallels in the study with the teams that I have worked with (agile or not). It’s dry, academic stuff, but if you are at all interested in the way that teams form, fight and resolve, it is pure gold.

Things That Divide Us

January 2, 2011

Organizational silos are the source of the most pernicious dysfunctions you can find within any company. What is a silo? Silos are the walls or barriers that we erect in order to separate “us” from “them.”

We are the ultimate corporate reductionists. We divide everyone in the organization down into the most specialized roles that we can tolerate and then we struggle to produce a product using the result. That division ends up reflected in everything that we do, from the products that we produce to the way that we hire new people to help us.

We break things down in so many ways that it boggles the imagination. For example:

  1. Management Responsibility: Executives, Managers, Workers
  2. Roles in the Product Development Process: Sales, Marketing, Development, Architecture, Project Management, QA, Operations, Customer Service
  3. Parts of the Application: UI, Middleware, DB
  4. Locations: Headquarters, satellite offices, international
  5. Languages: C++, Java, Ruby, English, French
  6. Processes: RUP, Agile, Lean

This is just a small sampling of some of the ways in which we divide ourselves within organizations. These divisions serve to isolate people in the organization within hyper-specialized roles. Ostensibly, we do this in order to help people succeed. The Justification might be that no one can be equally good at everything. Therefore, we compartmentalize our lives and those around us in order to filter out the extraneous noise. We try to create a space for focus and success. Ultimately, it is all an effort to help us manage the scope of the learning that needs to take place. All of these goals are necessary and helpful and they are things that come with a price.

Some of the costs of all of this division and compartmentalization are:

  1. Lost knowledge of upstream and downstream processes
  2. Lacking a holistic understanding of the product
  3. A narrow view of the people involved in product development
  4. Often little or no knowledge of the business domain itself

Of course, it does not have to be this way. You can deliver a product successfully without compartmentalizing everyone and everything in an organization within an inch of its life. It requires a different mindset. One needs inter-disciplinary thinking that considers different skills and tries to synthesize a whole rather than divide. This requires a mindset that favors skill over roles, knowledge over assignment.

This focus needs to extend through the entire human dimension: from the self, to the team, and all the way through the organization. In terms of the self, we need to be well-rounded product developers: people who appreciate the logic, art, science, and beauty of our craft and our product. As teams, we need to have the proper balance of skills, from development, QA, the customer, and delivery. Moreover, as an organization, we need to have the people in place to help support the teams and the people on the teams to develop themselves and deliver the best products.

Once we can do that, once we can see ourselves as more than cogs in a machine, once we can collaborate to craft beautiful things, and once the organization can appreciate the beauty of not only the products, but the people who create them, then we can move away from these silos that handicap our organizations now.