Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Nobody Expects the Agile Imposition (Part IX): Culture

by eyepopslikeamosquito (Chancellor)
on Jun 20, 2015 at 07:40 UTC ( #1131244=perlmeditation: print w/replies, xml ) Need Help??

In 2008, we were pretty much a Scrum company ... however, a few years later, we had grown into a bunch of teams and we found that some of the standard Scrum practices were actually getting in the way. Rules are a good start, then break them when needed. We decided that Agile matters more than Scrum.

Autonomy means the squad decides what to build, how to build it, and how to work together while doing it. One consequence of autonomy is that we have very little standardization. When people ask things like which code editor do you use or how do you plan, the answer is mostly depends on which squad. Some do Scrum sprints, others do Kanban ... it's really up to each squad. Instead of formal standards we have a strong culture of cross-pollination; when enough squads use a specific practice or tool, such as git, that becomes the path of least resistance, and other squads tend to pick the same tool.

Why is autonomy so important? Well, because it's motivating and motivated people build better stuff.

-- from Spotify Engineering Culture (Part 1) by Henrik Kniberg (0:30-4:40)

Instead of blindly following Scrum dogma, I advise you to analyse the problems you and your company face daily. Reason about them. Consider applying Agile and Lean principles to them. Experiment to see what works for you and what doesn't.

After feeling isolated and alone in resisting the Scrum imposition, reading Spotify's story has cheered me up. I've become more hopeful that things will change, that over time more folks will come to see the benefits of greater team autonomy.

Autonomy and Alignment

It's kind of like a jazz band, although each musician is autonomous and plays his own instrument, they listen to each other and focus on the whole song together. That's how great music is created. So our goal is loosely coupled but tightly aligned squads.

Down here is low alignment and low autonomy, a micromanagement culture, no high level purpose, just shut up and follow orders. High alignment and high autonomy means leaders focus on what problem to solve but let the teams figure out how to solve it. Alignment enables autonomy.

-- from Spotify Engineering Culture (Part 1) by Henrik Kniberg (3:20-3:50)

For high autonomy to work, you need high alignment. With low alignment, teams simply do whatever they want, with each team going off in a different direction.

Specialists vs Generalists

Each system is owned by one squad. But we have an internal open source model and our culture is more about sharing than owning. Suppose squad one here needs something done in system B and squad two knows that code best, they'll typically ask squad two to do it. However, if squad two doesn't have time, then squad one doesn't necessarily need to wait. Instead they're welcome to go ahead and edit the code themselves and then ask squad two to review the changes. So anyone can edit code but we have a culture of peer code review. This improves quality and spreads knowledge. Over time we've evolved design guidelines, code standards, and other things to reduce engineering friction, but only when badly needed.

-- from Spotify Engineering Culture (Part 1) by Henrik Kniberg (5:10-6:00)

The basic unit of development is the squad. Because each squad sticks with one mission and one part of the product for a long time, they can really become experts in that area. A tribe is a collection of squads that work in related areas. The chapter is your small family of people having similar skills and working within the same general competency area (e.g. QA or Web Development), within the same tribe. As a squad member, my chapter lead is my formal line manager, a servant leader, focusing on coaching and mentoring me as an engineer, so I can switch squads without getting a new manager. A guild is a more organic and wide-reaching "community of interest", a group of people that want to share knowledge, tools, code, and practices. Chapters are always local to a tribe, while a guild usually cuts across the whole organization. Some examples are: the web technology guild, the tester guild, the agile coach guild. Anyone can join or leave a guild at any time. Most organizational charts are an illusion, so our main focus is community rather than hierarchical structure.

-- from Scaling Agile @ Spotify (with Tribes, Squads, Chapters and Guilds) (pdf) by Henrik Kniberg & Anders Ivarsson and Spotify Engineering Culture (Part 1) by Henrik Kniberg (7:30-8:50)

Alistair Cockburn (one of the founding fathers of agile software development) visited Spotify and said "Nice - I've been looking for someone to implement this matrix format since 1992 :) so it is really welcome to see"

-- from Scaling Agile @ Spotify (with Tribes, Squads, Chapters and Guilds) (pdf) by Henrik Kniberg & Anders Ivarsson

In learning more about Spotify, I was also glad to learn how they deal with specialization. You see, this tricky topic has been a chronic nuisance for us, for a number of reasons.

First, code quality. New code written by a generalist, and reviewed by a generalist, is a long term code quality nightmare. Can you imagine what Perl code written by a ten-year Java veteran with a few days of Perl experience looks like? I can because I've seen it. There have been times when I've opened up a Perl file and rolled on the floor laughing because it was clear that whoever wrote it had no understanding of Perl at all.

Second, employee engagement. Though many programmers are happy to become generalists, a significant minority (including me) are deeply dissatisfied with that role. I derive much more job satisfaction from doing an expert job in something I deeply understand than from "cargo-culting" some code that seems to kinda work even though I lack a deep understanding of why. I find that dissatisfying. I do not want to write Java code that causes my Java expert colleague to roll on the floor laughing.

Another problem is that system architecture can be compromised if nobody focuses on the integrity of the system as a whole. To mitigate this risk, Spotify have a "System Owner" role. All systems have a system owner, or a pair of system owners (one with a developer perspective and one with an operations perspective is common). They further have a chief architect role, someone who coordinates work on high-level architectural issues that cut across multiple systems.

Healthy Culture Heals Broken Process

Trust is more important than control. Agile at scale requires trust at scale. And that means no politics. It also means no fear. Fear doesn't just kill trust, it kills innovation because if failure gets punished people won't dare try new things.

-- from Spotify Engineering Culture (Part 1) by Henrik Kniberg (12:40-13:00)

We focus on motivation, community and trust rather than structure and control. Healthy Culture Heals Broken Process.

-- from Spotify Engineering Culture (Part 2) by Henrik Kniberg (0:40-0:50, 12:00-12:30)

I've seen first hand how a healthy organisational culture of good communication and continuous improvement can effectively solve process problems.

I've also seen first hand how a culture of fear and an over-emphasis on control and structure can harm innovation.

Lean Startup

We aim to make mistakes faster than anyone else. Continuous improvement, driven from below and supported from above. Failure must be non-lethal and with a "limited blast radius". Lean startup principles: think it, build it, ship it, tweak it. The biggest risk is building the wrong thing. Release first to a small percentage of users, then use A/B testing, then gradually roll out to the rest of the world. Impact is more important than velocity. Innovation more important than predictability.

-- from Spotify Engineering Culture (Part 2) by Henrik Kniberg (2:00-)

These are some of the ideas used by Spotify to build and release product. Since there is enough in this installment already, I'll postpone a discussion of Lean startup and related ideas to a new series of articles.

Related PM Nodes

External References

  • Comment on Nobody Expects the Agile Imposition (Part IX): Culture

Replies are listed 'Best First'.
Re: Nobody Expects the Agile Imposition (Part IX): Culture
by mr_mischief (Monsignor) on Jun 22, 2015 at 06:33 UTC

    When people find the process is in the way of the progress, it's usually because they are taking the process too seriously. Scrum, Kanban, Lean, XP, and whatever else your consultant is selling are ideas and guidelines. Any good idea taken to extremes or sensible guideline followed blindly becomes a bad idea or a nonsensical guideline.

    All the above being true doesn't mean there's no value in those systems. It means they are not magic bullets. They don't work for every type of project. They don't work for every team. Applying the ideas from them judiciously can be helpful if you really need small teams to tend code with quick turnaround for iterative improvements.

Re: Nobody Expects the Agile Imposition (Part IX): Culture
by Your Mother (Bishop) on Jun 20, 2015 at 19:29 UTC

    First quote strikes me as very Adam Smith.

Re: Nobody Expects the Agile Imposition (Part IX): Culture
by sundialsvc4 (Abbot) on Jun 22, 2015 at 14:19 UTC

    It is frankly of great amusement to me, to listen to computer programmers carrying-on in this way:   debating whether they wish to be “generalists” or “specialists,” and blatantly self-justifying to one another the notion that a team should be entitled to make everything up as they go along.   And, that the project managers and everyone else in the company, whether above or below or alongside them, should be content to sit back and watch the blinking lights.

    Hogwash.   Programming projects, like any project, must be “managed.”   Someone(s) must be coordinating and tracking the progress of the many specialists, and the work that is being done must ... in fact ... be meticulously planned in advance.   Because:   programming projects are like any other project, and, unlike every other project.

    The one most-influential book I’ve yet read, Managing the Mechanism (Vincent North, ISBN 978-0-715-74383-7) is now available both for iBook and for Kindle, but not in print.   The key premise is that “software is a machine,” of terrible complexity, but also that it is “a self-directing machine.”   In the end, it will execute precisely the instructions of which it consists, and its behavior will either be correct (1) or not (0).   This is not a floating-point number.   There are no shades of gray.

    This is not a Jazz concert.   Neither is there any room for sports-analogies of any kind.   No Rugby team ever walked off the field, pressing a “Start” button as they passed by to cause an automatic mechanism to play the entire game by itself.   Culture, and inculturation, also have nothing to do with it:   if the machine fails (again ...) in mid-air, who cares whether the people who (obviously, failed to ...) build it “liked each other?”   Who cares if the wrong set of instructions were built ambidextrously?

    At the end of the day, the file of digital instructions stands alone, and nothing else matters.   If the instructions are truly complete and correct, then the mechanism is failure-proof because it cannot be otherwise.   Otherwise, the machine is set to fail the instant a particular pathway of instructions is taken, and again, it cannot do otherwise.   The project, first of all, must indeed be managed.   Secondly, it must be managed in the way that mechanism demands.

    It’s an observation that pretty much blows both “Scrum” and “Agile” cleanly out of the water.   The process that we put ourselves through as human beings, no matter what it is and no matter how uncomfortable, really does not matter to the machine that some innocent person will get into and attempt to ride (not drive ...) to the Moon.

      I find it amusing that someone with your background and experience still pretends to believe that all things always fit into your One True Cookie Cutter.

      Each environment, product, customer set, expectation set, and needs are different. Each requires a fresh look at how the approach should be handled.

      If your goal is to be the first to market to quickly fill a niche while there's a pocket of money to be made on it, you don't have time for process.

      They can always sell the company for a profit later to someone who will hire you to clean up the mess. But they're in it to fill a niche, make a buck, and move on.

      I hate that mentality, but that's reality, and I've learned to give it room to be what it is.

      There are many parallels we can draw in real life to support your All-Managed-All-The-Time view as well as the Never-Managed-Still-Got-A-Product-Out-The-Door view.

      Your view is valid for lots of stuff, I agree. But zero flexibility means nothing particularly new gets done without luck. Yes, that same luck that you drummed out of the picture.

      There's a time and a place for every kind of solution. If you truly cannot see that, then I may have to re-evaluate a few of my former assessments.

      A perfect half-boat sinks. A crappy full-boat can get you places. It might not be fun to operate. Heck, it might even be unsafe at times. But it's better than that really pretty full-featured if-you-ever-finish-it-gosh-it-will-be-great half boat sitting on the hard.

      Flexibility, man. Surely you see some value in it.

      Don't you?

        There are plenty of companies out there who ... (miracle of miracles!) ... manage to successfully produce all sorts of automated mechanisms that don’t fail to do whatever-it-is that they are supposed to do.   The bread you ate, the tin-full of tasty rolled-up cookies, the many miles of highway that you traversed.   Processes such as these certainly can be software-controlled, and those processes can be perfected in order to produce an outcome is consistently in-gauge.

        What I think we fail to realize, though, is that every piece of software is “a mechanism” unto itself, whether or not there is a piece-of-machinery involved (such as a car, a pastry-machine, or a paver).   In the use-cases aforesaid, there is always (a) a fixed outcome, and (b) a human being at the controls, who has the sovereign “last clear chance” to recognize that a problem has developed and to (at his/her human discretion) exercise corrective actions that the software in question merely has to facilitate.

        And so, here is the key difference in what we are doing:   there is no human-being in the loop.   In the case of “corrective actions,” the software must not only decide, but do.   Alone.   Unattended.

        If your goal is to be the first to market to quickly fill a niche while there's a pocket of money to be made on it, you don't have time for process.
        Why the hell not?   “Hurry up and build the apartment building so that we can hurry up and fill it with tenants, who cares if it subsequently catches fire?”   Of course you see my point.

        They can always sell the company for a profit later to someone who will hire you to clean up the mess.   But they’re in it to fill a niche, make a buck, and move on.
        Obviously, a point-of-view that would never pass muster with the legislative bodies in your domicile which license and regulate General Contractors of various sorts, with regard to physical buildings, physical machines (electrical and plumbing systems, etc. etc.).

        Personally, I think that it is well past time for anyone in our profession to assume that we can actually “build mission-critical mechanisms, which have direct or indirect impact on human lives and/or quality-of-life, and assume that we are somehow exempted from the strictures observed by those whose work-products can actually “lop-off a human arm.”   Our work-products certainly can do comparable things, and I very candidly believe that our days as “a completely un-licensed profession, not perceived to be In The Public Interest,” are:   Numbered.

        So far, we are blissfully behaving as though we were not “building things that can do Real Damage™ to The Public Interest.™”   I think that those days are Numbered, and that we had damn-well be Planning Accordingly!

      This blockbuster brought to you by the man that after 10 years of planning still hasn't managed to produce a single line of working code.

      "`“Caveat lector” ™ © '®"

        What a breathtaking extrapolation of ... nevermind.

        ROTFLMAO...   Thanks.   You made my day.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://1131244]
Front-paged by BrowserUk
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (5)
As of 2018-06-23 00:31 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (124 votes). Check out past polls.