Engineering Excellence: Declan Whelan on Technical Health, Agile Practices, and Team Culture

Transcript

Shane Hastie: Good day, folks. This is Shane Hastie for the InfoQ Engineering Culture podcast. Today, I have the privilege of sitting down with Declan Whelan. Declan, welcome. Thanks for taking the time to talk to us today.

Declan Whelan: Pleasure to be here, Shane, so nice to see and hear your voice again, and thank you for having me.

Shane Hastie: My pleasure. It's been a while. You and I obviously know each other, but I suspect a fair bunch of our audience haven't come across you and your work, so who's Declan?

Introductions [01:09]

Declan Whelan: Fair, good question. I live in Canada and I'm an electrical engineer by training and I've been involved in software my entire career and I've opted for a career path that always kept me close to code, bits under my fingernails. Along that journey, I've been a coach, I've been a individual participator, or contributor, and I've been a CTO of two different start-ups and just love doing technical work, and especially in an agile context. I still do a lot of training and some coaching and a lot of coding whenever I can.

Shane Hastie: One of the things that you've been known for over the years, it must be a decade at least, that you and I have known each other and you've been in that whole time, you've been strong on strong technical practices in teams and talking a lot about technical health as opposed to technical debt. What does technical health look like?

Technical Health over Technical Debt [02:12]

Declan Whelan: It's a good question. Well, let me start with technical debt, because as you know, and I suspect a lot of the listeners would know, that was a term invented by Ward Cunningham back in, I'm going to say the late 1990s. It was really around a mechanism to explain how if you take shortcuts in your software development in the short-term, you may get some immediate benefit such as you would for taking out a loan, you would have some cash in hand for which you could purchase something. If you took a bank loan, you might be able to buy a car or a house.

In technical terms, you might be able to deliver more features out to customers by taking shortcuts. Ward drew the metaphor, a financial metaphor for that action with your technical practices, you can choose to take shortcuts and you will have to pay back the reaper at some point in time.

I found that metaphor to be really strong, but also as any metaphor or any analogy has, it has its shortcomings. I initially wanted to start to have people think about technical debt in a more positive light because when we think about debt, it has a negative connotation to it. You're in debt, I'm in debt, and being out of debt is a good thing. We know from technical work, you're never out of debt. There's always some work to be done. I decided that having a more positive term would be better. Then I started to realize, well, technical, so I invented the term technical health as really just a different way of reframing technical debt.

Since I've used the term, I found some unanticipated benefits to that term, and one of them is that you can think about, let's say your own personal health. If you are just participating in the 2024 Olympics, if you're a world-class athlete, your notion of what is healthy is going to be very different than say yours or mine, where maybe we're just looking to stay fit and have our blood pressure at a certain level and so on.

The point being that different systems or different contexts can have different things that are applicable and important to them. I've been using the term technical health as a way to expand people's thinking about their technical practices and what's important to them in order for them to achieve the business outcomes that they want. Then recognizing that we all have health, it's just regardless. You can choose to have debt or not, but we all have health, and I think that's what happens when we build systems.

There's always going to be some work that you wish you had done differently or would choose to do better now that you've done it once, you know how to do it better. That's the technical health term. I wanted it to be more holistic and more relatable so that people could actually take more positive action with it. That was how I came up with the term and that's where it led me. Does that make sense?

Shane Hastie: It does. Technical health is every system will have technical health and it is a result of the environment. You said result of the system they're in. How do we improve that system to make, or should we improve that system to make the technical health of our products better?

Use metrics to help improve the organisational system to improve technical health [05:31]

Declan Whelan: Yes, great question. One of my more recent clients asked a similar question, how do we know that our technical practices are actually effective? A lot of companies would look at things like static code analysis and look at their cyclomatic complexity or code coverage and things like that, and those are good, but they're not customer focused. Those things don't matter to customers. What I think has shifted, and I think that I would credit the DevOps movement with a lot of that is really starting to think about things as they touch customers. Teams now are responsible for delivering to customers. The shift for me in terms of what you might do with me would first of all, to measure it. I think right now the metrics that I see that are most widely used and are available would be the DORA metrics, and they happen to be really useful.

In case anyone doesn't know, those would be deployment frequency, lead time for change, change failure rate, and meantime to recovery. Now, those just happen to be four metrics you could choose a whole slew of other ones, but those four in particular have strength because of the DevOps' work that has been done and these four metrics have bubbled out as being key differentiators for organizations that are delivering well. If you were to stick with DORA metrics, for example, and looping back to your question, how would you go about improving your technical health? It could be starting to measure these and decide, "Oh, wait, do we want to increase our deployment frequency? What's standing in the way of deployment frequency? What's slowing us down?"

Technical debt then just becomes the things in the code that slow you down and the technical health is more expansive. It's about how well is our system working. When I'm asked that question that you asked, which is really a good, one is would be yes, start to measure. If you're not measuring DORA metrics, DORA metrics to me are really a good indicator of how well you're building things technically.

Then beyond that, I've become more recently interested in value streams and having overall flow metrics. For example, in SAFe® now they have six flow metrics and I don't have a lot of experience with those. I'm now just starting to read a book called Flow Engineering from Steve Pereira and another author. It's really expanding around the DORA metrics to really focus more on the overall flow.

For example, in DORA metrics the lead time for change is from the time you commit code to when it goes into production, doesn't include the time for business analysis and product requirements understanding. An even better metric would be the overall cycle time or lead time, whatever works for you, but back earlier to when it actually comes from customers and touches customers, so the full cycle. Other times I've worked with companies where they sometimes would be all focused on their code coverage or something, and those are good, but they're all internal. What's really important is how frequently are you able to ship value to customers and focus on that, basically.

Shane Hastie: That shift from the internal metrics like cyclomatic complexity to the customer focus metrics more like DORA and then even thinking the end to end. Haven't we talked about shift lift and shift right for a long time?

Challenges in Shifting Left and Right [08:51]

Declan Whelan: That's true. We still need to do it. I think some of the leading companies are certainly doing well in that regard, but most companies I encounter really struggle with that shift left and shift right. Those are both big shifts to have. I still see, I don't know if you see it, but I still see a lot of separation, for example, between QA and development. Some organizations, maybe even though they might have those members on their teams, they might report differently, they're measured differently. I've even worked with teams where they say they're a team, but the QA is working in a different repo than the rest of the team. It's really, you may as well call them a team in name only because of the separation. Yes, it is a shift left, but I think that's still a really challenging shift for many organizations, even though they're trying to do that, they still struggle. At least that's what I see. Do you see differently?

Shane Hastie: I do see it and I just wonder why. What is it in our systems that is making that shift left, that true value stream, the true team, what's making it so hard?

Declan Whelan: I can only report on what I've seen, but where I've seen it, one thing is definitely, I worked with a company who was 150 years old, and I worked with them about 10 years ago. Then I was working with the development team and I was interested in talking to their testers, and I had to make an appointment to meet their, and as soon as I met the testers, I was asked for which project code are we going to bill this meeting to? I was refused, given a lot of passive aggressive resistance to talking to the testers, I guess, and it was because they reported differently up the organization. Fast forward, say seven years, and I went back, now that's no longer true. I no longer need a project code to have a conversation with a tester, but they still report differently.

They're still managing different repos, even though it's been seven years of them working towards teams, they're still not there. They physically are sitting on the same team in an org chart, but they're still working on completely separate streams of work. If you want to shift left, you really need your QA and your devs working collaboratively together. Your QA team can't do all the heavy lifting on its own. I see organizational barriers between QA and development still staying in the way, and I see that in banking and telecom, insurance, more traditional regulated environments where they're used to being separated. I would say many of these companies would still have another decade or more to solve that problem.

Shane Hastie: Can they afford to wait a decade?

Declan Whelan: Well, in Canada, all of those types of companies do quite well. They don't suffer. They're probably more profitable than they've ever been, in general. Maybe that's it, they could afford to do it. Shift right, I don't have a lot of experience with shifting right, which is basically being able to chaos monkey things in production sort of idea. I would say, yes, until you can shift left, I don't think you're ready to shift right. Until you can reliably design, build, release, yes, it'd be awfully difficult to get into putting in appropriate safeguards to shift right in production. I still think we're a ways from that in many places that I've worked, for sure.

Shane Hastie: If we go back to the beginnings of some of these practices, extreme program. Kent Beck gave us some great ideas in 1999, 2000. There was a period when XP was held up as this is the way to build software, and it seemed to then fade away. What's happening with those practices now?

Core Technical Practices from eXtreme Programming [12:36]

Declan Whelan: I grew up like you did, I think I got exposed to Agile through XP, so extreme programming and the technical practices in particular that it brought were very different than Scrum. I would say in my view, Scrum, because it specifically hasn't had an opinion on technical practices, in fact, it's the world of work, it's not just software development. There was no guidance from Scrum or Scrum.org or any Scrum material around technical work. What I think has happened, and by the way, yes, so SAFe® and other frameworks, they'll usually have something about technical and they'll almost always include extreme programming as the technical practices. Where I've worked in SAFe®, I've worked in four different SAFe® organizations, I've never seen the extreme programming technical practices used, although they're actually part of SAFe®. I think with Scrum, it doesn't matter. They had no opinion.

With SAFe®, even though it does have an opinion, what I've seen is it's so low in the order of things that they should take care of, there's so much to save. It's just another line item of things that you may or may not do or could not do. I've just seen just companies get overwhelmed with SAFe® so much that they actually don't get to the technical practices. I was at the Agile 2024 conference in Dallas, and I had exactly the same question that I had you had. I was like, "Where is Agile going and where is extreme programming in particular going? Because I like to work in extreme programming. It's the way I choose to work." I think one of the silver linings on what I would say is the downturn on Agile, which is the people feeling Agile has perhaps run its course.

We have certainly in late majority type of framework situations with Agile now. One of the good things is that as people have seen the challenges with it, and a lot of them are because I feel that some of the challenges Agile has had is because the lack of technical practices have led companies to be accruing technical debt at a rate that just overwhelms them at some point. Certainly, the more traditional companies I've seen, they just struggle so hard to be able to put out something every week or two is an extremely stressful situation, and they haven't shifted their technical practices to enable them to deliver at a more rapid rate. I think in that way, I feel like the lack of engineering practice has been a contributor to what is perceived as the downfall or certainly the perception of a downfall in Agile.

One thing that really made me feel better when I was in Dallas was, I went to a session on FAST, which is the fluid scaling technology. There, it was really interesting, but one of the things that they are really quite strong about is really being opinionated about using extreme programming practices as the core practices. I think there's an opportunity for a resurgence in some of the technical practices precisely because some of the traditional Agile approaches haven't panned out as well as people had hoped. I'm somewhat optimistic about the technical practices right now, so we'll see. Maybe I'm just a wishful thinker.

Shane Hastie: What are the good technical practices that if I'm setting up an engineering team, this is the way we do things? What does that solid technical core look like today?

Modern Technical Practices Beyond XP [16:04]

Declan Whelan: Well, certainly it's shifted since Kent Beck came up with extreme program, but if we start back in that day, there are really four core technical practices at extreme programming and pair programming, which now I would probably throw ensemble or mob programming in as an extension to that test driven development, always been a sort of controversial practice. Refactoring, I would say, has been entrenched in the industry, even though people may not be doing it, everyone would know what it is and everyone does it to some extent, and the last one will be simple design, the four rules of simple design.

That is what I usually, if I'm coaching or training, those would be things that I would be introducing to teams. Those are the core, if you're doing Scrum and you were to add those four practices, I think you would be in really good shape. What it doesn't include in that is some of the more modern practices around DevOps and being able to get to continuous delivery and having really good observability in your production systems is now going above and beyond what extreme programming had in mind.

We're into things like feature flags and things like that to achieve it. I would say the big shift from say extreme programming has been this notion of continuous delivery. The other part of it, which I'm only becoming more recently exposed to, is this notion of real stewardship of the services that we build.

Instead of having extreme programming, which was really focused on building what we're building the here and now, but really thinking about how do I build my services so that they're maintainable indefinitely into the future? What do I need to be building in so that others that follow can maintain them as well as we can? That's a powerful notion. I think that's really coming from team topologies and that notion that we're now building a more service-oriented systems than we were in the past, and stewardship, I think is a really nice addition to the way of thinking about technical practices.

Shane Hastie: That's some of the architectural shifts, the pendulum that seems to swing back and forth - from large monolith to microservice to too many microservices to distributed monoliths.

Distributed Monoliths [18:19]

Declan Whelan: Most of the companies I've worked with have really gone down the microservices or service-oriented approaches, for sure. I think one of the problems with service-oriented architectures that I've seen is that usually people will design their service boundaries pretty early. You have to, if you're going to spin up these services. You have to make decisions about what is their domain and what do they do.

It turns out with service-oriented systems like that, changing those service boundaries is going to be where you're going to have a lot of friction because whoever's using your service, those changes would potentially need to be coordinated to roll those out. I think architectures that allow you to defer your service boundaries or encapsulate your service boundaries perhaps within a monolith, seem to me to be good approaches. I'm not an expert.

One technology I'm always drawn to for this is Elixir and Erlang because they were built so that when you build something, it's not just a single service, but it's actually a set of cooperating services, making it much easier to roll out changes that might touch multiple parts of your system. Outside of say, Elixir or something like that which handles that internally would be moving towards mono repos where even though I'm changing multiple services, I just have one mono repo, and so when I make my change, I can actually change and eventually, deploy systems together or close together in time.

A mono repo with a single monolith which might have different services is certainly a direction that some companies are going. I've not actually worked, well, I've worked with one client who was just in the middle of doing a mono repo, so I haven't actually seen how well they work in the wild, but I can see the value in it for companies struggling with microservice boundaries.

Shane Hastie: This is the Engineering Culture podcast, we've been talking a lot about practices in here. What are the cultural influences that allow us to get into good technical practices? What is a good team?

Team Cultures that Enhance Technical Excellence [20:28]

Declan Whelan: For a good team, I'm not sure that technical practices would be different, or if I have a good team, it wouldn't matter whether they were doing technical work or not, I guess is what I'm trying to say. It would be about certainly the collective, having an idea that we're in this together and the extreme programming had the idea of whole code ownership. It's our team that owns this code, so having that idea of we're in this together, I really liked the idea of stewardship from, I think I mentioned it already from Matthew Skelton and the team topologies, that idea that we need to take care of this not just now, but for ourselves in the future. We don't want to be woken up at 3:00 AM with beeper calls if we can avoid it, so that idea of high-quality work. I love the word stewardship, really taking care of the products that we're releasing and recognizing that we're not just building for now, but we're building for the future as well.

Your horizon might differ, it could be three months or a year or whatever, but we need to be cognizant of not accruing too much technical debt by taking care of our future. Certainly, I've always been on a culture perspective, this sort of learning organization ideas from, just rereading the fifth discipline actually, and that idea that we're always, especially in our field in tech, things are always changing, so we need to be in a position and a culture of continuous learning.

That's one thing that I've always been attracted to the agile extreme programming practices, because they really focus on using pair program and ensemble programming to share and learn together as a team. The rising tide raises all boats. We're all going to learn together and that we learn from each other. I may know some keyboard shortcuts that you don't know, Shane, and Shane, you might possibly know a little bit more about business analysis than I do.

We could learn a lot by working together on things, so this team culture of true collaboration. What I see in a lot of companies, I started to say, yes, we need collaboration over cooperation and coordination. I often see teams being really good at coordinating their work. Oh, I've got this story. You've got that story, and they coordinate that. Then once they've figured it out, they divide and separate. They don't actually work together to solve the problems. I think teams that focus more on collaborative work rather than cooperative or coordinating work are also really positive cultural aspects that I would see.

Shane Hastie: If I can dig in a little bit to pair programming and ensemble programming, one of the core XP practices that was there from the beginning that is consistently told and shown, there's metrics that show that we get better quality product, but organizations still, but you're getting two people to do one person's work. How do you challenge that?

Pair and Ensemble Programming [23:22]

Declan Whelan: Well, I think there is data that shows it, but it's not super strong. The evidence is there, but it's not order of magnitude improvement. In fact, most of the work I've seen says something like, say, pair programming, you will be as productive as you would be if you had done it singly, but your defects will go down. That's usually, the work is higher quality, but it's not going to impact you speed-wise either faster or slower, to boil down most of the literature I've seen on pairing. For that, it would be just try it, just experiment, and if it works for you, then go for it. If it doesn't work for you, then try something else. I think mob programming is more challenging because it's not just, you could probably get two people that would agree to pair program, but getting a whole team would be even more challenging.

Again, the idea would be, well try it, let's try it, but you have to try it for long enough. It might be six weeks or more. A lot of my coaching work, I will coach as a learning hour if I have an opportunity, or you wouldn't have to be a coach, you could be a team lead or something and say, "Okay, let's every Friday get together for an hour. Let's work on a coding problem together." It could just be a toy exercise just to improve the skills. Finding ways to make small changes, small moves, and finding ways to incorporate that learning into the day-to-day work. For example, one thing I don't like are the improvement sprints or something at the end. Instead of doing that, why don't you try to have something that might be more regular? Experiments, quick, rapid feedback, and if it doesn't work, try something different.

Shane Hastie: An experiment. Declan, thanks very much. A lot of good advice, good ideas in here. If people want to continue the conversation, where do they find you?

Declan Whelan: Probably the best place is on LinkedIn. It's just Declan Whelan, all one, on LinkedIn. I'm easy to find. It's funny, Shane, I tore off my badge in half. Apparently, I do not know how to separate a beer ticket from a conference pass. I was ripping out a drink ticket and I ripped my ticket in half, so it only said Declan. Then I realized, that's probably all I need in the Agile community. It's Declan Whelan, and LinkedIn is probably the best way to reach me.

Shane Hastie: Wonderful. Thanks so much for taking the time to talk to us today.

Declan Whelan: Oh, it's always a pleasure, Shane. Always lovely to chat with you.

Mentioned:

• Declan Whelan on LinkedIn
• Elixir 
• Code Stewardship 
• Book: Flow Engineering