Episode 22 | Untangling & Optimizing Your Software Engineering Cycles, with Charity Majors

Charity: [00:00:00] You become a great engineer by, in large, by working on a great team. Great teams are where you learn the defaults the best practices, the things that it, it has taken, like, hundreds and thousands of years of engineering hours for people to figure out. And you just get to, like, absorb it. It becomes second nature to you. You become a high performing engineer just by being around other amazing engineers.

Intro: Welcome to Developing Leadership, the podcast for engineering leaders where Eiso Kant and Jason Warner share their lessons on the ins and outs of managing software teams. 

Today, we kick off season two with a special guest, Charity Majors, co-founder and CTO of Honeycomb. Charity is an infrastructure engineer who has gone back and forth between management and engineering since she can remember. She joins us to talk about everything that's wrong with engineering cycles, how throwing money at the problem by hiring more people is making it worse and what we can do about it. 

As always, this episode comes with accompanying show notes with a deep dive into the main topics, mental models, and key moments from this [00:01:00] episode. Find them at developingleadership.co or linked in the episode description.

Eiso: Hi, everyone, we're back again with another episode of Developing Leadership. Jason and I are very excited today about our guest, Charity, who we've been trying to get on for a little while. She is the co-founder and CTO of Honeycomb and also the co-author of Observability Engineering, which is probably, by the time this episode is out, has just come out and I highly recommend you to go check out. I have to admit, I haven't read it yet but I've been a longtime reader of Charity's blog so if it's anything like that, it's gonna be a good one. 

Charity, I'd love for you to maybe just introduce yourself a little bit and then we can dive into some of the topics that I know we're excited to speak to you about.

Charity: Sounds great. Thanks for the great introduction. My name is Charity, I am an operations engineer by trade, I, you know, I- I've been on call since I was 17, basically. My niche, as an engineer, has always been kind of I like to be the first infrastructure engineer that comes in when you've got, you [00:02:00] know, a group of software engineers, they think they've got something, and it might be real, we might have users soon, and then, you know, I like to come and help them grow up and I've kind of go- gone back and forth between management and engineering for the past, you know, God, many years. And, you know, I'm very much an accidental founder. I, I always ... I, I was never one of those kids who's like, "I'm gonna start a company." I, kind of, like, hate the founder industrial complex. You know, don't you just hate it when it's like, "As a founder, I..."?

 .

 Like, I just... It just grates on me so much. But there was a problem that had to get solved and, you know, Christine and I decided to try and solve it and that was six and a half years ago, and, and we're still alive. I, I never expected this outcome . I mean, sorry, I'm not supposed to say that .

 .

 I knew it , I knew we would survive right from, from the start. Yeah, [

laughs]. 

Eiso: I, I, I love this and, and the fact that you're saying the, the founder complex... I think one of the things that I know I appreciate a lot [00:03:00] about your writing, and I know, Jason, you've also commented when we spoke about Charity on this, is that the fact that you know, you, you dare to be authentic, and you dare to actually, you know, say what you think and not be parroting everyone around you in San Francisco. The-

Charity: Oh, they won't talk to me.

Eiso: That makes you the perfect guest for episode.

 .

 For our podcast. We, we had a little chat right before this episode and you touched upon something that I know we'd love to hear more about which is the notion of we are in an environment, a culture, and I would say the last years, even more than ever with money so freely flowing, that every time we find ourselves having a scaling challenge in engineering and not just scaling challenge, almost any challenges, the solutions is, "Let's throw money at it. Let's throw headcount at it." all the way from board saying this to the CEO to the individual teams at times believing so themselves, as well, that's the solution. I know you think very [00:04:00] differently about this. So I'd love for you to just go on a rant and share with us and then we'll go from there.

Charity: Oh, gosh. Where to start? I mean, I'm a fan of throwing money at problems. I, I really am. Any time you can throw money at a scaling problem and make it go away, ooof great, do it. Money is cheaper that time, right? But it's... We're not even doing it that intelligently. Like, it's easier for you to get approved to, like, hire five or 10 engineers, for like, $2 million, right? Than it is to, to get, you know, approved to, like, spend 500,000 on a tool that would replace, like, you know, 10 engineers. You know, they come from different, different parts of, of the, of the spreadsheet or whatever and so, like, you have to be like, "Please, can I..." It's, it's ridiculous because, you know, you do not just add people and get productivity. You, you can't... 

We operate in these incredibly complex socio-technical systems and when part of it is on fire, when part of it's slow, when part of it's [00:05:00] understaffed and the people are, like, you know, working overtime and they have too many commitments, they have too many contracts, they have too many expectations, they're spread too thin, throwing more people into that corner is very unlikely to help and it's very likely to make things worse.

And I'm not saying that people who are underwater never need help, like, they do and sometimes people can help. But, like, just adding people is not what actually helps you and w- we're so wasteful with the engineering cycles that we have. Have you guys read the, the Stripe developer report that came out-

Eiso: I did, yeah.

Charity: ... three years ago?

Eiso: Yeah.

Charity: Right? Where they're like- "You know, developers self report that 43% of their time is just wasted." Like, it's just, it's just doing the bullshit thing you have to do in order to get the stuff that you want to do. Right? Like s- stumbling around in the dark, like, fixing tests. Like, like, you know, reproducing bugs. Whatever. Like, it's, it's like half your time is just, like, straight up, straight up wasted. 

And why is this not, like, at the top of our list of, like, [00:06:00] things to, like, be concerned about? If engineering cycles are so incredibly expensive, and they are, they're the scarcest resource you will ever have, and yet, you know, it's hard for, it's hard for engineering managers to even, like, argue with product people to invest the cycles that they need into the system in order to be a forced amplifier and a forced multiplier and, like, think about it this way, like, you know, if... You've seen the DORA report, you know, and you see the metrics, you know, elite teams, you know, mediocre teams, et cetera. And year over year, like, more and more teams are, are climbing into the elite tier, right? It went from, like, 6%, to, like, 22%. You know? And, and they're going up, like, they're getting better. They're down for less time, they ship more quickly, they ship more often, they're, they're getting better and better.

The bottom 75% of teams are losing ground. These are not shitty teams. It's like most teams, year over year, are losing ground and getting worse because it's the nature of the job that, you know, y- you're sitting here, you're a team, you're working, you're just [00:07:00] accumulating surface area, stuff you need to maintain, stuff like bugs that you just need to be aware of. You're accumulating users, you're accumulating use cases, and then there's just, like, this, this just entropy, right? Tech that accumulates. Like, you, you constantly have to, as a team, you constantly have to be looking for ways to do more with less. You can't just stand still and you can't try and get yourself out of this by hiring more people. Because at best, that is like a linear amount of, you know, cycles that you're adding.

But to combat this, like, steady, like, decrease in productivity, you need to be constantly looking for, like, you know, uh, leaps. How can I make my team, like, 20 or 30% more productive than we were last year? By making it... You know, the number one thing that comes to mind is deploy times. You know, if it takes you two hours to get a line of code out, I don't care. You can hire the best fucking engineer in the world... Sorry, to come and work in your system, it's still gonna take them two hours to get that line of code out. Right? 

Versus if you have a system where everyone's code goes out in 15 minutes, [00:08:00] you hire any, you know, average engineer out there, their code's gonna go out in 15 minutes and that means, like, if you're deploying a few times a day, you're learning 15,000 times a year. 

This is where people think that, like, they think they need to hire 'great engineers' to get great teams and they don't get that the causality is exactly the other direction. You become a great engineer by, in large, by working on a great team. Great teams are where you learn the defaults, the, the best practices, the, the things that it, it has taken, like, hundreds and thousands of years of engineering hours f- for people to figure out. And you just get to, like, absorb it. It becomes second nature to you. You become a high performing engineer just by being around other amazing engineers. And sorry, I- I'm... I'm gonna, like, take a breath and, like-

 [inaudible 00:09:17].

 ... let someone else 

Jason: This is great. And obviously on a topic that, I care deeply about given GitHub's prior trajec- trajectory in the last five years or so when I was there. But I'm really curious, what do you think is majority root of this problem? Do you think it's a technical, or do you think it's a, a managerial, or do you think it's a [00:09:00] combo, thing? Like, what- what's going on here?

Charity: So, you know, it's a socio-technical system and I think that in order to improve it, we need to look at feedback loops. Right? These systems are not complicated, they're complex. Right? It's different. Like, they're not complicated in the way you can just be like, "Okay, insert token here, get it out, like, there." Right? But the feedback loops are where things get amplified and, and can spin out of control really quickly and so, like, what's the, the biggest feedback loop that's the heart of every engineering team? It's the process from getting the code, like, that you've written to your users, right? Usually we call it, like, CI/CD. Right?

My, My motto is 15 minutes or bust. It should take 15 minutes for you to be able to get a single line of code out to your users. And that doesn't mean that they have to immediately see it, right? You should be decoupling the deploys when it releases, you should be making it so that, you know, you have feature flags, you should be making it so... Maybe, like, a- at Honeycomb we have, like, a production environment, we have the a dog food environment, we have a kibble environment, so we auto-deploy to kibble, it gets promoted [00:10:00] to dog food, it gets promoted to production. Right? But the point is that, like, you can only get so much confidence in your code, you can only make so much until you get it out there where someone's using it. And then you can start gaining a lot more kinds of confidence. 

So, like, most people are just like, "That's not possible. You know, 15 minutes is not possible. It's not safe." It's absolutely safe. Like, the, the higher your, your bar for safety and security is, the more you should care about this and the more you should make this, this really work. Right? Have you ever, like, looked at an app and been like, "Oh, cool app." And you look at the company and you're like, "They have 800 engineers, what- What are they do-... What are they all doing?" Right? Like, I think of this, this feedback loop, it, it becomes, like, the software development death spiral when it gets longer and longer and you've got manual labor and gates involved because it's like, you know, the, the amount of time gets longer and so, like, the code reviews take longer and the diffs get bigger and this is a feedback loop that, like, continues to l-... loop in and of itself. And, and [00:11:00] once you get up to, like, an hour or so, you're absolutely going to be batching together mul- multiple people's changes, right?

 Yeah.

 The best practice is, you ship one diff out from one engineer at a time, right? If, if you can't do that quickly, then you're gonna start batching it up, you know, anywhere from one to 50 engineers' diffs and that means when something breaks, well, now everyone's afternoon is ruined. Right? You've severed that, that other feedback loop of, of just owning your code. Right? If you have the, the mentality of owning your code production, you make a change and you know it's gonna be live in 10 minutes, you're very likely to go look at it through the lens of your instrumentation. Be like, "Is, is it okay?" Right? It's like, muscle memory. But if you, if you make your change, then you're like, "Well, some point and then it's there too, my change is gonna go out along with zero to 100 other peoples'," no one is going to go look at it. Like, that- that's not code ownership. There- there's no feedback loop there. That's just shooting it off into the void and let God do the rest. Right?

Jason: Yeah. The interesting part I have found in organizations, particularly, obviously, like, with GitHub [00:12:00] selling into l- large enterprises and we talk about, like, the release processes and stuff like this, is that the further you get from 15 minutes, and let's say That a lot of these large organizations are nowhere even near days, let alone 15 minutes, possibly even weeks, is the increased risk profile and the increased likelihood that they don't actually have any sort of a release process that looks in the order of weeks. And they're talking, like, maybe multiple weeks, a month or two or three between releases.

 And then also they have a release process and a production release process and a q-... And, like, multiple other teams involved in this and then it becomes a, a coordination problem of, "Well, I'm scheduling you for-

 That's it.

 ... three releases-

 That's-

 ... from now."

 That's 

Charity: where the, the spiral goes because p- pretty soon you're spending so much time waiting on each other that you need a... That it incurs all the coordination costs and you need to hire more. This is where, like, my rule of thumb, which I've been working on this for a while and it seemed to be relatively accurate, if you have a team that's supporting, you know, writing and supporting your service, it deploys in 15 minutes or less, say that takes x-number of engineers. If it takes you an hour more to get your code out, it takes twice [00:13:00] as many people. If it takes you, like, about a day, it takes you twice as many people, again. If it takes you, like, a week, it takes you twice as many people, again. It's not just engineers but, like, engineers, and managers, and SREs, and, like, Project managers and, like, it just, it takes so many more people because it becomes so much more complicated and complex than it has to be.

And that- that's it's own feedback loop that just kind of spirals the fuck out of control. And then you're just like, "Well, any problem you have, well, let's hire someone to do that. , well, let's hire someone to do that." And it's so not fun to be one of these engineers because you get tasked with such a small slice of a slice of a thing. Right? 

Software engineering can be the most creative, satisfying, fun, like, it's, like, getting to solve puzzles and get paid. Right? Like, it can be so gratifying because, because that feedback loop is fast and you get to see the impact on your users and you get to experience their pain, and you get the, the dopamine hit of just, like, "Oh, I found it before they did," or, "oh, it worked," or... You know? And you're, you're tight. You're, you're close to it. And when that, when that gets untangled and it's [00:14:00] like, you know, you're living in your IDE and at some point code comes out the other end, you don't know, you don't really look at it, maybe a month or two from now, whatever, and somebody else is gonna find the bugs that you wrote because you've forgotten what you were doing. Like, it's an entirely different occupation and it's not, it's not f-... I understand IT people who get burned out and just, like, bored off their asses because it's, it's not, it's not fun, it's not energizing.

Jason: I, I had this conversation a while back, when I was at GitHub with, Kent Beck, when he was... I forget where he was. I think he might've been with Facebook at the time. And he was exploring something which was every time you press a key on your keyboard, I want it released to production.

 And the theoretical-

 Yeah.

 ... thought exercise was like , you know, what kind of systems would you need to build to go do that?

 Yeah.

 And we just, we were talking for hours because I love to talk at the extremes because I think extremes reveal a lot of the thinking and can help you, like, clarify some things. So one extreme is you don't release ever or maybe once a year or what... You know what I'm talking about. And, like-

 Yeah.

 ... what kind of baroque systems are you gonna have at that point? But then, if you theoretically, every key stroke is a [00:15:00] release to production, what kind of systems, what kind of things do you, do you need to build? Now, that was fascinating because of the safeties involved with that and, like, what you would end up doing. But the liberties you would get out of a system like that too, if it was in place. Oh my goodness.

 Have you played with Dark?

 Theoretically, you just, like, a-... You just unburden yourself with so many different things.

 Have you played with Dark Lang?

 Yes. Yeah.

 Oh.

 Yeah, yes. .

Charity: Paul's a visionary. Like, he's a good friend and, like, that's absolutely like 10 years from now, maybe, I, I think that that's absolutely the future. Like, anything we can do to colab- that time in there is, is wasted time. It is burnt time. There, there is... It should be as small as possible. It serves no purpose in the world . Sorry, I'm exaggerating slightly.

 There's some okay things it does. But, like, it's all cost. Right? It's all... What do they call operation-... Uh, cost centers, right? It's, it's a cost center. It's, It's not valuable in and of itself.

Jason: What do you think we can do now to, to-

Charity: so- You were asking interesting questions earlier about this and I kinda got derailed, by own [00:16:00] rants. I apologize.

But you were asking, like, why, why it's like this, what is the biggest prob... And I feel like I blame engineering managers. I think that... And, you know, I... Speaking as often one myself, like, I understand why. But, like, you know, I think that we have failed to make the case. We have failed... Like, engineering managers are sitting in a position where, you know, they should understand the technology, they understand the pain and suffering their engineers are going through, but they sit next to and talk next to business people and product people, and it's their job to not just be like, "We need to, like, shipping features so that we can, like, compact and blah, blah, blah, and do these... All these things that, like, everybody just tunes out." And they just hear, "We're not shipping, we're not shipping. This is a waste of time." But, like, we need to start explaining to, to them in terms of time, and velocity, and ultimately dollars, in a sense, like, they need to see how much money we're wasting, how much time [00:17:00] we're wasting, you know, by having, for example, these long build pipelines, or so many of these things that, you know, we t- we talk but we're just like, "Oh, if only, if only infrastructure engineers ran the world, you know, we would do all these things."

But, like, in fact, you know, it, it's true . It's true that they would be better off, product people would be so much better. They'd be so much happier. Think of all the product managers and marketing teams who have to sit here nervously coordinating a deploy with the engineers and crossing their fingers and it's not gonna break this time because the Wall Street Journal is publishing something at the top of the hour. And oh God, is it all gonna go down? You know, like, that's a nightmare. Isn't it so much better if we can build something where the code's have been out there, it's been, you know, tested, you know, small groups of users, et cetera, it's solid and there are new deploys in process 'cause all we have to do is flip the little feature flag which we've tested? You know, and it's just, it's a nothing burger. Like, think of how many stress that solves for everyone and think about, how much better everyone's lives are. Like, these investments, each and every one of them is, is valuable but, [00:18:00] like, they also build on each other.

Like, in the past five years, I think that there's been this real, like, swing, uh, from, you know, being focused on pre production stuff and CI/CD to being like, "Okay, production itself is, is where it's at. This is where the game's played and this is where we need to have good tools." We need to have, like, scalpel like tools that will help us with observability so that we can slice and dice and see specific rows and how this version is different from that version from those requests. We need CAS engineering so that we can, you know, run experiments in production and gain more confidence in the code. We need progressive delivery. We need feature flags. We need all of... Like, that... When we talk about the elite teams, you know, who are the small cream of the crop that's going like this, that's what they're doing. They're all investing in the same, like, inter- interwoven overlapping toolset that j- it just builds itself and accelerates and, and it, and it makes it so that it's almost impossible to imagine ever going back.

Jason: I have a long, probably, long involved rant-y th- thoughts on this [00:19:00] myself. But I will summarize. I'll try to summarize this which is, and I've talking about this on the podcast quite a bit, which is I think we need more technical CEOs in startups. I, I really do. And I don't mean, like, "I wrote code in college," technical CEOs type of thing, like, you know, coming up through the engineering ranks, that type of thing. I also think that, as you mentioned, the technical side of the house does a poor job at times of articulating the value because they get overruled by the business. I mean, they'll say, "Hey, we gotta, like, stop production releases to go do this," which is, you know, that's, that's kind of an impossibility. You've gotta be able to keep things going while you're building, these things at the same time, which is hard for a lot of engineers to rationalize about. But it's true, there is no, we're not sports. There is no off season where you can go rebuild a basketball player's shot or a golf player's swing. You actually have to do this while you're... With the season still going on.

Charity: I, I think that's a beautiful dream that we have a bunch of CEOs who have been engineers. I mean, that'd be great. I think that it is realistic though to expect our engineering managers and [00:20:00] directors to, you know, get together if necessary. Like, plan, come up with a real, "Here's how we're going to explain it. Here's how we're going to make it clear. Here are the meta- metaphors we're gonna use." Because, like, the business consists of software at this point, and little details that it seems like only engineering teams care about actually make a huge amount of difference on whether or not the organization's going to succeed or not. Like, it's not an imaginary amount of impact, it's just... And, and CEOs aren't stupid, right? Like, it is, it is kinda like, you know, uh, CEOs don't have to be construction workers in order to, like, receive updates on how the building is going and to see with their own eyes, "Oh, that estimate seems reasonable. Oh, yeah."

You know, like, I think that we really need to get better at... And this goes for senior engineers too. Like, some of the best explainers I've ever known have been senior engineers or DevRel folks. But, like, it's not easy and there isn't a lot of prior art so you really do kind of have to sit down and, and look [00:21:00] where are the big wins that, that you think you can make and then, and then do it, and then show the impact, and then point to it and say, "Ah, look at this. We saved, you know, 30% on headcount because we decided to buy this tool instead and spin up a small team." 

We, as engineers, I think we're so... We get so hung up on being accurate and it's Impossible to give accurate estimates. It's just impossible. Just accept that. Business people do. Business people are like, "Eh, something between 20 and 50%, good." Right? Like, we need to get comfortable with doing that and just being, like, "We..." it's imperfect but it's somewhere around in there. And then we'll get closer and we'll be able to make it a little bit more accurate. But engineers are just, like, allergic to doing that. 

Jason: There's, there's two things that, in my own experience, that have worked, in this scenario. One is you... U- unfortunately I think that sometimes you can make a business decision that says, "Hey, we'd rather invest headcount than do some of this work because it's actually cheaper for us to do this from a time perspective." And I think that's a fine answer. But what happens is th- that [00:22:00] becomes a lot of the default at times.

 Yeah.

 So I think we've got to get at better explaining, I think we've got to get better at, a- a- at doing all that, absolutely. I also find that this is, you know, really solidified for me when I was at Salesforce, is everything that you were building at Salesforce was always up against a business priority. Like, you want to do something in infrastructure? Up against a business priority, and I had to justify that and there 

Charity: what does that mean? Up against a business priority? Competing with? 

Jason: When you decide what, what the priority was going to be, like, "Hey, do we invest some time in this or do we invest to build these five features?" And then go and do that. And somebody somewhere made that decision. The closest I've gotten to b- being successful, to a degree, here is that there's a portion of engineering, we understand this as engineering leaders, and all of us, I think, in the industry who have run infrastructure organizations are, there's a set of people who are outside of, of the roadmap. They're just maintaining the business-

 Yeah.

 ... at this point.

 Yeah.

 This, you know, business. So what, you know, I've done, essentially, is ring fenced a lot of that and say, "Hey, in this bucket here, this is cost of growing. This is cost-

 Yeah.

 ... of doing business and we're gonna add a couple of-

 Yeah.

 ... [00:23:00] people each year to this."

 Yeah.

 And then out of that group is where you've got to push a lot of this work to a degree and help.

 Yeah.

 If you can't get it through on the other side of the fence.

Charity: Yeah. One thing that I think we have gotten better at doing just in the past, like, five, seven years is not building everything ourselves from scratch. It used to be we'd just be like, "Oh, I'll just build a tool for that." Right? And, like, slowly been realizing, it's fast for me to just build the tool but then you, you own it and you just pile up all these things that you own and have to maintain and so and all of this stuff detracts from what's our one job. W- what is our reason for existing? Let's spend as many of our engineering cycles on that thing as directly as possible. Right? You know, we a-... Every, every company used to have their own ops team in the, in the colo. I, I sp-... Started my career, like, racking servers and taking a taxi down in the middle of the night to push the power button under my SQL primary, right? And then we're like, "Wait, a minute, maybe AWS could do this for us." .

 Right? And so we keep moving up the stack and I think we've gotten [00:24:00] really good at that. So maybe there's something we can learn, learn there. But y- your story, yeah, I... Yeah, absolutely.

Eiso: But I think there's something here is and I've seen this a lot with engineering organizations that kind of get into that point where we think we're growing linearly by adding people but we've completely flattened out. Is that the engineering leaders in the organizations have started feeling that their job is... Has one single track which is ship tickets, ship features?

 And the number one thing, and it, it sounds ridiculously simple but I've seen it time and time in discussions with engineering leaders at some companies that you really felt like they should know this, to be honest, it's, like, "You have two responsibilities." You have the responsibility to continuously improve your engineering org, the developer experience, everything associated with it. And you have the responsibility to deliver value to the users.

 Yes.

 If you go too extreme on one of the two-

 Yes.

 ... and it's usually delivering value to users-

 Yes.

 ... then you get into that death spiral. And it's, like-

 Absolutely.

 ... we've just forgotten. Yeah, we've just forgotten this fact in so many places. It's-

 Yeah.

 [00:25:00] It's kinda crazy

Charity: And we act like they're intention. Like, we act like, "Oh, in order to make happy users, we have to, like, fall on our swords and beat our breasts- and be miserable." When in fact, like, the two tend to rise and fall together .

 Yeah.

 You don't really see happy... Very unhappy one without, without the other.

Eiso: I, I do think if we take a little bit, Jason, your example of taking things to the extreme and, and if we, if we go, you know, maybe 10, 15 years into the future, we have a couple of trends and you touched upon some already that are helping us. Right? The fact of, like, we're becoming more and more comfortable with everything becoming a tool, an API, a managed-

 Yeah.

 ... infrastructure, like, we've gotten a little bit stuck in this everyone has to run Kubernetes hell but we're getting out of it . Like-

Charity: Everyone makes mistakes.

Eiso: Yeah, exactly. Like, and if we, if we assume that, like, in any function in the world that is, you know, a-... H-... Is under supplied, like engineering is right now, as some point we will find a balance.

 Yeah.

 Right? The world will get to a balance where software engineering, I mean, I hope for all of us it lasts a little while longer, but, [00:26:00] you know, isn't the extremely highly, you know, paid job for everyone, that it is we have more engineers. So if we find ourselves in this world, at some point where we have a dichotomy between the people who build the tools for everyone else, right? The small minority of engineers. And the 90% that build for what we can admit, like, a lot of stuff out there is just crude web apps and-

 Right? And the future AR, whatever it is. Do you guys feel that we're gonna get out of this in 10, 15 years, where we have a small minority of, like, highly passionate and skilled people who have really extracted everything and everybody else is really just at the, like, surface layer building? Or do you envision a different future?

Charity: It's interesting to hear you describe it that way because I have thought about this a lot and I, I think it's the same thing. But, like, the way I think of it is, is more that I... So I'm an infrastructure engineer, obviously, so I'm thinking about it from my perspective which is just that-

 Yeah.

 ... you know, every company used to have operations and infrastructure engineers and they had, like, real jobs and, [00:27:00] and everything. And, and increasingly I think that people who identify as, you know, whether it's dev ops, ops, infrastructure, whatever, you've kinda got a fork in the road. You can either find a company that does infrastructure as a service. It solves a category problem, right? It solves some category problem for everyone else and so th- they have real, actual, hard, interesting infrastructure problems 'cause they're doing it for everyone. Right? So you can either do that, or you can go, like, you can join a team, you know, th- th- that, you know, does features or whatever but know that infrastructure is, is a cost center. It's something that will always be minimized as much as possible in order to create efficiencies that help them leap frog and become more effective.

And they're still gonna need people like us, they'll, like... They're still gonna need people who are, who are experts in system think-... Thinking about systems rather than code, people who are good at, you know, managing reliability and thinking about, you know, architecting tooling around deployments, thinking. 

There's always gonna be, like, bits of glue and shit just that you just need to, like, put things together to make it [00:28:00] all work that it seems to be pretty distinct from the act of just writing code. But, you know, your, your role there will probably be more of a, a consultant or embedded in a team, or, or something. And I do think that those two kind of... They're going to diverge in ways that are stronger than we probably see right now. Different job titles, different something or other. But, yeah. 

Jason: My guess would be that in 15 years, we continue to... Like, we've only really had the, the developer centric investment category from a VC perspective for, like, 10 years at this point.

 Yeah.

 And like, some, you know, before that it was verboten to invest in those types of things. And it's still dominated by, you know, we have startups in it and we have a lot of successful maybe, you know, billion dollar exits or multi-billion dollar exits, like HashiCorp and GitLab and GitHub and stuff like this. But the industry is still dominated by hyperscaler clouds and people like that. And just, let's just be really honest here, no one at the very, very top cares [00:29:00] about th- the problems we're talking about here because, at the end of the day, it's so far pushed down the stack that they're... We can get... That's an entirely different discussion, en- entirely, at some point.

 Do I think that they're gonna care about that? Well, th- they won't care unless it moves Wall Street. And y-... It can be argued that that investment doesn't matter as much to them because of the incremental value it might produce, and there's a lot of people who are on Twitter who would agree, they're like, "Well, don't invest over there because it only moves the number this much but if you invest in a brand new category by buying X, Y, Z company or doing this, it'll move it more." So I just don't know. I don't know. Like, I feel like it has to come from a startup or has to come from Dark and Paul and somebody like that an- and people like that, and they're not the ones who are running this, it's the hyperscalers at the moment. So it's gotta be almost like this, this supersonic jet that just takes off somewhere, and I'm not sure where we're gonna get one of those. ...

Charity: I mean, if you talk to, like, anyone who's working at Google now or recently, they, they will get very solemn and, and stare at you and go like, "Well, you know, [00:30:00] in five years or so AI is going to be taking over and, like, replacing the bottom 50% of all software engineering jobs that are everything to do with websites and, you know, spinning up, you know, basic APIs and services. You're just gonna speak it into your microphone and, and tell him what you want and it's gonna exist." And that seems n- not plausi-... It seems like... I, I believe there's some, there's some there there, but I also... People, yeah, computers are not going to do everything for us magically. I'm sorry. I, I would love to be proven wrong but somebody's gotta understand how to debug that thing .

Jason: My standing joke here is that self-driving cars have been one year away for 10 years. And it's... I mean, I feel like we're in the same category when it comes to this, is that- we've seen all the stuff out there. Yes, there's, there's a category of things that which we can do, but it's not gonna be, like, we... W- we're too far away from that to matter. And, like, I'm 40 years old at this point, I don't think we're gonna see that in my lifetime. 

Charity: And we're too... Like, maybe it could do it if we could describe what we wanted, but I can't describe what I want. I'm not gonna know what I want until I see [00:31:00] it. So do it different. 

Eiso: I spent five years of my life way before it was cool on, on machine learning on code and pretty much I, I like your self-driving cars, like, they're always five years away. To me, what I came out of those five years, and this was pre-GitHub Copilot, like, way, way back in the day already. And I came out of it realizing, we're gonna augment ourselves with writing code, but if anyone thinks being a software engineer is writing code, then you're not understanding the role.

 And I think this is where it really comes down to. You see this with any great engineer, and not only any great engineer, right? You're, like you said, you're solving problems, you're designing a complex system, it's, it's architecture, it's, like, yeah, remembering what code we need to write for problem X, Y, or Z will probably be solved and it will be a lot faster than us having to Google it or trying to remember it or even knowing that there's seven solutions out there and the third one is the best one. I think those questions will be answered for us by software. But it doesn't change- it will make it more fun for us, I think.

 It's just gonna [00:32:00] allow us to get to that point. You said it earlier, Charity, it's like, "Everybody gets excited when you see the changes you make get into production," and it all comes down to the fact of when we all stared programming, and I think this holds true for every single, you know, developer, unless some might have started in really, really dark corners of the world, like, you know, you wrote something and you got to see it on a screen.

 You were writing-

 Like, it was compiled. 

Charity: Yeah.

 You were editing it in Vim on the server and hitting save-

Eiso: Exactly.

Charity: It was great.

Eiso: And, I mean, you, you wrote Heroku, right, Jason, right? You- you've managed to capture some of that magic for people and we've spoken a lot of this podcast how we've kinda deteriorated in our industry and, and we might be kind of getting back to it. But I agree that we're probably not gonna get out there with the, the Citibanks of the world that have 20,000 developers and have just gotten themselves in a place that you can't get out of. 

We had a guest on our podcast who said something. He said, "I don't think about if my code scales in production, I think about if I can add people to my code. And the number of people adding to that code, can that scale?" And I don't think you can solve [00:33:00] that at a Citibank. But all the young companies that are gonna be the next Citibanks of the future, at least a lot of them, you have... You see this DNA coming, engineering ops team starting, VPs of engineering, who are thinking about this stuff.

Charity: Yeah, no, that- that's a really interesting point. Yeah, the big enterprises of the world. 

Yeah, there's, there's also a lot of times I think that and the thing that I often find myself wanting to say to engineers is I feel like they get this learned helplessness when in fact, if you, you are an engineer, a competent engineer at a company, you have within you hands the power of life. Right? If you are determined to figure something out or to go change something or fix it, take an afternoon and you can prototype something. You, you know, you have powers that probably many of you have never flexed and you've never tried them, but they exist. You can make people listen to you if you care enough. You don't have to ask for permission to fix your CI/CD pipeline. If you get mad enough, you can go do it in a couple of days. And then everyone will be super stoked with you and, like, praise you, and suddenly that'll be something that's cool that other people want to do. Right? 

Like, we have the [00:34:00] power to, like, do things, to, to act and that's something that most other functions never really... Y- you know, I feel so bad for anyone in any other function who doesn't write code who, who comes with their, with their cup out, you know, needing help from engineers, and so many smart people right now I think are learning code, no matter where they sit in the org 'cause just having that ability to kinda script something up for yourself, automate something, like, it makes your job, like radically better and radically different and people look at you in a different way.

You know, I've talked to people, these big old organizations where they're like, "There's so much red tape. It takes us two days to push. I'm not even allowed to touch it." But they bring up the new service and they do everything completely differently. And suddenly, everyone wants to work on that service. And so it gains gravity and momentum you know, and suddenly the managers can't ignore it anymore and so that's how they get permission to go fix the old things because the new things are just s-... You know, and I'm not saying it's easy but, like, this is what we do as engineers. We solve problems and we're lazy, and we look how to do things quickly and easily and, and, and I just think that that's, that's... I- it should be, it should be an inspiring and, and [00:35:00] fun thing rather than a, "Oh, God, I'm so constrained," 'cause we aren't really . 

Worse case what? You go get a new job.

Eiso: Honestly, I think we couldn't end today's episode on, on a better note. Everybody, take on arms, you know, rise up, listen to Charity , start by fixing that CI/CD pipeline. And, and, and actually that is still one of the best places to start. I loved this today, Charity. I, I have to say that I can already say that for sure we're gonna want you back because there's a lot more topics to discuss. But- your message today, I think, was very inspiring and... Yeah, I loved it. Think you did as well, right, Jason? 

Jason: I do. Thank you so much, Charity, for coming on. This has been a lot of fun. 

Charity: Super fun. Happy any time .

Eiso: All right, awesome.[00:36:00]