Over the last decade, DevOps has emerged as a core business philosophy and practice that helps teams drive high-quality software to market faster. While DevOps focuses on eliminating bottlenecks caused by the gap between development and operational resources, outside of that scope even the magical “10x developer” still toils daily with productivity challenges like slow builds, flaky tests, and other avoidable failures.
This talk demonstrates how Gradle Enterprise’s comprehensive Developer Productivity Engineering solution–including build and test acceleration, failure analytics, and observation technologies–can address these challenges.
Like Bigfoot and the Loch Ness monster, the 10x developer is also a mythological creature: the concept emerged in the 1980s based on a survey of software teams that actually revealed the characteristics of 10x organizations, not individual developers.
Since then, software practices like Lean, Agile, and DevOps have arisen to make software organizations more productive and aligned, but still fail to address many of today’s development bottlenecks–e.g. long build and test times, poor visibility into failures and flaky tests, etc. In this talk, Gradle describes the nascent evolution of DevOps into DPE, the problems that DPE solves, and how bringing productivity and joy back to coding can help us go beyond the 10x developer and build 10x organizations.
Justin Reock is the Field CTO and Chief Evangelist at Gradle Inc. Justin is an outspoken blogger, speaker, and free software advocate with over 20 years of experience working in various software roles. His focus is on delivering enterprise solutions, technical leadership, and community education on a range of topics.
Gradle Enterprise customers utilize Gradle Enterprise Build Scan™ to rapidly get deep insights into various DPE-based metrics, like build and test performance, flaky tests, dependencies, failures and regressions. From there, features like Build Cache reduce build and test times by avoiding re-running code that hasn’t changed since the last successful build, and Test Distribution further improves test times (often 90% of the entire build process) by parallelizing tests across all available infrastructure. You can learn more about these features by running a free Build Scan™ for Maven and Gradle Build Tool, watching videos, and registering for our free instructor-led Build Cache deep-dive training.
Check out these resources on keeping builds fast with Gradle Enterprise.
Watch our Build Scan™ getting started playlist of short videos to learn how to better optimize your builds and tests and to facilitate troubleshooting.
See how Test Distribution works to speed up tests in this short video.
Sign up for our free training class, Build Cache Deep Dive, to learn more how you can monitor the impact of code generation on build performance.
JUSTIN REOCK: Yeah, thanks everybody for joining. It looks like we’re the second to last session before the DPE Showdown. Everybody enjoy that yesterday? It was cool, right? Yeah, I liked having that in a really live format. We have a bunch of those archived too on our YouTube channel. So this wasn’t like the first time we’ve done the showdown, it’s been going on for a while. We also have a series called Lowdown, which is a little bit more like fireside chat. Definitely recommend that you check those out. So all right, what are we talking about today? We’re talking about breeding 10x developers in part by using the practice of Developer Productivity Engineering.
My name is Justin Reock. If we haven’t met before, I’m the Field CTO, Chief Evangelist at Gradle. I have about 20-ish years of experience in the industry. Spent a lot of my career writing code, moved into systems integration, enterprise architecture with a focus on integrating a lot of open source, which if you think about it, coming up with a good policy for using open source and keeping open first is a step that you can take towards better productivity. I think a lot of us realize at this point that the fact that it’s free, you don’t have to pay for it, it’s really not that interesting. It’s really more about being able to get access to high quality software really quickly, so it is very much about productivity. So to move into this space, which was about a year and a half ago, I started working for Gradle, was very natural transition for me to make to start talking about this subject.
All right, let’s get one thing out of the way. 10X engineers are mythological, all right? Similar to the yeti, the centaur, the chupacabra, the unicorn. I apologize, by the way, if any of you actually believe in these. I don’t mean to alienate or isolate, but I think it’s pretty common knowledge that these are well-recognized mythological creatures. So where did the origin of this myth actually come from probably? Hard to say for sure. In all likelihood, and you can take a look at a much deeper version of the study here with a lot of data, it came from an experiment that was done in the mid ’80s by two folks, Tom DeMarco, Timothy Lister. And what they did was they put together about 600 developers that were involved in this, about 92 different companies. Pretty specific, actually, it was exactly 92 companies. And they took two people from each company and they gave them a task to perform with software. One that’s intermediate difficulty, but it was the same task.
And it was not peer programmers, they weren’t working together. In fact, they were competing per organization, keeping track of how long it took for them to complete the task. And so nothing was really dictated. You could use whatever language or work station or whatever that you’re familiar with, you’re comfortable with. So they didn’t really dictate those types of rules. It was really just, make this thing work, and then tell us how long it took. And so, it was really about gathering data, and again, these weren’t two programmers working together, it was about seeing who in the organization is… What are the traits of a good developer? What are some of the traits of a good developer? What are some of the traits of a bad developer?
So, some really interesting data came out of this study, and it went into this book called Peopleware. Anybody familiar with this book? Great. Great. For those who haven’t checked this one out, if you manage software teams or you help to build software organizations, I highly, highly recommend this book. It’s got fantastic advice and I believe it’s in its third or fourth… This is the third edition. I think there may be a fourth edition that’s actually floating out around there now. But a lot of the data would come out of the study, would make its way into the principles that you see inside this book. And it’s effectively a book that makes the case that, I think we all heard some version of this before, computers are easy, people are hard. Organizationally speaking, typically, the problems that we face with productivity are human problems, not necessarily software problems.
So the best programmers in this study outperformed the worst by roughly a 10 to 1 ratio, which is again where this 10X figure probably came from. There were some really interesting non-factors in this study, things that actually didn’t seem to impact the productivity of these developers. The first one was language, interestingly enough, with the exception of assembly. Assembly programmers definitely took a little longer. But in general, the language didn’t really have much of an impact on the productivity of these developers. Years of experience, maybe a tough one for some of us who are veterans of the industry, myself included, to swallow, but really anybody within about the 2-10 year level of experience performed at roughly the same amount of productivity. It was when you started getting into six months or less that you saw real deviations in performance there.
Number of defects. Well, of course, okay, maybe these folks are cranking out the code really fast, but it’s crappy code, it’s full of bugs. But no, that was not the case at all. In fact, the defectless code, the people who produced defectless code were slightly more productive. They actually outperformed some of the folks that had a little bit buggier code. And another one that’s tough to swallow here, salary. The top performers in the study were only making on average about 10% more than the lowest performers in the study. All right. What did matter? Well, paired programmers from the same organizations actually performed at roughly the same level. It was about a 20% difference in performance on average between the two folks working for each company, which if you think about the fact that you’ve got a 10X ratio here, that’s kind of statistically insignificant at that point.
So people working within the same organization were producing at roughly the same level. So the average difference here only being about 21% between paired programmers. The best organizations performed at 11.1% or 1x better than the worst performing organizations. All right? So maybe we don’t have 10x developers per se, or 10x engineers, but we do have 10x organizations. So, the best performers were clustering in some organizations while the worst performers were clustering in others, some companies were doing a lot worse than others. And there’s something about their environment, something about the corporate culture failing to attract and keep good people or making it impossible for even good people to work effectively.
So let’s look at some of the data here. If you look at those who performed in the first quartile versus the bottom performers in the fourth quartile, look at how they answered some of these questions. All right? How much dedicated space do you have? 78 square feet versus 46 square feet. Is it acceptably quiet? 57% of the top performers said yes, only 29% of the bottom performers said yes to this. Is it acceptably private? 62% of the top performers said, “Yes, it’s acceptably private where I work,” whereas only 19% answered yes to this question in the bottom quartile. Can you silence your phone? It was a problem in the ’80s, it’s a problem now. They were different phones back then, for those who may not have been around, big bulky things that you kept on your desk, but you had the ability to silence it.
If you did, 52% of the top performing organizations said that you could do it, whereas only 10% of the bottom performing organizations answered yes to that question. Can you divert your calls? 76%, 76% in the top performing organizations said yes. Do people often interrupt you needlessly? Slack. 76% said yes in the bottom performing quartile, whereas only 38% said yes in the top performing quartile. So what do we derive from this data? Well, the phrase really hadn’t been coined yet, but productivity really came down to developer experience, and more importantly, the kind of developer experience that was being curated by the organization that these folks worked for. So it’s fair to say that 10x organizations are manufactured and not born.
So the myth is that people are just born with a 10x ability, but what the data really bears out is that we have to engineer this a bit. Now, this matters even more now than it did in the ’80s. Some of you may have already seen what I think is a very staggering, but also real statistic. IDC, back in late October, 2020, predicted that 65% of the global GDP would be digitally transformed by now. All right, now that was accelerated a bit by COVID, but it bore out. Let that sink in. 65% of the global GDP is digitally transformed. That is a lot of software and it is a very, very busy workforce, so we really need to be thinking about this stuff now. But so many organizations are still not even aligned in their vernacular and the way that they think about productivity.
You’ve probably heard a lot over the last couple of days about the SPACE Framework or DORA metrics, any of these things, very popular ways of gauging the productivity of an organization. So in a more recent study, not the ancient business wisdom of the ’70s and ’80s, which I will refer back to shortly, but this one was actually April of 2022, so a little bit more recent. Between Microsoft and the University of Victoria in BC, developers and managers were surveyed on their interpretation of the SPACE Framework. Now, SPACE is actually a really cool framework because it’s a normalizing framework. It allows people to give free-form responses about how they think about productivity, and then those bits of the responses can be folded into one of these five categories; satisfaction and well-being, performance, activity, collaboration and then efficiency and flow.
Now, in this study, developers or individual contributors and managers were asked a very similar question, but they answered very differently as you’ll see in the data. Developers were asked, “When thinking about your work, how do you define productivity?” And managers were asked, “When thinking about your team, how do you define productivity?” Now, overwhelmingly, the managers talked about things that represented performance. What’s performance? It’s things like meeting deadlines, getting our story points cranked out, getting our features out on time. This is what managers were thinking about. Developers were thinking about activity. All right, what’s the difference? This is like lines of code that I’m working on right now, actual toil, work that I’m doing.
So the developers, the individual contributors are thinking about performance in terms of the raw work that they’re cranking out, whereas managers are thinking about metrics that matter more to business objectives. Not surprising, but it should give you a pause for a minute. If we want to really unify as organizations and improve productivity, we have to be thinking about these things in the same way. We need a framework to allow us to think about these things in the same way. So getting back to the ancient business wisdom of the ’70s and ’80s, let’s talk a little bit about the physics of performance, what research has been done that gives us a lot of good data points and a good way of thinking about this? One of my favorites is The Goal, Eli Goldratt and Goldratt’s equations about productivity.
He was a physicist turned business novelist, gave him a very interesting perspective on how he thought about organizational productivity. And I love these models. Goldratt’s not the only thinker who connected the dots and said, “If we could use the laws of physics to talk about the way a business works, then we would have a really good and almost infallible way of making predictions about how that business is going to perform,” because you can’t break the laws of physics. Anybody read The Goal, by the way? A few of you, okay. I really recommend this one. His most famous work is the Theory of Constraints, which was born out in the book, The Goal, and it helped form the foundations of modern business productivity theory. So, practices like DevOps and Agile and Lean, these all stem from the thoughts that came out of the Theory of Constraints.
The theory focuses on treating organizations as complex machines, which they are. And he just took the physics of a complex machine and applied it to a human organization instead. And it focuses on things that you’re probably very familiar with, even if you haven’t read this book, if you’re not familiar directly with the Theory of Constraints; value stream mapping and value stream management, finding bottlenecks to productivity in that value stream, dealing with them proactively. These are the actionable outcomes that would come out of The Goal and the Theory of Constraints. By the way, this doesn’t read like a business reference book. It’s actually a page-turning fictional business drama. It’s a fun read. It’s about a manufacturing plant that’s struggling to keep pace with competition and a change agent that comes in and introduces the Theory of Constraints and helps to save the business.
So it’s a fun read, and it doesn’t feel like reading a reference book. And if you think a lot about productivity, I really would recommend that you check this out. If you hate novels or reading in that way, I have good news for you. There is a graphic novel of The Goal that was actually published, I wanna say, maybe like five years ago, and it’s actually a really cool edition. They do a good job. It’s got good artwork, but everything, all the messaging from the original book is preserved. What about this one, The Phoenix Project? All right, a few more hands on this one, great. So if you read the intro to The Phoenix Project, you might have noticed that they basically said, “Hey, we’re ripping off The Goal. We’re gonna write basically the same book, except it’s gonna be about a software organization instead of a manufacturing organization.”
But that’s exactly what The Phoenix Project is, and it modernizes the Theory of Constraints for software organizations. And it’s the same type of experience, and I think a lot of you may really resonate with this one, especially if you’re a manager of a team or if you lead organizations. It follows a VP of Software Engineering through a very difficult time for the software company, and then you keep thinking, “Oh, how’s this guy gonna deal with the situation next?” And the Change Agent Theory of Constraints, all of that stuff, except applied to a modern software business. So fun read, business drama, not a textbook format, demonstrates this relationship between DevOps and throughput optimization, and we’ll get to know what that means in a second.
It teaches the importance of value stream mapping, observability and continuous improvement, but not through the lens of business theory. Again, it does it through the lens of physics, unbreakable physical laws. So there’s a lot in the Theory of Constraints. There’s many equations and a lot to really get out of it, but one of the conclusions is that we have to change the way that we think about how our organizations work.
Now, we tended to, businesses who hadn’t really paid a lot of attention to constraints-based thinking tend to focus on cost first, optimizing the cost of the business. One of the fallacies that comes out of this is the idea that layoffs are actually gonna make our business more productive. It doesn’t. We control cost, but we’re not doing anything about business throughput, all right? So, what are these three components? Well, Theory of Constraints basically says that any complex machine has three forces; inventory, whether they’re the raw materials that make up the machine, the gears or whatever, or the raw materials that are going through the production facility to create something for the business. There’s cost, lot of cost, the energy associated with assembling those raw materials into something valuable or whatever that ends up being. And then throughput, the machine doing its work.
Now, when you think about this in terms of a software business, just take a quick intellectual leap and see that cost is still organizational cost, what we’re paying our engineers and the tooling and everything like that that’s associated with being able to produce things. Inventory, in this case, code. In case of a software organization, the inventory is code. It’s why there’s been such a focus in DevOps, in moving that source code out to production as fast as possible so that it’s doing what? Creating throughput, i.e, money for the business as fast as possible. And this has been a huge focus of DevOps. And when you start connecting the dots between Theory of Constraints and DevOps, these things really start sinking in. Why is it so important that we make it as short a loop as possible between the developer writing their code and that code actually being out in the wild making money for the business?
Because everything rots, because of entropy, because the longer that code sits lingering, not making money for the business, it’s not really standing still. Nothing stands still, it’s getting worse. Entropy denotes that nothing stands still. It rots. And we see this in code rot. We have lots of different ways that this can happen. We see code vulnerabilities that’ll pop out of nowhere and then the business has to stop and deal with that before it can actually get it out, or deprecations, or needing to modernize or upgrade a framework. All of this has a cost associated. The inventory has a cost associated with it, and there’s a loss, there’s a depreciation. So this is why DevOps focuses so much on turning that code into throughput for the business as fast as possible. And this is the essence of the Theory of Constraints and the way that you wanna reshape your thinking to bring your organization more in line with this theory.
Whoops. Okay. So another conclusion that’s drawn from the Theory of Constraints is that the only initiatives that are gonna positively impact performance are ones which increase throughput while simultaneously decreasing cost. And this book is very explicit about, “Hey, if it doesn’t do both of these things, don’t do it.” This goes back to the layoff question again. Layoffs can help us control costs, but they actually have a deleterious effect on throughput. There’s generally brain drain. You’re either working people too hard and you’re burning them out. So if you don’t have a plan for doing both, it’s not like you can never lay people off again, obviously, it wouldn’t work that way, but you also need to have a strategy in place to make sure that your throughput is being increased, or your business’s productivity is really not going to get any better as a result of this.
So whenever you start thinking about a new practice or a new policy or something you wanna put in place, apply this rule. Is this change going to both decrease the cost of my system and simultaneously increase that system’s throughput? And if it does both of those things, you’ve got a winner, according to the laws of physics, which, hard to break. So, it is time then for Developer Productivity Engineering. 65% digitally transformed global GDP. A huge, huge dearth of developers and a giant code debt that still needs to be addressed. So DPE has become this next practice following practices like DevOps and Agile, again, dipping into that ancient business wisdom of the ’70s and ’80s. But all of these practices that became popular, all of these practices that helped businesses become, let’s take a picture, become more performant, they did both of these things at once.
Just-in-time manufacturing, business process, re-engineering, change management, which would turn into Agile, Lean Six Sigma, which would evolve into DevOps, and now the consciousness of the industry is shifting towards thinking about developer experience and developer productivity. Some of you may be, if you’re familiar with Gartner reports, if you like reading the Gartner reports, the 2022 Gartner CEO report came out just a couple of months ago. It’s got some very interesting information in there. 40% increase just over the last year in CEOs saying that they are now concerned with developer experience. 70% increase just in the last two years. All right? So this isn’t necessarily to suggest that you have to have a DevOps mature organization or an Agile organization before you can start benefiting from DPE.
It’s really more again about where the consciousness of the industry has been shifting. And that’s why I think we can really draw some conclusions and say DPE is really lining up to be the next big thing in software development. All right. So DPE, we’ve talked about it a lot. Everybody’s got, I think, wonderful different definitions of it. And Gradle Enterprise, obviously we provide an enabling set of technologies for this practice, but the story doesn’t begin and end there. There’s so many ways that we can engineer better productivity. But I do like to start the story here. These kids are happy-looking. They’re ecstatic even. And I think they feel like a lot of us felt when we first learned to code. It’s kind of what they’re doing. They’re in this wonderful state of creative flow. They’ve just typed System.out.println, “Hello, World”, and they went going, boom, showed up on the screen. “Yeah, look at that. So cool.”
I think a lot of us have been there. I think a lot of us have felt this. Science tells us, we’ve actually mapped the brain while people are writing code, and it’s this wonderful soup of left brain and right brain activity. On the one hand, we’re creatively problem solving. We’re visualizing the problem in our head. But then we do this very scientific thing where we write code and we create this dialogue with the tool chain. And we make a hypothesis, “Hey, I think this code is going to solve this problem.” And then we get feedback. Then the build system tells us, “Yes, our code did what you wanted it to do. You can move on to the next task.” So I want you to put yourself in the shoes of these kids for a second. And then I want you to think, what would it take… What would happen if these kids were waiting 45 seconds, a minute, 10 minutes, an hour, 20 hours in some of the most extreme cases that we’ve seen for Hello World to flash up on the screen?
They’re not going to look like this. They’re going to look like this. And they’re probably going to go off and find some other hobby. I know I would. Why? Well, because they’ve broken their state of creative flow. Creative flow feels great. We’ve all been there. We know how it feels. It’s ecstatic. We crave that feeling. But when it’s broken, when instead we’re experiencing toil and friction and frustration, that’s not fun at all. It’s agony. It’s the kind of agony that a lot of us have even become numb to, I would say. A lot of us think about this as an occupational hazard of being a software developer. But it doesn’t have to be that way. So DevOps, Twelve-Factor, Agile, great practices. This is not to say anything demeaning about these practices. They’ve done wonderful things for innovation and for our industry, but they have not captured all the bottlenecks and friction and obstacles to throughput that are experienced by developers.
Many of them are just hiding in plain sight in the developer experience itself. So a 10x organization should think about reducing build and test feedback times, improving the consistency and reliability of builds. All three of these things are really important. Speed, I think we all kind of get that, it makes sense. But consistency and reliability are equally important. If you have a build that takes 30 seconds most of the time, and then every now and then it takes 10 minutes, you’re basically gonna plan on that build taking 10 minutes every time, and that’s gonna guide the types of changes that you introduced to the code base. So, unfortunately, even in 2022, this is still what a lot of our calendars look like. Get up in the morning, we’re coding, we’re happy, we’re in our flow state, we’re great, we’re doing what we love to do. And then we’re waiting, waiting for our local build to complete, and it failed.
So we’re gonna spend the rest of the morning debugging that failure, probably with ineffective tools and not enough data. And we’re gonna go to lunch, we’re gonna get back, we’re coding again, “Yay, we’re where we wanna be.” Waiting for our local build to complete, but it works this time. We debug the failure, it’s great. So then we pass it to CI, and then we wait for the CI build to complete, and then test for flaky, and we’re gonna spend the rest of the day investigating flaky tests instead of coding, all right? We have to do better than this. If you’re a developer, you deserve better than this. If you’re a person who manages developers, you owe it to this workforce to improve the situation, to really think about how you can optimize this. We wanna get out of this, all right? I love this cartoon. A lot of you have probably seen this cartoon, but it’s a sad cartoon. All right? I mean, this is legit. We’re slacking off. We’re sword fighting in the hallway.
Why? Because the code’s compiling. Nope, it’s cool, we’re legit, we’re working. Uh-uh, uh-uh. It’s not funny. It’s sad. All right, so what explicitly are we trying to solve with Developer Productivity Engineering? Well, I kind of see a few things here. Because imagine this is a developer, and these things floating around the developer or feedback cycles. The essence, probably the single base unit of developer experience is the feedback cycle. Sometimes things are fine. By the way, this could be local, remote, CI environment, doesn’t matter. Sometimes things are fine. Sometimes they take too long. All right? Hammered on a lot of that over the last couple of days. Sometimes they take too long to fix. All right? A break isn’t necessarily going to take a developer out of that flow state. It’s a new problem to solve. Great. We love solving problems. It’s why we do what we do.
But if we have bad tooling, if we can’t get the data that we need, we’re playing 20 questions with the build system, we’re going back and forth with the build engineer over Slack. If we’re not able to get the data we need to solve the problem, we’re out of flow state, we’re experiencing friction, we’re experiencing frustration, we’re not happy. Now the last one. The problems that should have been observable and therefore avoidable from the very beginning. If we had just applied a little bit of analytics and observation, we could have proactively dealt with these. Everything’s infinitely more than zero, and so these are the ones that we should really be concerned with. Toil that our developers are experiencing, that we really could have dealt with proactively and completely avoided their need to run into that.
We did our own study, Gradle, on talking to businesses who have established DPE practices. We found out that 81% of surveyed IT professionals agreed that DPE’s impact on their tool chain made their job more enjoyable. It’s made them happier. And we can absolutely link, you heard Hans’s opening talk, joy and productivity and productivity and joy. This is true. And it’s because it comes back to the flow state. It comes back to not experiencing that toil. So I think we all say that fast feedback cycles are important. I think we all sort of know this to a degree. And there are some very obvious outcomes and effects of faster feedback cycles, and there are some not so obvious ones too. So when we introduce faster feedback cycles to the business, well, we experience less idle and wait time, no-brainer. What does that mean? Less context switching.
Now, Hans got into the… Yesterday, he got into some of the neuroscience here. It’s very compelling information to look at. Over the last decade, of course, we’ve realized, hey, human beings can’t multitask. We convince ourselves that we can, but it’s a lie. It’s a trick we’re playing on ourselves. We can’t really do it. But it’s over the last couple of years that these new studies have come out that said not only can we not do this, we really shouldn’t. It’s bad for our executive functioning to try to force our brains to work this way. And so what Hans was talking about, about glutamate and other toxic buildup that happens in the prefrontal cortex as a result of trying to force our brains into this mode, it’s all very true and it’s very interesting science, and we’re still learning more about it.
So what do we have when we have less context switching because developers are not spending as much time waiting for builds and test cycles to complete? We have more focused developers, we have healthier developers it turns out, and we have better quality and better productivity. What about some of the not so obvious ones? Well, psychologically, if we have a faster feedback cycle, we’re gonna build our project more often. We’re gonna ask for feedback more often. We’re trying to refine our work. That’s the whole point of the feedback cycle in the first place. So we’re going to issue more builds, we’re gonna build more often, which means we get earlier quality checks. Instead of cramming a whole bunch of code in, because we know we’ve got to wait an hour for that thing to compile, instead, we’re able to introduce a much smaller change set, and if something breaks, it’s a lot easier to figure out what broke.
Leads to fewer expensive downstream incidents. Smaller change sets leads to fewer merge conflicts on the whole, which means more efficient troubleshooting and a faster mean time to resolve, which, again, all bubbles to quality and productivity. So just introducing faster feedback cycles can have these really significant and cascading impacts on how this business performs. All right, DPE organizations also work to eliminate avoidable failures, all right? Rather than just blindly addressing build failures or saying, I mean, I heard one loud developer saying, “I have the worst problem in the world. You have to fix this for me right now.” Instead, we try to aggregate data, keep track of failures and their frequency and how many developers they affect and they impact.
Like this, for instance, is Gradle Enterprise’s failure analytics. And we can see that this one failure, in this case, has impacted over a thousand builds over this is a week long period. It’s affected 18 different users across 247 hosts. And this data is highly actionable. This allows us to make informed decisions about which type of build failures we should be addressing first, rather than going anecdotally or dealing with the loudest developer in the room. So DPE organizations track failure rates. What about dealing with flaky tests? I feel like this is how most of us deal with them now. Try it again, rerun it, rerun it again. Diagnose, improve the PR. Several bad things here. I know we’ve got Spotify folks in the audience. I love borrowing a quote from one of Spotify’s blogs about flaky tests, where they refer to flaky tests as the infinite pit of sorrow for developers, and I think it’s such an apt description.
They’re maddening, they cause us to lose faith in the build tool chain, they cause us to repeat work, which can put additional pressure sometimes even on CI systems because we’re running things more often than we need to, and ultimately they, if we psychologically just pretend they didn’t exist by the time the test succeeds, we’re probably introducing a bug, all right? So there’s a lot of reasons to proactively deal with flaky tests and different approaches to doing so. So DPE organizations tend to analyze flaky tests. So for instance, our own build tool and Gradle engineering teams, our Gradle enterprise engineering teams, they make this data actionable by scheduling flaky test days. We keep track of, we detect flaky tests, we put them on a dashboard, we allow you to sort them by how often the test is flaky, and you just start at the top and you eliminate them.
And what is this doing again? This is proactively avoiding toil and frustration from our developers. This is the eliminating altogether and proactively eliminating a developer dealing with that frustration in the first place. By the way, all of this does improve CI. We accelerate our feedback cycles. We run less things in general because we have eliminated flaky tests and things like that from the system. A lot of folks, again, because as an industry, we become numb to some of these problems, like slow feedback cycle times and things like that. Because of that numbness, we don’t always know that we have a problem. To borrow another quote that I really love from Danny Thomas, who I know some of you know from Netflix. We interviewed him on our DevProdEng Lowdown, and he said something to me that was just, it’s indelibly etched in my brain because it’s really the essence of everything that we’re doing.
He said, “It is staggering the amount of friction and frustration and toil that engineers are willing to put up with.” And I think that really sums it up. It’s the essence of it. But sometimes we don’t feel things because developers aren’t complaining. Nobody’s complaining, everything’s fine. Whatever, 15 minute build time is great. We all love it. But sometimes we feel the symptoms in CI. Sometimes what’s really going on is the CI agents are backing up, jobs are queuing, things are taking too long. And so when we start introducing these same principles that are really meant to help the local experience for developers, we end up improving CI as a result too, and sometimes that’s how it’s more immediately felt by the business. So all of this is to say that we really feel that DPE will become standard practice because the world should foster developer joy more than ever before.
And I think we’re really seeing things align across the whole industry around taking care of our developers and doing better by them from a standpoint of productivity. All right, so what can we do? Well, you’ve learned a lot of stuff. I’m not gonna presume to try to give you any better solutions that you’ve already kind of seen, thanks over the last couple of days. But some high level stuff. First of all, DPE organizations tend to build focused teams. All right, productivity should not be a part-time responsibility of some proactive and thoughtful senior developer or development manager. It won’t work. It needs focus. You need to have a dedicated team. So if you look at some of these companies that are doing DPE really, really well, I think these folks really serve as some of the best examples of organizations that have really been thoughtful about their approach to productivity. They tend to build a center of excellence.
This, by the way, I feel like is where we start really being able to see some of the differences between Developer Productivity Engineering and DevOps. Like what’s the ideal DevOps team? No team at all, right? We want everything automated. There is no central DevOps team. Everything’s just bots and automation, right? But with a DPE team, you really need people. There has to be that level of empathy, that understanding of developer experience so that there’s passion in actually proactively dealing with this in the organization. Another bit of good news here is that most of these companies really only started these initiatives around five years ago, give or take. So it’s pretty new. Challenge yourself for a minute and think, what if I was the person who introduced DevOps practices to my organization five years ago, or even 10 years ago at this point? What does that make you?
What does that do for your career? Wonderful, wonderful things. You’re providing a lot of value. Well, the good news is that you can take this path now. You can look at Developer Productivity Engineering. You can make this part of your career path and you can be a hero for your organization much in the same way. There’s a book for this. Some of you may have already seen this in some of the other presentations. We have an 81-page book about Developer Productivity Engineering. Lays out a lot of the principles that we’ve talked about, but in much more detail. It’s on its second revision now, and you can download it from here. You’ll see the Gradle logo on the front of it, and I actually think that’s the only time you see Gradle throughout the whole book. It’s very much intentionally designed as a vendor-agnostic bit of material for you to check out. Because, again, this practice really does transcend any more than just one company.
We also have this kind of cool resource. Etienne, in the last talk, referenced this a little bit, in our recent donation that we’ve made to the Apache Software Foundation, which I’m very, very proud of. We will be giving an instance of Gradle Enterprise to the infrastructure team at ASF. Any projects that are JVM-based projects are gonna be able to use it for free, it’s fantastic. We already have 17 other projects, open source projects, that are in this program. We call it Revved Up by Gradle. This is a really… If you’re not a Gradle Enterprise customer, but if you’re interested in what the interface looks like and you wanna play around, or if you just want some inspiration on what type of data you should be gathering and how you wanna visualize it, all of these projects have their dashboards publicly available.
So you can go to the Revved Up by Gradle site and you can start clicking on some of the blue buttons here and you can see how Spring is using this or how JUnit is using this or how Micrometer or Test Containers, or any of these other projects that are part of this program are using this. And that’s really it. So thank you. I think we probably have a little bit of time for questions, but I really appreciate the attention. Thanks.