Aurimas: Well. Hello, everyone. Hopefully you're in the right room and you do want to hear about chasing Gradle build speeds because that's what I'm going to talk about. Yeah. Thank you for the introduction. Yeah, I'll just get started. All right. So you might have heard about Google. We're a little mom and pop shop. We run a search engine. We also happened to build AndroidX libraries, which are like a set of libraries that pretty much every Android app uses. So I will be talking about that. Google does a lot of things. I'm going to be focusing on that. And, you know, if you know, Google also doesn't really necessarily build everything with Gradle, obviously there's the whole other infrastructure, but I will be talking kind of the Gradle bits and what we use there. Yeah, I was going to say about me being in Google for ten years. Six of those have been on Android team and a lot of part of it was the working on support library that eventually became AndroidX. I'm currently a team lead working on builds, tests and a lot of policy enforcement, so API tracking and whatnot. So I'm here to kind of tell you about the story of our growth and kind of like more focus on like history, with focus of the build. And while you might not get any really specific advice out of this talk, my hope is to inspire you that your build speeds don't have to suck. Right? Like as your project goes. It doesn't have to go to crap. Right. So that's. That's what I'm trying to do. So if we go, we're going to get started. A little bit of concepts, at least the way I think about it. So for me, you know, the way I want to think about it, you know, there's things that make your build faster and the things that make a build slower and you know, project size, lines of code, the number of projects and so on affects your build speed and usually in a negative way. But that tends to be like a thing that people want to do, they ship features, right? They want to do that, but it makes their build slower, and then the opposite side kind of like, on addition on that you can also make your build slower by adding more capabilities. So if you want to run Ktlint, or you want to do some static analysis, or you want to do, insert a thing you want to do, and that also makes it slower potentially. And then the other side is kind of the optimizations of the build. So what can you do to make it run faster? Right. And that's the thing. Well, it's a delicate dance if you know, like how do you make it do all the things you want to do fast? You know, it's tricky. But I think the part that I've seen a lot of teams struggle with, is kind of the optimization step. It's really as easy to add and you plugin, it's really easy to add new code. But, you know, optimizing your build while it's not really sexy and it's also kind of painful. So that's what we'll be talking about. So where did it all begin?

Support libraries started in 2011, so a while ago and the focus was on kind of like extending what Android platform can do, and just kind of like provide some like a layer of a shim so you don't have to worry about backwards compatibility as much. It was 8000 lines of code when it started, a single library that was shipped and was using Make the build system, which happened to be the build system that the Android platform was using. So it's kind of like piggybacking on everything we have already. And it was written by an Android engineer, Diane Hackborn, which you might have, if you know Android Circle, she was there from the very beginning. If you want to jump forward a little bit, you know, we added four more projects. You know, this is jumping to 2013 and this is the first time we add Gradle to our build, so now we have two parallel builds and if that's ringing scary bells of like you have two build systems. Well that's what we had. At this point we're at 52,000 lines of code, so we're like we're growing in size as well. This is using Gradle 1.11, so maybe pre-history for many of you, and then this is AGP pre 1.0, so this is like really old stuff. And then two build systems, they said not a great thing. Every little library you add, you have to have matching build files for one build system and another. And this is going to become a story as we build. Yeah. So in 2015, this is when I joined the team, so I will have a lot more context from there on out. We only use Gradle for essentially creating a Maven archive, that's the only thing it was kind of useful for at the time. No pre-summit testing whatsoever, like not even building anything. All we did is post submit YOLO, you know, hopefully works and then you know, and Make is the only build system too, like we don't even test Gradle, like if someone broke the build well good luck. Sink it and learn that it doesn't work anymore.

Jump forward and I will be doing this in half a year increments. When I say next, that's where it comes and goes. We go by 15 projects. So like, you know, it's a sizable growth. And then I will be showing lines of code as well along the way. So you can kind of track of where we're at. At this point, 15 projects we're really kind of like picking up speed here. Our Gradle system can actually build device tests and run them. Well, you know, it seems very basic today for you. At the time, it was really kind of like, oh, yeah, we can do this now because we have to do Make and like, you know, all of a fun platform, things that we had to do to make it work. So this is like advancement for us. Jumping in, you know, half year forward we had a little bit more projects. But luckily for us, now we kind of shift gears where like Gradle is the main build system, Make is still around because Android platform uses all of our libraries, and we can't just drop the support because, well, they're funding us, so we can't do that. We introduced Doclava, which was the tool at the time that we use for documentation and API tracking. We added Android Lint, Android studio support and then we finally have presubmit, but we use Gradle both in presubmit and postsubmit and that's really helpful for us, because now we're like a lot more sure that our build doesn't break randomly. And this is the first time around where we actually started like looking at build performance because we were like, oh, well, the CI is not great, like what's going on? And we just bumped the heap to three gigs of ram and we're like, oh, problem solved. Now we're done. Probably not going to touch CI ever again. Well, that's not true. All right. 2017, we jump and we start growing projects. Now we added five more like we're at 41. And I think that kind of shows the pattern that we've seen at the talk this morning. For example, if like, you know, your build.gradle files are getting 300 lines a pop and you just copy pasting a single thing over and over again. And that's when we're like, okay, we're going to have a convention plugin kind of our own thing. So we shove it in the buildSrc. We wrote it in Groovy because that was exciting at the time. Mistakes. And then we also split our build.gradle in the root directory as well because it was doing a lot of things and it was kind of like doing very different things, right? There's publishing logic that was, you know, kind of repo set up and whatnot. So we did all of that. And we also enabled configuration on demand, which was kind of the incubating feature but it's still incubating Gradle. And then we're using that and it's making our builds better, right? Like our lines of code, you know, it's like getting sizable, you know, 300,000 lines of code. It's not a joke. So kind of a recap of where we are. Okay, so project counts going up, you know, steady rate, not crazy. We're not like growing, you know, hundreds of projects every time. And if we look at our build speed in presubmit, it's kind of flat, you know, it's nothing crazy. Right. We have not saturated the CI yet, right? CI has X number of cores, our project count is kind of like a little under that, like so we're not really like impacted as we're growing new projects. A little bit foreshadowing here. All right. We move forward. This is the time where we actually split our build into two. And I really don't love to say that, but we did that because we had a secret project, arch components at the time, which is like a set of libraries that we shipped for Android, and it was secret that we could launch it. We were developing in AOSP and public and then we had all this stuff but we didn't tell people about, so we split the build into two. It was a miserable experience for me as a build engineer, but hey, that's what we had to do. This is the first time we added Kotlin, again, it doesn't help the build speeds. We started really working on, like Maven plugin, kind of enhancing, you know, making sure we ship our sources. We started doing error static analysis of tests that were error prone. So all of this is kind of like slowing us down a little bit, right? But and then we also start, you know, like, what can we do to improve?

We started moving our Groovy code to Kotlin and Java, and we started finding out like, oh, like when you have like statically compiled language, you can actually get all these benefits of like not crashing at runtime randomly. And then we also started moving our library versions and dependencies into buildsrc, because like we'll have so much better than having it scattered across like hundreds of files. You know, I want update JUnis to a new version. I don't want to touch 400 files and you know, have an interesting time with merge conflicts and all of that good stuff. So that was a big kind of help for us. And then yet again, we just keep cranking the JVM args, you know, like we're going 4 gigabytes. Okay, good. That's good enough for everyone. And this is the first time kind of in our development where we got like a full time dedicated build engineer. Up until that, it was kind of like 20% project, like, whoever jumps in makes it better. Yeah, keep going. We're still split into two. Not great. Our project count is growing. We add jetifier, which, you know, if anyone is getting flashbacks about this, well, we added it to our own build because that's the year we migrated from Android.Support Library to AndroidX, and we package migrated everything, and it turns out that a lot of tooling had a lot of dependencies on having package hardcoded, so that was something we add to our build and it kind of made it slower. We decided that splitting our JVM test and a separate builder, great idea because well, we were doing everything in one big builder and if you pulled a test to do a separate thing, well, you could potentially, you know, double your speed depending on how slow your tests are. And then we also migrated all of our buildsrc to Kotlin, we decided that Groovy and Java was just like causing us pain. There's no reason to keep that. And then we started adding plugins that were kind of like one per type of project. So, you know, if you had a Java project you apply JavaX plugin, and if you had an Android project, you had AndroidX plugin or Android plugin and so on. So it allowed us to kind of like do a little bit of like, this is the flavor of project you are and this is the conventions that we want to apply for those. Kind of jump forward. We finally merged the builds back. And one thing that I kind of mentioned, did I not mentioned? I did. Anyway we merged it back. And the nice thing for us here is when we merged back, we also got a chance to kill Make. You probably didn't hear about this in the meantime, but like we were suffering all the way along, you know, adding/bolting on Kotlin into Make is not a fun experience. So, like, having two build languages, we have to kind of like support both of them at the same time was not great. So like merging it back and having like a one thing that we really care about that, it was really beneficial for us. As we added Kotlin, we also realized that we have to like start tracking it, right? Like you need to document it. So we had to include Dokka, yet another, two documentation tools. And then we added API tracking with metalava which is our tool that we wrote for tracking kind of Kotlin API service to make sure that we don't break binary compatibility for users. So now we have two API trackers, two documentation tools, and like, you know, we're adding, adding, you know, to all of this. This is the first time we enabled local cache for Gradle. Really annoying that Gradle has a default of not turning it on. You know, if you see anyone from Gradle here, you maybe you should nudge them to change that. Another thing we do is we also split out our test APKs for Android into a separate target because again, this is very similar to JVM host tests. Like if you can just build your tests and then start running them, you get a lot of benefit of like, you don't have to run Lint to be able to start running tests like splitting into separate targets. That was kind of like the pattern and you can kind of see it through time. We also created this thing called affected module detector, which you might have heard from Dropbox, but they kind of copied our code. But that's okay. An open source, that's how it works. But what it does essentially, if you give it a set of files, and I change three files, it will tell me what projects in Gradle are affected. Like, if I change this file, is any project depending on this or is the project itself changed. And allowed us to really narrow it down to what do we test? So it was like one thing we did, and then yet again, we cranked the JVM to eight gigabytes. You know, this is looking pretty good.

All right. We jumped a little bit again, we're really starting to grow in terms of projects. Now, we're close to 200 projects. So I feel like it's fairly sizable, especially for 2019. We started adding custom lint checks, you know lint, not really fast, you add more lint checks, well, turns out to be and gets even slower. We started tracking Android resources just because we want to make sure that if you expose a new string or a new style or something else in our AARs, the developers' take, we don't accidentally remove it and then break our developers. We started doing max deps version testing, what I mean by that is, normally we pin our libraries to use the lowest version of the library dependency we can. But of course that exposes us that, you know, if someone goes in to remove some API, you don't want to break on that, right? So what we do is we essentially dynamically have a separate build that swaps and inserts the essentially tip of tree dependency like ahead and being able to kind of do that testing to make sure that we don't break ourselves that way. For the optimization side, we start doing migration to lazy tasks kind of thing. That was novel at the time, but for us it was like, oh, like, you don't actually have to do all this work in configuration phase. So just kind of like moving on and like starting to fix all of it. And it also expanded our affected module detector onto a lot of more tasks. So like, for example, lint, like if your module has not been changed in the files that we touched in the PR, like why even bother running lint for those modules? So like that's another way kinda could cut down on time. This is a huge growth. So we grew 111 projects for us it was like drastic bump and this is because of compose. So if you might have heard of Jetpack Compose, which is our like UI framework. It was a big project. There's a lot of people working it. They wrote a lot of code. As you can see, our Kotlin jumped out to 270,000 lines of code, and of course, it made our builds, you know, a lot heavier. The build got split again because compose at the time was secret. This is the pattern you see, like secret projects, and keep it splitting the build. You know, it's not a fun time. But, you know, we started doing kind of like more things there. We started doing ktlint because we started adding so much code. We wanted to make sure that our code is consistent across. So ktlint runs there. We started doing build metrics on the library. So like, you know, how many methods are there in this library, how many how big is the jar? So we'll kind of like have over time graphs and we're able to kind of track, as we're growing. But then jumping on a kind of the optimization side. This is the first time we realized that like every laptop is not the same as every CI machine, so maybe you don't need to use the same JVM args between the two. So in here, the first thing that we actually switched on is, we made it so that CI always pre allocates all of the heap. There's no reason for us to grow because we know we're going to get to eight gigabytes. So that was a big thing and actually help us with the build. We also moved our to a single plugin, so that's kind of the reversal of our multiple plugins. We kind of learned that we ended up making like the same code changed to like seven different plugins. We merge it all into one and it's reactive to a plugin. You know, if Java plugin is applied, we do a little bit of work. If Kotlin plugin is applied, we do a little work and so on. We moved our build source tests. Another not so great decision by Gradle is that, if you have build source for tests, they will run before everything else. When you just want to do configuration of the main project. Move that out. Saves us some time. We also really focused on task up to dateness because we're like starting to use the cache more and more. So we really did like a lot of enforcement for task up to datenes. So essentially we set up a system where we run our build twice and the second run there should be no tasks. And of course there is. So we have an allow list of like, Oh, we know these tasks are bad and we slowly started clamping it down and it even helped us catch like regressions upstream where sometimes Android Gradle plugin goes and makes a thing that causes tasks not to be run correctly. So it's nice. Yeah. Let's keep going. Next thing is, we actually started doing testing. Surprise, surprise for our build logic. You know, it's sometimes not very obvious that you might want to do that, but Gradle provides you a Gradle test kit that allows you to kind of set up your plugin in the way that you can maybe have like a demo app or your demo library and then do the testing that way, so you have a much more predictable thing, as opposed to like, let's ship the production and see what happens. The other thing we started doing is it's really kind of containing Gradle, like all of the Gradle outputs, like Gradle home, Gradle temporary directory, like everything we shove in a single directory. And if someone wants to nuke that directory, we know we cleared everything. Whereas today if you, if you just use vanilla Gradle, you run a build, you clean your build directory and you run again. You're actually going to get stuff from different types of caches, transformation caches, download caches, you know, configuration cache, and so on. So we really started to honing down on like containing all of that and allowing us to do a very reproducible type of environment. And the other thing that was kind of like big brain moment for us is starting to be able to use workers. So our metalava, which is our API tracker, it was just a JVM process we spun off. It runs a lot of stuff and then it comes back at us and we kind of learned out of the workers, you can actually have multiple tasks run at the same time within the same project. And that's really helpful when you have many tasks. And we were tracking like public API service, internal API service, experimental API surface and it was like same task which is blocked on each other. And now we could use all of the cores we had. So it was a good move for us. Again, zooming out, let's look back on where we are. Right? The graph is pretty flat. And then we suddenly started really picking up and like, you know, even like the last half year, like 111 projects, it's like a big growth. And especially for us, you know, the build team didn't grow by the same proportion. It's the same build team and you know, kind of like, look, I tried to align the graphs a little bit here, but the code is going up. Java is pretty steady drumbeat, but Kotlin is really shooting up because of compose, right? So like we're really growing in that, and you can kind of see the build was flat for a while, the last graph you saw and it really started a climb up. So if we kind of zoom in a little bit off like what did happen in those little parts? And we'll kind of walk you through some anecdotal things that I found that were interesting. So A the point was when we merge arch components back into the main build, it turns out when you have two builds that are separate, they build pretty fast and if you merge them together, it gets slower because you have more projects. B was actually the drop in or we actually well sorry. So lower is better because there's a number of minutes in CI. So dropping down in B was eight gigabytes on ram of heap for us. So that was nice decrease. C was a regression when we upgrade Gradle 5.0 because they had performance regressions. D was up as well when we started tracking our internal API surfaces. Initially, we only checked our public service, but we also wanted to make sure that we don't break our libraries within each other. So we started tracking that. So that also kind of shot up. For E you can start to see like a very steady climb over there that's compose. We hired a lot of people to write a lot of code. It turns out that causes you builds to go up because they write a lot of code and a lot of Kotlin, which is like not very fast to compile and then F is a huge drop down. When we actually moved Kotlin compiler out of process, we had a bug that we hit specifically for CI where we couldn't run Kotlin out of process, like we couldn't have a Kotlin daemon. And finally that bug was fixed upstream and we're able to move back the separate process, and that gave Gradle Daemon way more breathing room in terms of memory and that made our build speed up drastically.

Right, keep going, 2020, keep moving forward. So the other thing here is you will notice is that we actually split AGP like Android Gradle plugin between compose on our main project. This is another pane when you split the build that people can start diverting. And so like, you know, if your monorepo, it's not a fun time to have manage like, oh, does it work this studio version, does it work on this one and so on. So that was becoming slightly painful. We started doing work of integration of Android Studio. Like if you use inspectors, for example, for Compose, we kind of have to do like cross team collaboration where we have to build some stuff in our repo and then also in Android studio and I'm kind of like started depending a lot between each other. We also started a builder of building like head of Android Gradle plugin and Android X to making sure that we don't regress. So like when the plugin evolves, we actually don't kind of break. And that was actually a big benefit to kind of the ecosystem, but you didn't get to see. But like when something breaks now I go and yell at the Gradle studio team like, you know, minutes after they broke something as opposed to, you know, it coming from Bug Tracker a months later. So we added support for Robo Electric that was also nice so that our developers could like run stuff, stuff just a little bit faster. And then on the kind of the optimization side, we enabled the remote cache for local builds due to Corp policy, which I'm sure some of you run into that we couldn't do it on the CI because we wanted to have builds that are very reproducible and they are not like going to the network and fetching things and so on. So at the time, we could only do local builds that were remote caching. So you know, Joe would go build and run a task, and then Mary when she goes and runs it, it goes in caches the same way. We spend a lot of time doing cache update, cacheability fixes, because when we started using the remote CI, we started actually seeing a lot of issues where it was issues of paths. So if you had, you know, a place, you know, in one place and then the other and we're hitting all sorts of problems there. And then we did a JDK 11 update and that was also very nice. Compose merge backend. Yay, awesome. We have one build again. We set up a GitHub mirror for contributions. So you know, plug if you want to contribute to our code base, and now you can also do it on GitHub, but it also actually added build complexity because normally our build runs completely in hermedic mode. Like when you download Gradle, all of the dependencies are in git. It's kind of like self-contained. You know, you go download it and you can like plug off the internet, then you're going to be able to build all of it. Whereas a GitHub, that's not really how it goes. So like being able to do like the flexible, like, oh, you can reach network now you can go download some of the dependencies, here's a verification we have to turn off and so on. So that kind of made our build more complex. We finally learned what input normalizers are. Turns out it took us to 2020 to get to know that. But actually that was causing a lot of invalidation for our tasks because we started building a compose compiler plugin for Kotlin and if you don't normalize, it will be a different jar for every local machine. And essentially it made us, you know, kind of have no cache hits. It's for anyone who was working on Compose and we kind of missed it because, well, we didn't really see it. Someone's machine of like, why is yours being so slow? So we have to kind of like learn about that and fix a lot of it. We rewrote a bunch of like our docs implementations here to kind of like make it an explicit list. Before we were trying to be clever doing dynamic docs. I would avoid that. I highly suggest having like a complete list of what you want to publish as a compromise for composing merging back into our main build. We also have to give them a latch or escape hatch for Studio, as some people you might have mentioned a couple of times like, Gradle sync can get real, real slow when you have that many projects. And here we're at 500 or 400, not great. So the escape hatch was essentially developing something similar to very, very similar to Dropbox focus, which essentially we say like, oh, I want to open like this set of projects in the set and now the compose team to continue having that nicer experience, only opening like a couple of hundred projects as opposed to the whole thing. Yeah. Do you see tweaking? You know, that's the pattern. We just keep going, you know, we enabled parallel GC. Turns out it's actually a better type of GC for JDK 11. And in CI we actually gave it more RAM. So it's now 16 gigs in CI, in local we have to keep it as 8, because turns out if you run Android Studio and Gradle, and then sometimes studio launches another daemon of Gradle because it's running a different Java version, you kind of use up a lot of RAM and like if you go top end Mac, there's a peak of how many gigs around you can get. We also started doing a lot more on build output validation. So like we essentially detect if it's a new type of error that we've never seen and we failed on build on you. And this is the first time we forced our, we have owners essentially files in Google, which essentially is like who can approve a change. And this is the first time we kind of use no set, no parent, which essentially means like nobody except for a select few people in a team can review our build changes because we ended up finding that people would accidentally go and shoot themselves in the foot because, well, you know, writing real code is sometimes hard. Right.

Let's see, we added a couple more features. We essentially was just kind of like bundling in some artifacts inside of the other. So it's not a huge growth. But like our project growth keeps going. Right. We are, you know, pretty sizable numbers here. The big thing here for us kind of in optimization side adopting version catalogs, we moved out of build source defining all of our dependencies to build catalogs and actually like helped us reduce build cache in validations by like a dramatic amount, right? Like not every bump now it validates everything. And then also CI, just keep cranking 24 gigs now. All right. Keep moving. This is the kind of monumental stuff for us, because Kotlin surpassed Java in terms of lines of code, which essentially kind of like shows, kind of the trend of where Android at least is going, is Kotlin first right? So that was kind of like the thing, but also means the Kotlin tooling is that much more important now for us. We enable dependency verification. It's kind of like our extended effort of like, trying to make sure that we know what we build and what we ship is actually the stuff we trust. We also started a third build tool, or documentation tool because why not? This one is called Dackka. This was a rewrite of Dokka that we had on top of Dokka plugin architecture. So a new build tool, now we have three. Let's see. Configuration caching in warn mode. We have to enable that or we wanted to enable it to get the benefits that we've seen from couple several people mention it, and we started fixing a lot of issues. So we're kind of slowly going down that list. We split our build source into two parts, public and private, and we only load on the class path of build.gradle the public bits and then we runtime inject the other ones, and helped us kind of reduce of what people are able to poke into our code, and also helps with validations. If you want to chat about that, catch me on a side and we can talk more. We also started tweaking Kotlin daemon arguments. It turns out now we have two daemons to worry about, so you also got to do that. And we started doing arguments there. JVM testing, yet another set of arguments. I don't know if you sensing the pattern, but JVM arguments very important. We have to bump that to 2 gigs because we started just like OOMing on random tests. So it's not great. And this is the time we actually run kind of a collaboration of Gradle of enabling Gradle Enterprise for our GitHub mirror of our project. And that's actually led to a fair number of remote cache restoration fixes where we were like accidentally using absolute paths or whatnot. Right if we're going. Ah, we added Max support. Well, Max shipped an arm version and so we needed all of our tooling, JDK and you know, everything that has like a native component to ARRs or AAPT and AGP and all of that good stuff. So we added that, you know, kind of in a way it makes you build more complicated. JVM for tests, go ups again. You know, it's that pattern you might have seen. This is the big one for us because we enable remote cache for Gradle in CI. It was a lot of battle to get the right approvals from the right people in the thing, but the impact was massive. We were able to turn off the affected modules detector, which is essentially a big hack around Gradle and we left it only for tests because we haven't quite solved that. And then the remote cache only pushes from CI and on all of the local developers only read. So that also helped us alot of reproducibility of issues. So, so it's not someone like local push some bad cache entry and, and now the rest of the team cant work. Yeah. And also this is a big push for like team to actually start adopting remote cache. Some people choose not to, but when you show them some numbers of like, oh, you can run all of build in like, you know, X number of minutes, then they're a lot more convinced, and then yeah, more fixes on remote cache restoration.

So today, where are we today, we are at 657 projects and over 2 million lines of code. I would say it's pretty sizable project. You've seen some slides flexing and this is like a science project for Google in a way. Like we have a mono repo that's a real mono repo and this is just like Android libraries. So for us, I feel like we've really grown. You know, where we started on like 8000 lines of code. So even through my tenure, it's been massive, right? We added JVM based layout lib testing. So screenshot tests that are running an Android device was kind of like more capabilities, slower build. We did the JDK 17 upgrade. Turns out that made our build 10% faster by just upgrading to JDK. Well, who knew? We actually turned down all the other documentation tools. We're down to one. Yeah, it actually simplifies the build, makes things better, but, you know, we're not ready. Like, the rewrite took a long time, and it's like one of those things, you know, migrations, you know, if someone tells you migrations, like, it's never as fast as you think it's going to be. And then, you know, the other thing that our team really started focusing on is profiling builds. Using YouTrack, like YourKit, or any Gradle profiler, you know, to start really analyzing your build as if it was an application, right? If it goes and does one thing, you might be like really wasteful. So people talk about configuration phase being slow in Gradle. Right. It might take 60 seconds. And if you look at what's going on in there, if you have 600 projects, any one expensive operation gets multiplied by the number of projects you have because it's currently run serially. And so looking and like finding bugs like that is pretty important. So an example of that gradle, core gradle, the tool was doing a string format in one of the operations where they wanted to log some data and now was costing a hundred milliseconds. Like there was a single line change that I submitted to upstream Gradle that like made our configuration 100 milliseconds faster might not seem a lot, but like if you have 100 of these, that's like a second like that. You're like really talking real numbers and then so like that's, that's kind of like the thing we really started focusing on as a team. So kind of taking a step back again, you know how we're growing. It was steady, it picked up. I was like, okay, well, fine. But like, we are not stopping. So I think it's one of those things where like it looks kind of scary when we look at the graph, something like that, right lines of code. Kotlin is just skyrocketing right is really going up in this you know overtaking Java. So let's look at some examples of kind of fun things that happened in the meantime. A, was a Kotlin 1.4 upgrade our builds got slower. Turns out newer version had a bug in the Gradle implementation. B, was actually upgraded to 7.6 of Gradle is actually improved our builds. So sometimes it is worth it. So that's the kind of thing we've got. You got to measure. If you don't measure, you're not going to know what your getting out of it. C, was an example where we moved to only building one variant of our projects. So by default, depending on what kind of command you run, you might build like release and debug. Well, it turns out in certain CI scenarios you only need only debug or only release. So that was kind of for us a big drop there. And then in D, you kind of see some crazy spikes. And that was we had some outages. It was not fun. But then the steady decrease was the enabling of remote cache. So we were like rolling it out per different builder and then it was slowly getting better and better. So that was pretty exciting to see. And I'm just not going to be like, Oh, they have a lot of projects. So this is the artifacts on Maven Google.com, that ship from our code base. So it's not just like, kind of like imaginary. I created 600 sample projects. Look how fast he's doing. Like we are shipping. I think the last time I looked it was a little over 400 artifacts, like unique Maven artifacts, not including versions, just like, you know, artifacts. So, you know, it's a scale not in the core project which has demo apps and test apps and all sorts of things. But like even the stuff that we ship. So we're, we're kind of going, what's like what's upcoming for us? I think for us, big one is going to be Gradle Project Isolation and you probably going to hear about this for next three years because it's going to take a while to launch. But it's like really focusing on that feature and should like look up and read what it is because it's going to affect you in the future. But it will allow essentially Gradle's configuration to run in parallel of all projects at the same time. So speed up an number of cores you have in similar for Gradle sync in studio, which is going to be huge for a lot of developers. The other thing is working on shadow builds, meaning that we want to test essentially against, tip of tree nightly of Gradle like are we going to break when gradle 8.0 ships, right? We want to be ready for that. Same of Kotlin gradle plugin, same of the next JDK and so on. So being able to kind of get ahead of it. So when you know when something breaks like, kind of like at a daily basis, you know, Gradle introduces a regression, we know the next day that it did happen, right? So that's the kind of thing and an updating JDK like we're in 17 would be nice if we run 19, right? Cool. Some takeaways, so, I think like everyone is convinced here, but like maybe not everyone outside of this room is convinced, but that productive engineers make better quality products. That's what we found within our team. If someone is not twiddling their thumbs or going off to Twitter or, you know, creating memes, like they end up doing a lot more productive work. So like, you know, going from, you know, a couple of minutes builds to a couple of seconds build. It makes a huge difference. The other thing is monitoring. You've got to monitor if you want to catch regressions, like if you currently don't have any sort of monitoring of how slow your build is. Maybe that's the first thing you do when you get back to your office, and that's the only way to catch regressions. And we've got many, like we've caught stuff in AGP and upstream Gradle, Kotlin Gradle plugin and so on, and we catch some stuff. But like your build might be unique and you might not get the benefit of what we do. Keep your tools up to date, JDK for example. Like, you know, if someone told you your build going to be 10% faster for like just upgrading to a new JDK, like, wouldn't you go and do that? Like, that's the kind of thing, you know? Same with Gradle know, you're going to want to use the latest version. They do a lot of improvements, sometimes regressions, but hopefully improvements. And then also read the release notes as those tools come out, like sometimes interesting features and things that you might poke at. And speaking of features, some of those features might be really painful to adopt, but it might be worth it. So, for example, configuration cache, it was a multiyear effort of trying to get it to being enabled. But once we got it, the impact was big. So, you know, that's the kind of thing to keep in mind. And so finally, again, still like a wrap up. Like all I want to kind of say is that, you know, if you have a project growth in terms of like code and everything, it doesn't mean that linear CI times know if you get ten times more code, it shouldn't be ten times slower build. That's kind of like, you know, if you take anything away, that's what you should take away. And it's kind of like one big elephant in the room. Right? Before I ask some more, ask for your questions, is I didn't talk about testing. I only talked about build. So testing on Android is kind of a pain in the butt because you have to run on physical devices for the most part or emulators, and inherently Android can only run one thing at a time. So you have to have an number of emulators or an number of devices to do any source of parallelization. And our graph is not looking hot. If we started forever ago, you know, we had ten minute to resubmit for tests. That was good times. I was looking at that and like I had a tear when I saw this graph. I was like, wow, that was good times. And now on average, this is average, not even p90, it takes 60 minutes for tests. Just the people, the developers are sitting there and waiting for CI to come back for an hour. Right. And this is like, we're like only detecting what needs to run. We only running small and medium tests and like doing all these optimizations and it's still kind of crappy experience. And P90 is even worse. Like, running all of our tests in AndroidX around 7 hours worth of device time. So it's wild, right? I mean, we do run like 60,000, I believe, Android tests. So it's a lot of tests. But, you know, that's the kind of thing. So what I want to say about this is we don't have a solution for this one yet. But, you know, that's the next focus for me and my team. And if anything exciting comes out of it, well, maybe I can come and do another talk about that. So I want to thank you for coming and listening to me. I see at least some of you are awake, so that's good. And then now it's the time to ask any questions, if you have any.