Learn about the enhancements AndroidX has made over the years to their increasingly complex Gradle build configuration. AndroidX is a large collection of Google OSS Kotlin and Java libraries built to make Android app developers’ lives easier. This talk will take you on a journey exploring how Gradle Build has evolved and grown from 12 to over 650 projects. Learn how Gradle Build Tool kept CI times and quality at bay.
Aurimas Liutikas demonstrates a wealth of tweaks, including adding capabilities like ktlint, migrating to tasks.register, adding dependency signature validation, enabling license checking for dependencies, setting up Github mirrors, and much more.
Aurimas Liutikas is a software engineer at Google working on AndroidX libraries. The need to write efficient code has been instilled in him through his work on Chrome for Android, and later Android OS itself. This drive continued when taking on the build and test infrastructure for AndroidX – one of the largest set of open source libraries built using Gradle. Aurimas enjoys doing talks, blog posts, and generally pushing the Gradle ecosystem forward.

Gradle Enterprise build and test acceleration technologies address the pain of unnecessary developer idle time waiting for builds and tests to complete.
Interested in Build Optimization? Try these next steps:
- Learn how CashApp used Gradle Enterprise observability features like geolocation metrics to manage their remote build cache at scale.
- See this short video about how to use performance profiling to analyze your build cache efficiency.
- Sign up for our free Build Cache Deep Dive trainings for Gradle and Maven to get live hands-on, instructor led training to speed up your projects builds.
Aurimas: Well. Hello, everyone. Hopefully you're in the right room and you
do want to hear about chasing Gradle build speeds because that's what I'm going
to talk about. Yeah. Thank you for the introduction. Yeah, I'll just get
started. All right. So you might have heard about Google. We're a little mom and
pop shop. We run a search engine. We also happened to build AndroidX libraries,
which are like a set of libraries that pretty much every Android app uses. So I
will be talking about that. Google does a lot of things. I'm going to be
focusing on that. And, you know, if you know, Google also doesn't really
necessarily build everything with Gradle, obviously there's the whole other
infrastructure, but I will be talking kind of the Gradle bits and what we use
there. Yeah, I was going to say about me being in Google for ten years. Six of
those have been on Android team and a lot of part of it was the working on
support library that eventually became AndroidX. I'm currently a team lead
working on builds, tests and a lot of policy enforcement, so API tracking and
whatnot. So I'm here to kind of tell you about the story of our growth and kind
of like more focus on like history, with focus of the build. And while you might
not get any really specific advice out of this talk, my hope is to inspire you
that your build speeds don't have to suck. Right? Like as your project goes. It
doesn't have to go to crap. Right. So that's. That's what I'm trying to do. So
if we go, we're going to get started. A little bit of concepts, at least the way
I think about it. So for me, you know, the way I want to think about it, you
know, there's things that make your build faster and the things that make a
build slower and you know, project size, lines of code, the number of projects
and so on affects your build speed and usually in a negative way. But that tends
to be like a thing that people want to do, they ship features, right? They want
to do that, but it makes their build slower, and then the opposite side kind of
like, on addition on that you can also make your build slower by adding more
capabilities. So if you want to run Ktlint, or you want to do some static
analysis, or you want to do, insert a thing you want to do, and that also makes
it slower potentially. And then the other side is kind of the optimizations of
the build. So what can you do to make it run faster? Right. And that's the
thing. Well, it's a delicate dance if you know, like how do you make it do all
the things you want to do fast? You know, it's tricky. But I think the part that
I've seen a lot of teams struggle with, is kind of the optimization step. It's
really as easy to add and you plugin, it's really easy to add new code. But, you
know, optimizing your build while it's not really sexy and it's also kind of
painful. So that's what we'll be talking about. So where did it all begin?
Support libraries started in 2011, so a while ago and the focus was on kind of
like extending what Android platform can do, and just kind of like provide some
like a layer of a shim so you don't have to worry about backwards compatibility
as much. It was 8000 lines of code when it started, a single library that was
shipped and was using Make the build system, which happened to be the build
system that the Android platform was using. So it's kind of like piggybacking on
everything we have already. And it was written by an Android engineer, Diane
Hackborn, which you might have, if you know Android Circle, she was there from
the very beginning. If you want to jump forward a little bit, you know, we added
four more projects. You know, this is jumping to 2013 and this is the first time
we add Gradle to our build, so now we have two parallel builds and if that's
ringing scary bells of like you have two build systems. Well that's what we had.
At this point we're at 52,000 lines of code, so we're like we're growing in size
as well. This is using Gradle 1.11, so maybe pre-history for many of you, and
then this is AGP pre 1.0, so this is like really old stuff. And then two build
systems, they said not a great thing. Every little library you add, you have to
have matching build files for one build system and another. And this is going to
become a story as we build. Yeah. So in 2015, this is when I joined the team, so
I will have a lot more context from there on out. We only use Gradle for
essentially creating a Maven archive, that's the only thing it was kind of
useful for at the time. No pre-summit testing whatsoever, like not even building
anything. All we did is post submit YOLO, you know, hopefully works and then you
know, and Make is the only build system too, like we don't even test Gradle,
like if someone broke the build well good luck. Sink it and learn that it
doesn't work anymore.
Jump forward and I will be doing this in half a year increments. When I say
next, that's where it comes and goes. We go by 15 projects. So like, you know,
it's a sizable growth. And then I will be showing lines of code as well along
the way. So you can kind of track of where we're at. At this point, 15 projects
we're really kind of like picking up speed here. Our Gradle system can actually
build device tests and run them. Well, you know, it seems very basic today for
you. At the time, it was really kind of like, oh, yeah, we can do this now
because we have to do Make and like, you know, all of a fun platform, things
that we had to do to make it work. So this is like advancement for us. Jumping
in, you know, half year forward we had a little bit more projects. But luckily
for us, now we kind of shift gears where like Gradle is the main build system,
Make is still around because Android platform uses all of our libraries, and we
can't just drop the support because, well, they're funding us, so we can't do
that. We introduced Doclava, which was the tool at the time that we use for
documentation and API tracking. We added Android Lint, Android studio support
and then we finally have presubmit, but we use Gradle both in presubmit and
postsubmit and that's really helpful for us, because now we're like a lot more
sure that our build doesn't break randomly. And this is the first time around
where we actually started like looking at build performance because we were
like, oh, well, the CI is not great, like what's going on? And we just bumped
the heap to three gigs of ram and we're like, oh, problem solved. Now we're
done. Probably not going to touch CI ever again. Well, that's not true. All
right. 2017, we jump and we start growing projects. Now we added five more like
we're at 41. And I think that kind of shows the pattern that we've seen at the
talk this morning. For example, if like, you know, your build.gradle files are
getting 300 lines a pop and you just copy pasting a single thing over and over
again. And that's when we're like, okay, we're going to have a convention plugin
kind of our own thing. So we shove it in the buildSrc. We wrote it in Groovy
because that was exciting at the time. Mistakes. And then we also split our
build.gradle in the root directory as well because it was doing a lot of things
and it was kind of like doing very different things, right? There's publishing
logic that was, you know, kind of repo set up and whatnot. So we did all of
that. And we also enabled configuration on demand, which was kind of the
incubating feature but it's still incubating Gradle. And then we're using that
and it's making our builds better, right? Like our lines of code, you know, it's
like getting sizable, you know, 300,000 lines of code. It's not a joke. So kind
of a recap of where we are. Okay, so project counts going up, you know, steady
rate, not crazy. We're not like growing, you know, hundreds of projects every
time. And if we look at our build speed in presubmit, it's kind of flat, you
know, it's nothing crazy. Right. We have not saturated the CI yet, right? CI has
X number of cores, our project count is kind of like a little under that, like
so we're not really like impacted as we're growing new projects. A little bit
foreshadowing here. All right. We move forward. This is the time where we
actually split our build into two. And I really don't love to say that, but we
did that because we had a secret project, arch components at the time, which is
like a set of libraries that we shipped for Android, and it was secret that we
could launch it. We were developing in AOSP and public and then we had all this
stuff but we didn't tell people about, so we split the build into two. It was a
miserable experience for me as a build engineer, but hey, that's what we had to
do. This is the first time we added Kotlin, again, it doesn't help the build
speeds. We started really working on, like Maven plugin, kind of enhancing, you
know, making sure we ship our sources. We started doing error static analysis of
tests that were error prone. So all of this is kind of like slowing us down a
little bit, right? But and then we also start, you know, like, what can we do to
improve?
We started moving our Groovy code to Kotlin and Java, and we started finding out
like, oh, like when you have like statically compiled language, you can actually
get all these benefits of like not crashing at runtime randomly. And then we
also started moving our library versions and dependencies into buildsrc, because
like we'll have so much better than having it scattered across like hundreds of
files. You know, I want update JUnis to a new version. I don't want to touch 400
files and you know, have an interesting time with merge conflicts and all of
that good stuff. So that was a big kind of help for us. And then yet again, we
just keep cranking the JVM args, you know, like we're going 4 gigabytes. Okay,
good. That's good enough for everyone. And this is the first time kind of in our
development where we got like a full time dedicated build engineer. Up until
that, it was kind of like 20% project, like, whoever jumps in makes it better.
Yeah, keep going. We're still split into two. Not great. Our project count is
growing. We add jetifier, which, you know, if anyone is getting flashbacks about
this, well, we added it to our own build because that's the year we migrated
from Android.Support Library to AndroidX, and we package migrated everything,
and it turns out that a lot of tooling had a lot of dependencies on having
package hardcoded, so that was something we add to our build and it kind of made
it slower. We decided that splitting our JVM test and a separate builder, great
idea because well, we were doing everything in one big builder and if you pulled
a test to do a separate thing, well, you could potentially, you know, double
your speed depending on how slow your tests are. And then we also migrated all
of our buildsrc to Kotlin, we decided that Groovy and Java was just like causing
us pain. There's no reason to keep that. And then we started adding plugins that
were kind of like one per type of project. So, you know, if you had a Java
project you apply JavaX plugin, and if you had an Android project, you had
AndroidX plugin or Android plugin and so on. So it allowed us to kind of like do
a little bit of like, this is the flavor of project you are and this is the
conventions that we want to apply for those. Kind of jump forward. We finally
merged the builds back. And one thing that I kind of mentioned, did I not
mentioned? I did. Anyway we merged it back. And the nice thing for us here is
when we merged back, we also got a chance to kill Make. You probably didn't hear
about this in the meantime, but like we were suffering all the way along, you
know, adding/bolting on Kotlin into Make is not a fun experience. So, like,
having two build languages, we have to kind of like support both of them at the
same time was not great. So like merging it back and having like a one thing
that we really care about that, it was really beneficial for us. As we added
Kotlin, we also realized that we have to like start tracking it, right? Like you
need to document it. So we had to include Dokka, yet another, two documentation
tools. And then we added API tracking with metalava which is our tool that we
wrote for tracking kind of Kotlin API service to make sure that we don't break
binary compatibility for users. So now we have two API trackers, two
documentation tools, and like, you know, we're adding, adding, you know, to all
of this. This is the first time we enabled local cache for Gradle. Really
annoying that Gradle has a default of not turning it on. You know, if you see
anyone from Gradle here, you maybe you should nudge them to change that. Another
thing we do is we also split out our test APKs for Android into a separate
target because again, this is very similar to JVM host tests. Like if you can
just build your tests and then start running them, you get a lot of benefit of
like, you don't have to run Lint to be able to start running tests like
splitting into separate targets. That was kind of like the pattern and you can
kind of see it through time. We also created this thing called affected module
detector, which you might have heard from Dropbox, but they kind of copied our
code. But that's okay. An open source, that's how it works. But what it does
essentially, if you give it a set of files, and I change three files, it will
tell me what projects in Gradle are affected. Like, if I change this file, is
any project depending on this or is the project itself changed. And allowed us
to really narrow it down to what do we test? So it was like one thing we did,
and then yet again, we cranked the JVM to eight gigabytes. You know, this is
looking pretty good.
All right. We jumped a little bit again, we're really starting to grow in terms
of projects. Now, we're close to 200 projects. So I feel like it's fairly
sizable, especially for 2019. We started adding custom lint checks, you know
lint, not really fast, you add more lint checks, well, turns out to be and gets
even slower. We started tracking Android resources just because we want to make
sure that if you expose a new string or a new style or something else in our
AARs, the developers' take, we don't accidentally remove it and then break our
developers. We started doing max deps version testing, what I mean by that is,
normally we pin our libraries to use the lowest version of the library
dependency we can. But of course that exposes us that, you know, if someone goes
in to remove some API, you don't want to break on that, right? So what we do is
we essentially dynamically have a separate build that swaps and inserts the
essentially tip of tree dependency like ahead and being able to kind of do that
testing to make sure that we don't break ourselves that way. For the
optimization side, we start doing migration to lazy tasks kind of thing. That
was novel at the time, but for us it was like, oh, like, you don't actually have
to do all this work in configuration phase. So just kind of like moving on and
like starting to fix all of it. And it also expanded our affected module
detector onto a lot of more tasks. So like, for example, lint, like if your
module has not been changed in the files that we touched in the PR, like why
even bother running lint for those modules? So like that's another way kinda
could cut down on time. This is a huge growth. So we grew 111 projects for us it
was like drastic bump and this is because of compose. So if you might have heard
of Jetpack Compose, which is our like UI framework. It was a big project.
There's a lot of people working it. They wrote a lot of code. As you can see,
our Kotlin jumped out to 270,000 lines of code, and of course, it made our
builds, you know, a lot heavier. The build got split again because compose at
the time was secret. This is the pattern you see, like secret projects, and keep
it splitting the build. You know, it's not a fun time. But, you know, we started
doing kind of like more things there. We started doing ktlint because we started
adding so much code. We wanted to make sure that our code is consistent across.
So ktlint runs there. We started doing build metrics on the library. So like,
you know, how many methods are there in this library, how many how big is the
jar? So we'll kind of like have over time graphs and we're able to kind of
track, as we're growing. But then jumping on a kind of the optimization side.
This is the first time we realized that like every laptop is not the same as
every CI machine, so maybe you don't need to use the same JVM args between the
two. So in here, the first thing that we actually switched on is, we made it so
that CI always pre allocates all of the heap. There's no reason for us to grow
because we know we're going to get to eight gigabytes. So that was a big thing
and actually help us with the build. We also moved our to a single plugin, so
that's kind of the reversal of our multiple plugins. We kind of learned that we
ended up making like the same code changed to like seven different plugins. We
merge it all into one and it's reactive to a plugin. You know, if Java plugin is
applied, we do a little bit of work. If Kotlin plugin is applied, we do a little
work and so on. We moved our build source tests. Another not so great decision
by Gradle is that, if you have build source for tests, they will run before
everything else. When you just want to do configuration of the main project.
Move that out. Saves us some time. We also really focused on task up to dateness
because we're like starting to use the cache more and more. So we really did
like a lot of enforcement for task up to datenes. So essentially we set up a
system where we run our build twice and the second run there should be no tasks.
And of course there is. So we have an allow list of like, Oh, we know these
tasks are bad and we slowly started clamping it down and it even helped us catch
like regressions upstream where sometimes Android Gradle plugin goes and makes a
thing that causes tasks not to be run correctly. So it's nice. Yeah. Let's keep
going. Next thing is, we actually started doing testing. Surprise, surprise for
our build logic. You know, it's sometimes not very obvious that you might want
to do that, but Gradle provides you a Gradle test kit that allows you to kind of
set up your plugin in the way that you can maybe have like a demo app or your
demo library and then do the testing that way, so you have a much more
predictable thing, as opposed to like, let's ship the production and see what
happens. The other thing we started doing is it's really kind of containing
Gradle, like all of the Gradle outputs, like Gradle home, Gradle temporary
directory, like everything we shove in a single directory. And if someone wants
to nuke that directory, we know we cleared everything. Whereas today if you, if
you just use vanilla Gradle, you run a build, you clean your build directory and
you run again. You're actually going to get stuff from different types of
caches, transformation caches, download caches, you know, configuration cache,
and so on. So we really started to honing down on like containing all of that
and allowing us to do a very reproducible type of environment. And the other
thing that was kind of like big brain moment for us is starting to be able to
use workers. So our metalava, which is our API tracker, it was just a JVM
process we spun off. It runs a lot of stuff and then it comes back at us and we
kind of learned out of the workers, you can actually have multiple tasks run at
the same time within the same project. And that's really helpful when you have
many tasks. And we were tracking like public API service, internal API service,
experimental API surface and it was like same task which is blocked on each
other. And now we could use all of the cores we had. So it was a good move for
us. Again, zooming out, let's look back on where we are. Right? The graph is
pretty flat. And then we suddenly started really picking up and like, you know,
even like the last half year, like 111 projects, it's like a big growth. And
especially for us, you know, the build team didn't grow by the same proportion.
It's the same build team and you know, kind of like, look, I tried to align the
graphs a little bit here, but the code is going up. Java is pretty steady
drumbeat, but Kotlin is really shooting up because of compose, right? So like
we're really growing in that, and you can kind of see the build was flat for a
while, the last graph you saw and it really started a climb up. So if we kind of
zoom in a little bit off like what did happen in those little parts? And we'll
kind of walk you through some anecdotal things that I found that were
interesting. So A the point was when we merge arch components back into the main
build, it turns out when you have two builds that are separate, they build
pretty fast and if you merge them together, it gets slower because you have more
projects. B was actually the drop in or we actually well sorry. So lower is
better because there's a number of minutes in CI. So dropping down in B was
eight gigabytes on ram of heap for us. So that was nice decrease. C was a
regression when we upgrade Gradle 5.0 because they had performance regressions.
D was up as well when we started tracking our internal API surfaces. Initially,
we only checked our public service, but we also wanted to make sure that we
don't break our libraries within each other. So we started tracking that. So
that also kind of shot up. For E you can start to see like a very steady climb
over there that's compose. We hired a lot of people to write a lot of code. It
turns out that causes you builds to go up because they write a lot of code and a
lot of Kotlin, which is like not very fast to compile and then F is a huge drop
down. When we actually moved Kotlin compiler out of process, we had a bug that
we hit specifically for CI where we couldn't run Kotlin out of process, like we
couldn't have a Kotlin daemon. And finally that bug was fixed upstream and we're
able to move back the separate process, and that gave Gradle Daemon way more
breathing room in terms of memory and that made our build speed up drastically.
Right, keep going, 2020, keep moving forward. So the other thing here is you
will notice is that we actually split AGP like Android Gradle plugin between
compose on our main project. This is another pane when you split the build that
people can start diverting. And so like, you know, if your monorepo, it's not a
fun time to have manage like, oh, does it work this studio version, does it work
on this one and so on. So that was becoming slightly painful. We started doing
work of integration of Android Studio. Like if you use inspectors, for example,
for Compose, we kind of have to do like cross team collaboration where we have
to build some stuff in our repo and then also in Android studio and I'm kind of
like started depending a lot between each other. We also started a builder of
building like head of Android Gradle plugin and Android X to making sure that we
don't regress. So like when the plugin evolves, we actually don't kind of break.
And that was actually a big benefit to kind of the ecosystem, but you didn't get
to see. But like when something breaks now I go and yell at the Gradle studio
team like, you know, minutes after they broke something as opposed to, you know,
it coming from Bug Tracker a months later. So we added support for Robo Electric
that was also nice so that our developers could like run stuff, stuff just a
little bit faster. And then on the kind of the optimization side, we enabled the
remote cache for local builds due to Corp policy, which I'm sure some of you run
into that we couldn't do it on the CI because we wanted to have builds that are
very reproducible and they are not like going to the network and fetching things
and so on. So at the time, we could only do local builds that were remote
caching. So you know, Joe would go build and run a task, and then Mary when she
goes and runs it, it goes in caches the same way. We spend a lot of time doing
cache update, cacheability fixes, because when we started using the remote CI,
we started actually seeing a lot of issues where it was issues of paths. So if
you had, you know, a place, you know, in one place and then the other and we're
hitting all sorts of problems there. And then we did a JDK 11 update and that
was also very nice. Compose merge backend. Yay, awesome. We have one build
again. We set up a GitHub mirror for contributions. So you know, plug if you
want to contribute to our code base, and now you can also do it on GitHub, but
it also actually added build complexity because normally our build runs
completely in hermedic mode. Like when you download Gradle, all of the
dependencies are in git. It's kind of like self-contained. You know, you go
download it and you can like plug off the internet, then you're going to be able
to build all of it. Whereas a GitHub, that's not really how it goes. So like
being able to do like the flexible, like, oh, you can reach network now you can
go download some of the dependencies, here's a verification we have to turn off
and so on. So that kind of made our build more complex. We finally learned what
input normalizers are. Turns out it took us to 2020 to get to know that. But
actually that was causing a lot of invalidation for our tasks because we started
building a compose compiler plugin for Kotlin and if you don't normalize, it
will be a different jar for every local machine. And essentially it made us, you
know, kind of have no cache hits. It's for anyone who was working on Compose and
we kind of missed it because, well, we didn't really see it. Someone's machine
of like, why is yours being so slow? So we have to kind of like learn about that
and fix a lot of it. We rewrote a bunch of like our docs implementations here to
kind of like make it an explicit list. Before we were trying to be clever doing
dynamic docs. I would avoid that. I highly suggest having like a complete list
of what you want to publish as a compromise for composing merging back into our
main build. We also have to give them a latch or escape hatch for Studio, as
some people you might have mentioned a couple of times like, Gradle sync can get
real, real slow when you have that many projects. And here we're at 500 or 400,
not great. So the escape hatch was essentially developing something similar to
very, very similar to Dropbox focus, which essentially we say like, oh, I want
to open like this set of projects in the set and now the compose team to
continue having that nicer experience, only opening like a couple of hundred
projects as opposed to the whole thing. Yeah. Do you see tweaking? You know,
that's the pattern. We just keep going, you know, we enabled parallel GC. Turns
out it's actually a better type of GC for JDK 11. And in CI we actually gave it
more RAM. So it's now 16 gigs in CI, in local we have to keep it as 8, because
turns out if you run Android Studio and Gradle, and then sometimes studio
launches another daemon of Gradle because it's running a different Java version,
you kind of use up a lot of RAM and like if you go top end Mac, there's a peak
of how many gigs around you can get. We also started doing a lot more on build
output validation. So like we essentially detect if it's a new type of error
that we've never seen and we failed on build on you. And this is the first time
we forced our, we have owners essentially files in Google, which essentially is
like who can approve a change. And this is the first time we kind of use no set,
no parent, which essentially means like nobody except for a select few people in
a team can review our build changes because we ended up finding that people
would accidentally go and shoot themselves in the foot because, well, you know,
writing real code is sometimes hard. Right.
Let's see, we added a couple more features. We essentially was just kind of like
bundling in some artifacts inside of the other. So it's not a huge growth. But
like our project growth keeps going. Right. We are, you know, pretty sizable
numbers here. The big thing here for us kind of in optimization side adopting
version catalogs, we moved out of build source defining all of our dependencies
to build catalogs and actually like helped us reduce build cache in validations
by like a dramatic amount, right? Like not every bump now it validates
everything. And then also CI, just keep cranking 24 gigs now. All right. Keep
moving. This is the kind of monumental stuff for us, because Kotlin surpassed
Java in terms of lines of code, which essentially kind of like shows, kind of
the trend of where Android at least is going, is Kotlin first right? So that was
kind of like the thing, but also means the Kotlin tooling is that much more
important now for us. We enable dependency verification. It's kind of like our
extended effort of like, trying to make sure that we know what we build and what
we ship is actually the stuff we trust. We also started a third build tool, or
documentation tool because why not? This one is called Dackka. This was a
rewrite of Dokka that we had on top of Dokka plugin architecture. So a new build
tool, now we have three. Let's see. Configuration caching in warn mode. We have
to enable that or we wanted to enable it to get the benefits that we've seen
from couple several people mention it, and we started fixing a lot of issues. So
we're kind of slowly going down that list. We split our build source into two
parts, public and private, and we only load on the class path of build.gradle
the public bits and then we runtime inject the other ones, and helped us kind of
reduce of what people are able to poke into our code, and also helps with
validations. If you want to chat about that, catch me on a side and we can talk
more. We also started tweaking Kotlin daemon arguments. It turns out now we have
two daemons to worry about, so you also got to do that. And we started doing
arguments there. JVM testing, yet another set of arguments. I don't know if you
sensing the pattern, but JVM arguments very important. We have to bump that to 2
gigs because we started just like OOMing on random tests. So it's not great. And
this is the time we actually run kind of a collaboration of Gradle of enabling
Gradle Enterprise for our GitHub mirror of our project. And that's actually led
to a fair number of remote cache restoration fixes where we were like
accidentally using absolute paths or whatnot. Right if we're going. Ah, we added
Max support. Well, Max shipped an arm version and so we needed all of our
tooling, JDK and you know, everything that has like a native component to ARRs
or AAPT and AGP and all of that good stuff. So we added that, you know, kind of
in a way it makes you build more complicated. JVM for tests, go ups again. You
know, it's that pattern you might have seen. This is the big one for us because
we enable remote cache for Gradle in CI. It was a lot of battle to get the right
approvals from the right people in the thing, but the impact was massive. We
were able to turn off the affected modules detector, which is essentially a big
hack around Gradle and we left it only for tests because we haven't quite solved
that. And then the remote cache only pushes from CI and on all of the local
developers only read. So that also helped us alot of reproducibility of issues.
So, so it's not someone like local push some bad cache entry and, and now the
rest of the team cant work. Yeah. And also this is a big push for like team to
actually start adopting remote cache. Some people choose not to, but when you
show them some numbers of like, oh, you can run all of build in like, you know,
X number of minutes, then they're a lot more convinced, and then yeah, more
fixes on remote cache restoration.
So today, where are we today, we are at 657 projects and over 2 million lines of
code. I would say it's pretty sizable project. You've seen some slides flexing
and this is like a science project for Google in a way. Like we have a mono repo
that's a real mono repo and this is just like Android libraries. So for us, I
feel like we've really grown. You know, where we started on like 8000 lines of
code. So even through my tenure, it's been massive, right? We added JVM based
layout lib testing. So screenshot tests that are running an Android device was
kind of like more capabilities, slower build. We did the JDK 17 upgrade. Turns
out that made our build 10% faster by just upgrading to JDK. Well, who knew? We
actually turned down all the other documentation tools. We're down to one. Yeah,
it actually simplifies the build, makes things better, but, you know, we're not
ready. Like, the rewrite took a long time, and it's like one of those things,
you know, migrations, you know, if someone tells you migrations, like, it's
never as fast as you think it's going to be. And then, you know, the other thing
that our team really started focusing on is profiling builds. Using YouTrack,
like YourKit, or any Gradle profiler, you know, to start really analyzing your
build as if it was an application, right? If it goes and does one thing, you
might be like really wasteful. So people talk about configuration phase being
slow in Gradle. Right. It might take 60 seconds. And if you look at what's going
on in there, if you have 600 projects, any one expensive operation gets
multiplied by the number of projects you have because it's currently run
serially. And so looking and like finding bugs like that is pretty important. So
an example of that gradle, core gradle, the tool was doing a string format in
one of the operations where they wanted to log some data and now was costing a
hundred milliseconds. Like there was a single line change that I submitted to
upstream Gradle that like made our configuration 100 milliseconds faster might
not seem a lot, but like if you have 100 of these, that's like a second like
that. You're like really talking real numbers and then so like that's, that's
kind of like the thing we really started focusing on as a team. So kind of
taking a step back again, you know how we're growing. It was steady, it picked
up. I was like, okay, well, fine. But like, we are not stopping. So I think it's
one of those things where like it looks kind of scary when we look at the graph,
something like that, right lines of code. Kotlin is just skyrocketing right is
really going up in this you know overtaking Java. So let's look at some examples
of kind of fun things that happened in the meantime. A, was a Kotlin 1.4 upgrade
our builds got slower. Turns out newer version had a bug in the Gradle
implementation. B, was actually upgraded to 7.6 of Gradle is actually improved
our builds. So sometimes it is worth it. So that's the kind of thing we've got.
You got to measure. If you don't measure, you're not going to know what your
getting out of it. C, was an example where we moved to only building one variant
of our projects. So by default, depending on what kind of command you run, you
might build like release and debug. Well, it turns out in certain CI scenarios
you only need only debug or only release. So that was kind of for us a big drop
there. And then in D, you kind of see some crazy spikes. And that was we had
some outages. It was not fun. But then the steady decrease was the enabling of
remote cache. So we were like rolling it out per different builder and then it
was slowly getting better and better. So that was pretty exciting to see. And
I'm just not going to be like, Oh, they have a lot of projects. So this is the
artifacts on Maven Google.com, that ship from our code base. So it's not just
like, kind of like imaginary. I created 600 sample projects. Look how fast he's
doing. Like we are shipping. I think the last time I looked it was a little over
400 artifacts, like unique Maven artifacts, not including versions, just like,
you know, artifacts. So, you know, it's a scale not in the core project which
has demo apps and test apps and all sorts of things. But like even the stuff
that we ship. So we're, we're kind of going, what's like what's upcoming for us?
I think for us, big one is going to be Gradle Project Isolation and you probably
going to hear about this for next three years because it's going to take a while
to launch. But it's like really focusing on that feature and should like look up
and read what it is because it's going to affect you in the future. But it will
allow essentially Gradle's configuration to run in parallel of all projects at
the same time. So speed up an number of cores you have in similar for Gradle
sync in studio, which is going to be huge for a lot of developers. The other
thing is working on shadow builds, meaning that we want to test essentially
against, tip of tree nightly of Gradle like are we going to break when gradle
8.0 ships, right? We want to be ready for that. Same of Kotlin gradle plugin,
same of the next JDK and so on. So being able to kind of get ahead of it. So
when you know when something breaks like, kind of like at a daily basis, you
know, Gradle introduces a regression, we know the next day that it did happen,
right? So that's the kind of thing and an updating JDK like we're in 17 would be
nice if we run 19, right? Cool. Some takeaways, so, I think like everyone is
convinced here, but like maybe not everyone outside of this room is convinced,
but that productive engineers make better quality products. That's what we found
within our team. If someone is not twiddling their thumbs or going off to
Twitter or, you know, creating memes, like they end up doing a lot more
productive work. So like, you know, going from, you know, a couple of minutes
builds to a couple of seconds build. It makes a huge difference. The other thing
is monitoring. You've got to monitor if you want to catch regressions, like if
you currently don't have any sort of monitoring of how slow your build is. Maybe
that's the first thing you do when you get back to your office, and that's the
only way to catch regressions. And we've got many, like we've caught stuff in
AGP and upstream Gradle, Kotlin Gradle plugin and so on, and we catch some
stuff. But like your build might be unique and you might not get the benefit of
what we do. Keep your tools up to date, JDK for example. Like, you know, if
someone told you your build going to be 10% faster for like just upgrading to a
new JDK, like, wouldn't you go and do that? Like, that's the kind of thing, you
know? Same with Gradle know, you're going to want to use the latest version.
They do a lot of improvements, sometimes regressions, but hopefully
improvements. And then also read the release notes as those tools come out, like
sometimes interesting features and things that you might poke at. And speaking
of features, some of those features might be really painful to adopt, but it
might be worth it. So, for example, configuration cache, it was a multiyear
effort of trying to get it to being enabled. But once we got it, the impact was
big. So, you know, that's the kind of thing to keep in mind. And so finally,
again, still like a wrap up. Like all I want to kind of say is that, you know,
if you have a project growth in terms of like code and everything, it doesn't
mean that linear CI times know if you get ten times more code, it shouldn't be
ten times slower build. That's kind of like, you know, if you take anything
away, that's what you should take away. And it's kind of like one big elephant
in the room. Right? Before I ask some more, ask for your questions, is I didn't
talk about testing. I only talked about build. So testing on Android is kind of
a pain in the butt because you have to run on physical devices for the most part
or emulators, and inherently Android can only run one thing at a time. So you
have to have an number of emulators or an number of devices to do any source of
parallelization. And our graph is not looking hot. If we started forever ago,
you know, we had ten minute to resubmit for tests. That was good times. I was
looking at that and like I had a tear when I saw this graph. I was like, wow,
that was good times. And now on average, this is average, not even p90, it takes
60 minutes for tests. Just the people, the developers are sitting there and
waiting for CI to come back for an hour. Right. And this is like, we're like
only detecting what needs to run. We only running small and medium tests and
like doing all these optimizations and it's still kind of crappy experience. And
P90 is even worse. Like, running all of our tests in AndroidX around 7 hours
worth of device time. So it's wild, right? I mean, we do run like 60,000, I
believe, Android tests. So it's a lot of tests. But, you know, that's the kind
of thing. So what I want to say about this is we don't have a solution for this
one yet. But, you know, that's the next focus for me and my team. And if
anything exciting comes out of it, well, maybe I can come and do another talk
about that. So I want to thank you for coming and listening to me. I see at
least some of you are awake, so that's good. And then now it's the time to ask
any questions, if you have any.