Structuring Large Projects

As software projects grow, it is common to organize large systems into components that are connected following a certain software architecture. Usually, it makes sense to also organize the artifacts (source code etc.) that make up the software in repositories and folder structures that reflect component boundaries and architecture. If Gradle is used to build such a software system, it can help you to perform this organisation and enforce boundaries between components.

You can look at Gradle as a modelling tool for your software: It allows you to describe the coarse grained structure and architecture of your software in a model written in Gradle’s DSLs. The build tool Gradle can then interpret this model to build the software.

An example

How you architect your software and how you divide it into components depends on what you are building. There is no one-fits-all solution. Hence, Gradle does not enforce a particular structure on you, but rather offers the tools to model your individual setup.

Still, to exemplify these features, we explore a sample project with the following architecture:

You can download the full sample to explore, build and run it.

The structure follows a typical setup that can be found in similar form in many commonly used software architectures.

At the bottom we define our domain model. There are two components: a domain-model component contains the model definition (i.e. a set of data classes) and a state component is responsible for managing a modifiable state of the model during application runtime.
On top of the model, business logic for different features is implemented independently of each other and independently of a concrete application technology. In this example, we have two features: user and admin.
At the top, we have concrete applications users use to interact with the features. In the example, we build a Spring Boot application that supports both features via a web browser. And an Android app that only supports the user feature.

Our components may rely on existing components that are retrieved from binary repositories. For example, the Spring Boot and Android frameworks.

Apart from the production code, there are also components that deal with building and delivering the product:

The build-logic component contains the configuration details about building the software. For example, defining a Java version to use or configuring a test framework. It may also contain additional build logic for Gradle (custom plugins with custom tasks) that is not covered by commonly available Gradle plugins.
The platforms component is a central place to define which versions of external components are to be used in all of our own components. By that, it defines the constraints for the environments – that is, the platforms – to build, test and run the software product.
The aggregation component contains the setup of the delivery pipeline that is required to push the product to production and doing automated end-to-end testing as part of that. Basically, this is the part of the build that is not required on local development machines.

The domain of our example is to build a tool to inform people about Gradle Build Tool releases. Concretely, the application lists Gradle releases with links to release notes (user feature) and offers an administration interface for the range of releases to be listed (admin feature).

Reflecting software architecture in project structure

Let’s look at how to implement the architecture of the sample with Gradle. We can represent each of our components as a separate Gradle build. We’ll get to the details of what that means and how components are connected in a bit.

Each Gradle build has its own folder. The minimum to make these folders Gradle builds is to add an empty settings.gradle(.kts) file to each of them. Let’s do this for all the components we have in our software:

├── android-app
│   └── settings.gradle.kts
├── server-application
│   └── settings.gradle.kts
│
├── admin-feature
│   └── settings.gradle.kts
├── user-feature
│   └── settings.gradle.kts
│
├── state
│   └── settings.gradle.kts
│
├── domain-model
│   └── settings.gradle.kts
│
├── build-logic
│   └── settings.gradle.kts
│
├── platforms
│   └── settings.gradle.kts
│
└── aggregation
    └── settings.gradle.kts

├── android-app
│   └── settings.gradle
├── server-application
│   └── settings.gradle
│
├── admin-feature
│   └── settings.gradle
├── user-feature
│   └── settings.gradle
│
├── state
│   └── settings.gradle
│
├── domain-model
│   └── settings.gradle
│
├── build-logic
│   └── settings.gradle
│
├── platforms
│   └── settings.gradle
│
└── aggregation
    └── settings.gradle

In the listing, each component lives in a separate folder. Here we arrange them as a flat list in a root folder. This root folder can be used as root of a Git repository for example.

This is only the setup of the sample. You can freely choose where to physically locate your components. For instance, you can group all components that live in one "layer" in a common subfolder. Or, since these are all independent Gradle builds, you can have each component live in a separate repository. It’s up to you to decide what works best for you, the software you are building, and the teams working on it.

Defining an inner structure for components

Before we get to the topic of connecting the components, let’s first look at them individually.

So far, each component is just an empty folder with an empty settings.gradle(.kts) file indicating that this is a component Gradle can work with in some form. To fill the component with content, you should define at least one project (referred to as subproject in Gradle’s DSLs) in it.

You can start with components consisting of a single project each, but introduce additional projects later to structure a single component more internally. In our sample, we start with a single project in each component.

A project is added by using the include() construct in the settings file.

Example 1. Use include() to add a project to a component

domain-model/settings.gradle.kts

include("release") // a project for data classes that represent software releases

domain-model/settings.gradle

include('release') // a project for data classes that represent software releases

Once included, you may create a folder matching the project name and create a build.gradle(.kts) file in it to configure that part of the component. You can find more information in the chapter about structuring Gradle builds for a single software component.

Assigning types to components

Let’s zoom into the domain-model component:

└── domain-model              <-- component
    ├── settings.gradle.kts   <-- define inner structure of component and where to locate other components
    └── release               <-- project in component
        └── build.gradle.kts  <-- defines type of the project and its dependencies

└── domain-model              <-- component
    ├── settings.gradle       <-- define inner structure of component and where to locate other components
    └── release               <-- project in component
        └── build.gradle      <-- defines type of the project and its dependencies

Initially, release/build.gradle(.kts) is empty. The project is of no specific type and does not offer any useful content. If we add more files to the domain-model/release folder now, for example Java source files, Gradle won’t know what to do with these files and will just ignore them. We need to assign a type to the project to make Gradle aware of the purpose of such files.

In Gradle, you assign a type to a project by applying a plugin. The simplest thing you can do is to apply one of Gradle’s core plugins, like base or java-library.

However, usually you have additional configuration to do in the context of the product you are building. For example, if your project should be a "Java Library", it would not only apply the java-library plugin but also configure details such as setting the Java version to 11. You can add details like that directly in release/build.gradle(.kts) but than you would have to repeat them in other components that also contain "Java Library" projects.

Thus, the recommendation is to start using custom project types right away:

Example 2. Assign the custom type com.example.java-library to a project in the user-feature component

user-feature/data/build.gradle.kts

plugins {
    id("com.example.java-library")
}

user-feature/data/build.gradle

plugins {
    id('com.example.java-library')
}

Example 3. Assign the custom type com.example.kotlin-library to a project in the domain-model component

domain-model/release/build.gradle.kts

plugins {
    id("com.example.kotlin-library")
}

domain-model/release/build.gradle

plugins {
    id('com.example.kotlin-library')
}

As stated above, a project type is represented by a plugin in Gradle. We thus define custom project types, such as com.example.java-library and com.example.kotlin-library, as plugins. The next section explains how to define such plugins.

Defining custom project types as convention plugins

Where do we get com.example.kotlin-library plugin from? This is what our build-logic component is for.

The build-logic component contains project types that Gradle itself understands as build configuration. That is, Gradle plugins for your specific needs, which we call convention plugins.

Currently, there are different project types you can use to define convention plugins depending on which tools and languages you prefer. In general, any JVM language (Java, Groovy, Kotlin, Scala) can be used to write Gradle plugins as classes that implement the Plugin<Project> interface. The most compact way however, is to write them as scripts in Gradle’s Groovy or Kotlin DSL.

Which method you choose is up to you. If you are familiar with one of Gradle’s DSLs you may choose that, as it is the most compact way to write convention plugins. If you are new to Gradle (and also new to Groovy and Kotlin) you may prefer to write the convention plugins in Java or another language like Scala. Then you reduce the interaction with Gradle’s Groovy or Kotlin DSL to a minimum.

You need to use one of the following project types (i.e. Gradle core plugins) in projects of your build-logic component:

kotlin-dsl – Build logic projects with this type (i.e., that apply the kotlin-dsl plugin) allow you to write convention plugins as .gradle.kts files in src/main/kotlin.
groovy-gradle-plugin – Build logic projects with this type (i.e., that apply the groovy-gradle-plugin plugin) allow you to write convention plugins as .gradle files in src/main/groovy.
java-gradle-plugin – Build logic projects with this type (i.e., that apply the java-gradle-plugin plugin) allow you to write convention plugins as .java classes that implement the Plugin<Project> interface in src/main/java. If you apply other JVM language plugins on top, like groovy, scala or org.jetbrains.kotlin.jvm, you can also write the plugin class in the corresponding language.

In our sample, we choose the option of using Gradle’s DSLs for the convention plugins. The build-logic component has several projects that each define a project type through a convention plugin - one of: java-library, kotlin-library, spring-application, android-application. Also, there is a project called commons for build configuration shared by all our project types.

Example 4. Assign a custom type to a component’s project by applying a convention plugin

build-logic/spring-boot-application/build.gradle.kts

plugins {
    `kotlin-dsl` (1)
}

dependencies {
    implementation(platform("com.example.platform:plugins-platform")) (2)

    implementation(project(":commons")) (3)

    implementation("org.springframework.boot:org.springframework.boot.gradle.plugin")  (4)
}

build-logic/spring-boot-application/build.gradle

plugins {
    id('groovy-gradle-plugin') (1)
}

dependencies {
    implementation(platform('com.example.platform:plugins-platform')) (2)

    implementation(project(':commons')) (3)

    implementation('org.springframework.boot:org.springframework.boot.gradle.plugin')  (4)
}

Looking at the build.gradle(.kts) of the build-logic project for spring boot applications, we see:

1	That it is of type `groovy-gradle-plugin` or `kotlin-dsl` to allow convention plugins written in the corresponding DSL
2	It depends on our own `plugins-platform` from the `platforms` component
3	It depends on the `commons` project from `build-logic` to have access to our own commons convention plugin
4	It depends on the Spring Boot Gradle plugin from the Gradle Plugin Portal so that we may apply that plugin to our Spring Boot projects

Now, we can write the convention plugin for Spring application like this:

Example 5. Defining a custom project type as convention plugin

build-logic/spring-boot-application/src/main/kotlin/com.example.spring-boot-application.gradle.kts

plugins {
    id("com.example.commons")
    id("org.springframework.boot")
}

dependencies {
    implementation("org.springframework.boot:spring-boot-starter-web")
    implementation("org.springframework.boot:spring-boot-starter-thymeleaf")
}

build-logic/spring-boot-application/src/main/groovy/com.example.spring-boot-application.gradle

plugins {
    id('com.example.commons')
    id('org.springframework.boot')
}

dependencies {
    implementation('org.springframework.boot:spring-boot-starter-web')
    implementation('org.springframework.boot:spring-boot-starter-thymeleaf')
}

We see that we apply our own com.example.commons plugin which is another convention plugin that, among other things, configures the Java version we are targeting and adds a dependency to a platform (com.example.platform:product-platform from our platforms component). And we apply the spring boot plugin. Furthermore, we add two dependencies that Spring Boot projects should always have in our context.

Similarly, we define convention plugins for "Java Libraries", "Kotlin Libraries" and "Android Applications". With that, we have four different project types defined that we assign to the projects of our production code components.

You can find more information about writing convention plugins in section on sharing build logic and the associated sample. For using classes to implement plugins, and for writing more advanced custom build logic, consult the chapter on Gradle plugin development.

Connecting components

As demonstrated in the architecture figure, our production code components depend on each other. Above, we already saw that the platforms component is used in the build-logic component. We also said that we want to use the build-logic component, which declares project types through convention plugins, to assign those types to the projects in our production code components.

How do you define these dependencies? There are two distinct things to do:

Make components (builds) known to each other. This is done by adding includeBuild(…) statements to settings.gradle(.kts). This is not adding a dependency between (projects of) components. It just makes the physical location of one component known to another. In that sense it is similar to repository declarations to discover binary components. Consult the section on defining composite builds for more information about how to include builds.
Declare dependencies between (projects of) components. This is done similarly to declaring dependencies to binary components by using GA (group and artifact) coordinates in the dependencies { } block of a build.gradle(.kts) file: implementation("com.example.platform:product-platform"). Or, if the included component provides a plugin, you apply the plugin by ID similar to how you would apply a plugin from the plugin portal: plugins { id("com.example.java-library") }

As another example, consider the setup of our server-application component:

Example 6. Defining component locations and inner structure of the server-application component

server-application/settings.gradle.kts

// == Define locations for build logic ==
pluginManagement {
    repositories {
        gradlePluginPortal()
    }
    includeBuild("../build-logic")
}

// == Define locations for components ==
dependencyResolutionManagement {
    repositories {
        mavenCentral()
    }
}
includeBuild("../platforms")
includeBuild("../user-feature")
includeBuild("../admin-feature")

// == Define the inner structure of this component ==
rootProject.name = "server-application" // the component name
include("app")

server-application/settings.gradle

// == Define locations for build logic ==
pluginManagement {
    repositories {
        gradlePluginPortal()
    }
    includeBuild('../build-logic')
}

// == Define locations for components ==
dependencyResolutionManagement {
    repositories {
        mavenCentral()
    }
}
includeBuild('../platforms')
includeBuild('../user-feature')
includeBuild('../admin-feature')

// == Define the inner structure of this component ==
rootProject.name = 'server-application' // the component name
include('app')

We see that the settings.gradle(.kts) file only defines location for build logic components, other production code components and the inner structure of the component. Only the build.gradle(.kts) file in the app project then defines actual dependencies by applying a convention plugin and utilizing the dependencies block.

Example 7. Configuring type and dependencies of the app project inside the server-application component

server-application/app/build.gradle.kts

plugins {
    id("com.example.spring-boot-application")
}

group = "${group}.server-application"

dependencies {
    implementation("com.example.myproduct.user-feature:table")
    implementation("com.example.myproduct.admin-feature:config")

    implementation("org.apache.juneau:juneau-marshall")
}

server-application/app/build.gradle

plugins {
    id('com.example.spring-boot-application')
}

group = "${group}.server-application"

dependencies {
    implementation('com.example.myproduct.user-feature:table')
    implementation('com.example.myproduct.admin-feature:config')

    implementation('org.apache.juneau:juneau-marshall')
}

The model of your software

That’s it. This chapter gave an overview of which techniques to use to structure a software project into components with Gradle by following a sample. Download the full sample to explore further details. The next chapter covers more details about how to work with and evolve this kind of project structure. The chapter on composite builds gives you more technical background about the capabilities build composition offers.

To summarize, if you follow the suggestions from this chapter, your setup should clearly separate the following concerns to give you a flexible and clean model of your software product:

Write compact build.gradle(.kts) files. While in traditional Gradle builds these files tend to grow and mix a lot of different concerns, the structure presented here keeps these files compact. In most cases they only declare a project type by applying a single convention plugin and dependencies in the dependencies {} block. They might include minimal project-specific configuration, but these should be kept as minimal as possible. This also makes builds less dependent on Gradle’s DSLs: If you put your build logic into convention plugins, you can write it directly in Java if you like.
Isolate cross-cutting technical concerns into project types. Technically motivated build configuration often cuts across the entire software architecture. Whether or not a project is a "Java Library" or "Kotlin Library" may be totally independent of where it is located in the hierarchy of your components. If you use convention plugins, this allows you to isolate the definition of such project types in a central place, while still reusing them wherever needed. (This is a huge advantage over so called cross project configuration, which has been popular with older Gradle versions, but is bound to the hierarchy of your project.)
Declare the origins of components in a central place. In this structure, the places where a build can find other components, independent of whether they are located in a binary repository or are available locally as other Gradle builds, are defined centrally in the settings.gradle(.kts) file. This makes it easy to change the origin of a component and move from a binary to a from-source version of a component. Note that there are different strategies to avoid duplicating this information in each settings.gradle(.kts) of each of your components.
Declare platforms in a central place. Having a platform component as in the example is optional. You could do things without one of these, e.g. by declaring dependency constraints directly in your convention plugins. However, platforms are a good option to ensure all the boundaries for the environment in which your software operates defined in a central place.