Build Scan Storage

Build Scan data storage management is critical for ensuring the long-term availability of build data while controlling disk usage.

Develocity stores Build Scan data either in its database or in Object Storage. By default, it uses the database that Develocity is configured to connect to.

 To store Build Scan data in Object Storage, Develocity must connect to external Object Storage. The embedded Object Storage that Develocity provides cannot hold Build Scan data.

Disk Space Management

In most installations, Build Scan data is the largest consumer of storage space. How much space it uses depends on how many builds publish a Build Scan and how much information each build records. In the database, Build Scan data is compressed and deduplicated, so its growth is non-linear. Storing data for twice as many builds does not take twice the space. In Object Storage, each Build Scan is stored as a compressed, self-contained object. That makes the required storage easier to estimate: doubling the Build Scan count roughly doubles the storage.

To control the amount of storage space used, we recommend that you configure Develocity to remove Build Scan data by age. To avoid running out of disk space, set a free-space percentage at which Develocity takes each of these actions:

  • Delete old Build Scan data automatically.

  • Reject incoming data.

You can also have Develocity send a warning email when free space drops below a percentage that you set.

These settings can be changed by going to “Administration” via the top right user menu, then “Build Scans”. They can also be configured through unattended configuration or inline in your Helm values. See the configuration reference for examples.

A configuration that maintains a predictable Build Scan retention period is:

  1. Specify a maximum Build Scan age

  2. Send a warning email when there is less than 10% of space free

  3. Reject incoming data when there is less than 5% of space free

When storing scans in the database, an alternative configuration that stores as much Build Scan history as space permits is:

  1. Do not specify a maximum Build Scan age

  2. Automatically delete old Build Scan data when there is less than 15% of space free

  3. Send a warning email when there is less than 10% of space free

  4. Reject incoming data when there is less than 5% of space free

Automatic deletion on low disk space carries a risk. If another process fills the volume that Develocity uses, Develocity deletes all Build Scan data.

 If backups are created on the same volume, make sure to leave enough room for them in your thresholds. For example, if your backups take up 40% of the disk, the recommended thresholds above become 55%, 50% and 45%. Store backups on a separate volume to simplify space management.

Percentage thresholds for disk space management require the embedded database. With a user-managed database, consult your database provider for monitoring and alerts on database disk space.

Space that Build Scan data uses in Object Storage does not count toward the thresholds above. If such an installation runs low on database space, Build Scan data is unlikely to be the cause, so deleting it automatically recovers little space. We do not recommend setting an auto-deletion threshold when you store Build Scan data in Object Storage.

Build Scan Object Storage

You can store Build Scan data in Object Storage, such as Amazon S3, Google Cloud Storage, and Microsoft Azure Blob Storage. Using Object Storage is a trade-off. There are several benefits:

  • High traffic installations typically see an improvement in Build Scan processing and cleanup performance.

  • Cloud-based Object Storage services are often highly scalable and fault-tolerant, typically more so than an individual database.

  • Storing more Build Scan data is typically cheaper.

  • With the vast majority of Build Scan data in Object Storage, the main Develocity database typically requires much less storage. You can then provision the installation (for embedded databases) or the user-managed database with fewer resources.

  • Database backup management is easier when the database is much smaller, and backups themselves take up less space and are cheaper to store.

However, there are also downsides and limitations:

  • Installation is more complex.

  • More memory must be allocated to the Develocity application (2GB is the increase we recommend).

  • The persistent data of Develocity is no longer fully contained within a database backup, because Build Scan data is no longer stored in the database. This makes it more complicated to clone a Develocity instance.

  • Develocity should run as close as possible to your Object Storage service. For example, run it on AWS for Amazon S3, GCP for Google Cloud Storage, or Azure for Azure Blob Storage. Within that cloud provider, also use the same geographic region.

See the Kubernetes Helm Chart Configuration Guide or Standalone Helm Chart Configuration Guide for Object Storage configuration details.

Some scenarios require a custom endpoint URL for your Object Storage service:

  • If you need to connect to S3 directly from a VPC using a gateway VPC endpoint

  • If your Object Storage service is not S3 but provides an S3-compatible interface

Build Scan Storage Configuration

After you configure Object Storage in your Helm chart, follow these steps to store incoming Build Scan data there:

  • Go to Administration  Build Scans

  • Select Store incoming Build Scans in object storage

  • Click Save

  • You will be prompted to restart to allow the configuration changes to be applied.

When it stores Build Scan data in Object Storage, Develocity requires more memory than when it uses the database. On Kubernetes cluster installations you should increase Develocity’s memory requests and limits, see the configuration reference

If you manually write a Develocity configuration file to do an unattended installation, see the configuration reference for examples.

If you are using EKS service account credentials, ensure that you have configured Helm with the necessary service account annotation and redeployed.

After Develocity restarts, it stores all new Build Scan data in the configured Object Storage. It keeps loading existing Build Scan data from the database until the configured age-based retention period removes it. If you adopted Object Storage to shrink your database, it does not start shrinking until existing Build Scan data begins to age out. To shrink the database, Develocity has to delete the Build Scan data and then reclaim the space it took up. This process runs both daily and weekly.

Build Scan Object Storage Develocity Configuration Reference

To configure Develocity to store incoming Build Scan data in Object Storage, add the following snippet to your configuration file:

unattended-config.yaml
buildScans:
  incomingStorageType: objectStorage

On Kubernetes cluster installations, increase Develocity’s memory requests and limits by 2Gi, from the default of 6Gi:

values.yaml
enterprise:
  resources:
    requests:
      memory: 8Gi (1)
    limits:
      memory: 8Gi (1)
1 If you have already set a custom value here, instead increase it by 2Gi.

On a standalone installation the values above do not apply. There is no chart default to increase. Instead, automatic resource management sizes the application’s memory from the host, the same way whether or not Build Scan data goes to Object Storage.

What it gives the application depends on the host. Near the documented 24 GiB minimum it is little above the application’s 6Gi base, which is less than the cluster figure above. On a larger host it is considerably more. Read the limit in force before deciding whether you need more:

kubectl get pod --namespace NAMESPACE \
  --selector app.kubernetes.io/component=enterprise-app \
  -o jsonpath='{.items[*].spec.containers[*].resources.limits.memory}'

Replace NAMESPACE with the namespace your Develocity instance is installed in.

If that is less than your instance needs, pin a larger limit with enterprise.resources.limits.memory. Automatic resource management reserves a pinned limit out of the budget before it sizes the other components. A pinned limit also no longer follows the host, so revisit it if you resize the host. The alternative to pinning is a larger host, which raises the application’s share without freezing it.

If you are using EKS service account based credentials, ensure that you have configured Helm with the necessary service account annotation and redeployed.

Deprecated S3 Bucket Prefix Configuration

Releases prior to 2023.2 allowed specifying a prefix to use within a bucket when storing Build Scan data. This is now deprecated, and all Build Scan data will be stored under the prefix build-scans, which was the previous default value used if not specified.

If your installation stores significant Build Scan data under a custom prefix, you can add that prefix as an advanced app parameter. Add the following snippet to your configuration file:

unattended-config.yaml
advanced:
  app:
    params:
      buildScanStorage.prefix: my/prefix

This override will be removed in a future release. Administrators using a custom prefix are encouraged to migrate their data to the standard prefix and remove the override. See Migrating Build Scan Data to the Standard Prefix for the migration procedure.