Convert your Deployment Manager configurations with DM Convert

This page describes the process of using DM Convert to convert your Deployment Manager configurations to Kubernetes Resource Model (KRM) or Terraform.

Set up your environment

Set up your environment variables

Save the following environment variables, which the rest of this guide uses:

export PROJECT_ID=$(gcloud config get-value project) \
export DM_CONVERT_IMAGE="us-central1-docker.pkg.dev/\
dm-convert-host/deployment-manager/dm-convert:public-preview"

Set up your tools

You must have access to the following tools:

• gcloud

• docker

• kubectl

• bq

• jq

If you use Cloud Shell to run DM Convert, you already have access to them.

Convert your configurations

At a high level, you migrate your Deployment Manager configuration to Terraform or KRM by:

1. Preparing a Deployment Manager deployment for conversion.

2. Converting the configuration to either the HCL (HashiCorp configuration language, for Terraform) or the KRM (Kubernetes Resource Model) format.

3. Using Terraform or Config Connector to apply the converted configuration.

4. Abandoning the existing Deployment Manager deployment.

Prepare your existing deployment

DM Convert operates on Deployment Manager configuration files and templates. Throughout the guide, these files will be created and saved locally as input for the DM Convert tool.

You can create a configuration file yourself or acquire a configuration from a live deployment.

Convert a configuration file

You can use the following sample configuration to try out the converter. Replace PROJECT_ID with your Google Cloud project ID, and save the below contents to a file called deployment.yaml:

  resources:
  - name: bigquerydataset
    type: bigquery.v2.dataset
    properties:
      datasetReference:
        datasetId: bigquerydataset
        projectId: PROJECT_ID
      defaultTableExpirationMs: 36000000
      location: us-west1
  - type: bigquery.v2.table
    name: bigquerytable
    properties:
      datasetId: bigquerydataset
      labels:
        data-source: external
        schema-type: auto-junk
      tableReference:
        projectId: PROJECT_ID
        tableId: bigquerytable
    metadata:
      dependsOn:
      - bigquerydataset

• Acquire a configuration from a live deployment

  If you want to acquire and convert the configuration of a live deployment, you can retrieve the expanded configuration and save it to disk by running the following commands, replacing DEPLOYMENT_NAME with the name of the deployment.

  # Configure your project/deployment
  DEPLOYMENT_NAME=DEPLOYMENT_NAME
  PROJECT_ID=PROJECT_ID
  
  # Fetch the latest manifest for the given deployment
  gcloud deployment-manager deployments describe $DEPLOYMENT_NAME \
    --project $PROJECT_ID --format="value(deployment.manifest)"
  https://www.googleapis.com/deploymentmanager/v2/projects/$PROJECT_ID/global/deployments/bq/manifests/manifest-1618872644848
  
  # The manifest name is the last path segment from the URI
  # in the above command output
  MANIFEST_NAME="manifest-1618872644848"
  # Save the expanded manifest to deployment.yaml
  gcloud deployment-manager manifests describe $MANIFEST_NAME \
    --deployment $DEPLOYMENT_NAME --project $PROJECT_ID \
    --format="value(expandedConfig)" > deployment.yaml
  

Convert your deployment

To convert resources in deployment.yaml to HCL or KRM format and save them as converted outputs, run the following command in the same directory as deployment.yaml with the desired substitutions:

CONVERTED_RESOURCES=OUTPUT_FILE

docker run --rm -it --workdir=/convert \
--volume=$(pwd):/convert \
$DM_CONVERT_IMAGE \
--config deployment.yaml \
--output_format OUTPUT_FORMAT \
--output_file OUTPUT_FILE \
--output_tf_import_file OUTPUT_IMPORT_FILE \
--deployment_name DEPLOYMENT_NAME \
--project_id $PROJECT_ID

Make the following substitutions:

• OUTPUT_FORMAT: The output format for the conversion. This can be either TF for Terraform or KRM for KRM.

• OUTPUT_FILE: The name of the file to which the converted output is saved.

• (Terraform Only) OUTPUT_IMPORT_FILE: The name of the file to which the Terraform import commands are saved. If a project_id flag is specified, the import commands are generated based on that flag. If a project_id flag isn't specified, the import commands are generated based on the projectId attribute from the resource configuration.

• DEPLOYMENT_NAME: The name of the deployment. This is relevant if you're using templates in your Deployment Manager configuration, and are also using the deployment environment variable. For more information, visit Using an environment variable.

View the conversions

# Print output file
cat OUTPUT_FILE

Apply your converted configuration

# Configure default project
cat <<EOF > echo > main.tf
provider "google" {
  project = "$PROJECT_ID"
}
EOF

# NOTE: if Terraform state gets corrupted during testing,
# use init --reconfigure to reset backend
terraform init
echo "***************  TERRAFORM PLAN  ******************"
terraform plan
echo "**************  TERRAFORM APPLY  ******************"
terraform apply

# NOTE: if Terraform state gets corrupted during testing,
# use init --reconfigure to reset backend
terraform init

cat OUTPUT_IMPORT_FILE

# Make the import file executable
chmod +x OUTPUT_IMPORT_FILE
# Perform the import
./OUTPUT_IMPORT_FILE

terraform plan

Terraform will perform the following actions:

# google_bigquery_dataset.bigquerydataset will be updated in-place
~ resource "google_bigquery_dataset" "bigquerydataset" {
    ...
    ~ labels = {
        # the label value will be based on the deployment name and may not
        # match
        - "goog-dm" = "bq-for-import" -> null
      }
    ...
  }

# google_bigquery_table.bigquerytable will be updated in-place
~ resource "google_bigquery_table" "bigquerytable" {
    ...
    ~ labels = {
        # the label value will be based on the deployment name and may not
        # match
        - "goog-dm" = "bq-for-import" -> null
      }
    ...
  }

Plan: 0 to add, 2 to change, 0 to destroy.

# Accept changes by entering yes when prompted
terraform apply

...
# change from 10 hrs to 12 hrs
default_table_expiration_ms = 43200000
...

# Accept changes by entering yes when prompted
terraform apply
# Access the dataset properties via bq to verify the changes
bq show --format=prettyjson bigquerydataset | jq '.defaultTableExpirationMs'

gcloud container clusters get-credentials GKE_CLUSTER

# Ensure that the namespace is annotated to create resources in the correct
# project/folder/organization. https://cloud.google.com/config-connector/docs/how-to/install-upgrade-uninstall#specify
kubectl apply -n CONFIG_CONNECTOR_NAMESPACE \
  -f OUTPUT_FILE

# Wait for the resources to become healthy
kubectl wait -n CONFIG_CONNECTOR_NAMESPACE \
  --for=condition=Ready \
  --timeout=5m -f OUTPUT_FILE

Clean up

Clean up the sample dataset and table

# NOTE: if Terraform state gets corrupted during testing,
# use init --reconfigure to reset backend
echo "***************  TERRAFORM INIT  ******************"
terraform init
# Remove delete protection on BigQuery table
sed -i "/resource \"google_bigquery_table\"/a deletion_protection=\"false\"" \
OUTPUT_FILE
terraform apply
echo "***************  TERRAFORM DESTROY ****************"
terraform destroy

# If the resource was created via Config Connector:
kubectl delete -n CONFIG_CONNECTOR_NAMESPACE \
  -f OUTPUT_FILE

Abandon the sample Deployment Manager deployment

To abandon a live deployment that you successfully converted to KRM or Terraform, run:

gcloud deployment-manager deployments delete DEPLOYMENT_NAME --delete-policy ABANDON

Supported resources for conversion

docker run --rm -it \
us-central1-docker.pkg.dev/dm-convert-host/deployment-manager/dm-convert:public-preview \
--output_format tf \
--list_supported_types

docker run --rm -it \
us-central1-docker.pkg.dev/dm-convert-host/deployment-manager/dm-convert:public-preview \
--output_format krm \
--list_supported_types

Next steps

Review best practices and recommendations for the converted configuration.