A project can have multiple datasets, each used to train a separate model. You can get a list of the available datasets, get a specific dataset, export a dataset and can delete a dataset you no longer need.
Listing datasets
A project can include numerous datasets. This section describes how to retrieve a list of the available datasets for a project.
Web UI
To see a list of the available datasets using the AutoML Vision Object Detection UI, click the Datasets link at the top of the left navigation menu.
To see the datasets for a different project, select the project from the drop-down list on the left side of the title bar.
REST
Before using any of the request data, make the following replacements:
- project-id: your GCP project ID.
HTTP method and URL:
GET https://automl.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/datasets
To send your request, choose one of these options:
curl
Execute the following command:
curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: project-id" \
"https://automl.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/datasets"
PowerShell
Execute the following command:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred"; "x-goog-user-project" = "project-id" }
Invoke-WebRequest `
-Method GET `
-Headers $headers `
-Uri "https://automl.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/datasets" | Select-Object -Expand Content
You should receive a JSON response similar to the following:
{ "datasets": [ { "name": "projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID", "displayName": "DISPLAY_NAME", "createTime": "2018-10-29T15:45:53.353442Z", "exampleCount": 227, "imageObjectDetectionDatasetMetadata": {} } ] }
Go
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Java
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Node.js
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Python
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Additional languages
C#: Please follow the C# setup instructions on the client libraries page and then visit the AutoML Vision Object Detection reference documentation for .NET.
PHP: Please follow the PHP setup instructions on the client libraries page and then visit the AutoML Vision Object Detection reference documentation for PHP.
Ruby: Please follow the Ruby setup instructions on the client libraries page and then visit the AutoML Vision Object Detection reference documentation for Ruby.
Get a dataset
You can also get a specific dataset using a dataset ID.
Web UI
To see a list of the available datasets using the AutoML Vision Object Detection UI, click the Datasets link at the top of the left navigation menu.
To see the datasets for a different project, select the project from the drop-down list on the left side of the title bar.
Access a specific dataset by selecting its name from the list.
REST
Before using any of the request data, make the following replacements:
- project-id: your GCP project ID.
- dataset-id: the ID of your dataset. The ID is the last element of the name
of your dataset. For example:
- dataset name:
projects/project-id/locations/location-id/datasets/3104518874390609379
- dataset id:
3104518874390609379
- dataset name:
HTTP method and URL:
GET https://automl.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID
To send your request, choose one of these options:
curl
Execute the following command:
curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: project-id" \
"https://automl.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID"
PowerShell
Execute the following command:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred"; "x-goog-user-project" = "project-id" }
Invoke-WebRequest `
-Method GET `
-Headers $headers `
-Uri "https://automl.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID" | Select-Object -Expand Content
You should receive a JSON response similar to the following:
{ "name": "projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID", "displayName": "DISPLAY_NAME", "createTime": "2019-03-31T22:29:41.136184Z", "etag": "AB3BwFo-bssF99O7d4iI4_kwfnSi5pIK8FQ4D8h6Z_EaC4thAeZFbgbaIDvqXWuzjx9s", "exampleCount": 225, "imageObjectDetectionDatasetMetadata": {} }
Go
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Java
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Node.js
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Python
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Additional languages
C#: Please follow the C# setup instructions on the client libraries page and then visit the AutoML Vision Object Detection reference documentation for .NET.
PHP: Please follow the PHP setup instructions on the client libraries page and then visit the AutoML Vision Object Detection reference documentation for PHP.
Ruby: Please follow the Ruby setup instructions on the client libraries page and then visit the AutoML Vision Object Detection reference documentation for Ruby.
Export a dataset
You can export a CSV file with all a dataset's information to a Google Cloud Storage bucket. This is particularly helpful for the cases where you added, deleted, or modified training image annotations in the UI.
Web UI
To export a non-empty dataset, complete the following steps:
Select the non-empty dataset from the Datasets page.
Selecting the non-empty dataset will take you to the Dataset details page.
Select the Export data option at the top of the Dataset details page.
This opens a window where you can choose a Google Cloud Storage bucket location, or create a new bucket and designate it as the location to store the CSV file.
Select Export CSV after you have selected a new or existing Google Cloud Storage bucket location.
You will receive an email when the data export process has completed.
REST
Before using any of the request data, make the following replacements:
- project-id: your GCP project ID.
- dataset-id: the ID of your dataset. The ID is the last element of the name
of your dataset. For example:
- dataset name:
projects/project-id/locations/location-id/datasets/3104518874390609379
- dataset id:
3104518874390609379
- dataset name:
- output-storage-bucket: a Google Cloud Storage
bucket/directory to save output files to, expressed in the following form:
gs://bucket/directory/
. The requesting user must have write permission to the bucket.
HTTP method and URL:
POST https://automl.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID:exportData
Request JSON body:
{ "outputConfig": { "gcsDestination": { "outputUriPrefix": "CLOUD_STORAGE_BUCKET" } } }
To send your request, choose one of these options:
curl
Save the request body in a file named request.json
,
and execute the following command:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: project-id" \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://automl.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID:exportData"
PowerShell
Save the request body in a file named request.json
,
and execute the following command:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred"; "x-goog-user-project" = "project-id" }
Invoke-WebRequest `
-Method POST `
-Headers $headers `
-ContentType: "application/json; charset=utf-8" `
-InFile request.json `
-Uri "https://automl.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID:exportData" | Select-Object -Expand Content
You should see output similar to the following. You can use the operation ID to get the status of the task. For an example, see Working with long-running operations.
{ "name": "projects/PROJECT_ID/locations/us-central1/operations/OPERATION_ID", "metadata": { "@type": "type.googleapis.com/google.cloud.automl.v1.OperationMetadata", "createTime": "2019-08-12T18:52:31.637075Z", "updateTime": "2019-08-12T18:52:31.637075Z", "exportDataDetails": { "outputInfo": { "gcsOutputDirectory": "CLOUD_STORAGE_BUCKET/export_data-DATASET_NAME-TIMESTAMP_OF_EXPORT_CALL/" } } } }
Go
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Java
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Node.js
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Python
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Additional languages
C#: Please follow the C# setup instructions on the client libraries page and then visit the AutoML Vision Object Detection reference documentation for .NET.
PHP: Please follow the PHP setup instructions on the client libraries page and then visit the AutoML Vision Object Detection reference documentation for PHP.
Ruby: Please follow the Ruby setup instructions on the client libraries page and then visit the AutoML Vision Object Detection reference documentation for Ruby.
Exported CSV format
The exported CSV file contains the same formatting as the training data import CSV:
set,path,label,x_min,y_min,x_max,y_min,x_max,y_max,x_min,y_max
This CSV file is saved in a created export folder that is distinguished by a unique timestamp. Below are some sample lines from an exported CSV file:
/export_data-salad_dataset-2019-05-29T18:12:18.750Z/image_object_detection_1.csv
TRAIN,gs://my-storage-bucket/img/img009.jpg,Cheese,0.643239,0.362779,0.662498,0.362779,0.662498,0.416544,0.643239,0.416544 TRAIN,gs://my-storage-bucket/img/img009.jpg,Salad,0.205697,0.255249,0.459074,0.255249,0.459074,0.775244,0.205697,0.775244 TEST,gs://my-storage-bucket/img/img118.jpg,Cheese,0.320334,0.501238,0.726751,0.501238,0.726751,0.741431,0.320334,0.741431 TEST,gs://my-storage-bucket/img/img118.jpg,Salad,0.0,0.037361,1.0,0.037361,1.0,0.926321,0.0,0.926321 TEST,gs://my-storage-bucket/img/img118.jpg,Cheese,0.358745,0.29076,0.740381,0.29076,0.740381,0.497936,0.358745,0.497936 TRAIN,gs://my-storage-bucket/img/img375.jpg,Tomato,0.027274,0.41247,0.43122,0.41247,0.43122,0.702593,0.027274,0.702593 VALIDATION,gs://my-storage-bucket/img/img852.jpg,Tomato,0.716958,0.178534,0.805999,0.178534,0.805999,0.329861,0.716958,0.329861 VALIDATION,gs://my-storage-bucket/img/img852.jpg,Tomato,0.858044,0.297255,0.950847,0.297255,0.950847,0.39173,0.858044,0.39173 VALIDATION,gs://my-storage-bucket/img/img852.jpg,Tomato,0.199644,0.624155,0.321919,0.624155,0.321919,0.796384,0.199644,0.796384 VALIDATION,gs://my-storage-bucket/img/img852.jpg,Cheese,0.399672,0.277189,0.600955,0.277189,0.600955,0.47032,0.399672,0.47032
Note that there is still only one representation of a bounding box and its label per line. This information tells you the following:
img009.jpg
- is in the TRAIN set and contains two bounding boxes withCheese
andSalad
labelsimg118.jpg
- is in the TEST set and has three bounding boxes withCheese
,Salad
, andCheese
labelsimg375.jpg
- is in the TRAIN set and has one bounding box with aTomato
labelimg852.jpg
is in the VALIDATION set and has 4 distinct bounding boxes withTomato
,Tomato
,Tomato
, andCheese
labels
Deleting a dataset
You can delete a dataset resource using the dataset's ID.
Web UI
In the AutoML Vision Object Detection UI, click the Datasets link at the top of the left navigation menu to display the list of available datasets.
Click the three-dot menu at the far right of the row you want to delete and select Delete dataset.
Click Delete in the confirmation dialog box.
REST
Before using any of the request data, make the following replacements:
- project-id: your GCP project ID.
- dataset-id: the ID of your dataset. The ID is the last element of the name
of your dataset. For example:
- dataset name:
projects/project-id/locations/location-id/datasets/3104518874390609379
- dataset id:
3104518874390609379
- dataset name:
HTTP method and URL:
DELETE https://automl.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID
To send your request, choose one of these options:
curl
Execute the following command:
curl -X DELETE \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project: project-id" \
"https://automl.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID"
PowerShell
Execute the following command:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred"; "x-goog-user-project" = "project-id" }
Invoke-WebRequest `
-Method DELETE `
-Headers $headers `
-Uri "https://automl.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/datasets/DATASET_ID" | Select-Object -Expand Content
You should see output similar to the following. You can use the operation ID to get the status of the task. For an example, see Working with long-running operations.
{ "name": "projects/PROJECT_ID/locations/us-central1/operations/OPERATION_ID", "metadata": { "@type": "type.googleapis.com/google.cloud.automl.v1.OperationMetadata", "createTime": "2019-11-08T22:37:19.822128Z", "updateTime": "2019-11-08T22:37:19.822128Z", "deleteDetails": {} }, "done": true, "response": { "@type": "type.googleapis.com/google.protobuf.Empty" } }
Go
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Java
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Node.js
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Python
Before trying this sample, follow the setup instructions for this language on the Client Libraries page.
Additional languages
C#: Please follow the C# setup instructions on the client libraries page and then visit the AutoML Vision Object Detection reference documentation for .NET.
PHP: Please follow the PHP setup instructions on the client libraries page and then visit the AutoML Vision Object Detection reference documentation for PHP.
Ruby: Please follow the Ruby setup instructions on the client libraries page and then visit the AutoML Vision Object Detection reference documentation for Ruby.