gcloud dataflow flex-template build - builds a flex template file from the specified parameters
gcloud dataflow flex-template build TEMPLATE_FILE_GCS_PATH --sdk-language=SDK_LANGUAGE (--image=IMAGE | --env=[ENV,...] --flex-template-base-image=FLEX_TEMPLATE_BASE_IMAGE --image-gcr-path=IMAGE_GCR_PATH (--go-binary-path=GO_BINARY_PATH | --jar=[JAR,...] | --py-path=[PY_PATH,...])) [--additional-experiments=[ADDITIONAL_EXPERIMENTS,...]] [--additional-user-labels=[ADDITIONAL_USER_LABELS,...]] [--dataflow-kms-key=DATAFLOW_KMS_KEY] [--disable-public-ips] [--enable-streaming-engine] [--gcs-log-dir=GCS_LOG_DIR] [--image-repository-cert-path=IMAGE_REPOSITORY_CERT_PATH] [--image-repository-password-secret-id=IMAGE_REPOSITORY_PASSWORD_SECRET_ID] [--image-repository-username-secret-id=IMAGE_REPOSITORY_USERNAME_SECRET_ID] [--max-workers=MAX_WORKERS] [--metadata-file=METADATA_FILE] [--network=NETWORK] [--num-workers=NUM_WORKERS] [--print-only] [--service-account-email=SERVICE_ACCOUNT_EMAIL] [--staging-location=STAGING_LOCATION] [--subnetwork=SUBNETWORK] [--temp-location=TEMP_LOCATION] [--worker-machine-type=WORKER_MACHINE_TYPE] [--worker-region=WORKER_REGION | --worker-zone=WORKER_ZONE] [GCLOUD_WIDE_FLAG ...]
Builds a flex template file from the specified parameters.
To build and store a the flex template json file, run:
$ gcloud dataflow flex-template build gs://template-file-gcs-path \ --image=gcr://image-path \ --metadata-file=/local/path/to/metadata.json --sdk-language=JAVA
If using prebuilt template image from private registry, run:
$ gcloud dataflow flex-template build gs://template-file-gcs-path \ --image=private.registry.com:3000/image-path \ --image-repository-username-secret-id="projects/test-project/sec\ rets/username-secret" \ --image-repository-password-secret-id="projects/test-project/sec\ rets/password-secret/versions/latest" \ --metadata-file=metadata.json --sdk-language=JAVA
To build the template image and flex template json file, run:
$ gcloud dataflow flex-template build gs://template-file-gcs-path \ --image-gcr-path=gcr://path-tos-tore-image \ --jar=path/to/pipeline.jar --jar=path/to/dependency.jar \ --env=FLEX_TEMPLATE_JAVA_MAIN_CLASS=classpath \ --flex-template-base-image=JAVA11 \ --metadata-file=/local/path/to/metadata.json --sdk-language=JAVA
- TEMPLATE_FILE_GCS_PATH
The Google Cloud Storage location of the flex template file.Overrides if file already exists.
- --sdk-language=SDK_LANGUAGE
SDK language of the flex template job. SDK_LANGUAGE must be one of: JAVA, PYTHON, GO.
- Exactly one of these must be specified:
- --image=IMAGE
Path to the any image registry location of the prebuilt flex template image.
- --env=[ENV,...]
Environment variables to create for the Dockerfile. You can pass them as a comma separated list or repeat individually with --env flag. Ex: --env="A=B,C=D" or --env A=B, --env C=D.When you reference files/dir in env variables, please specify relative path to the paths passed via --py-path.Ex: if you pass. --py-path="path/pipleline/" then set FLEX_TEMPLATE_PYTHON_PY_FILE="pipeline/pipeline.py" You can find the list of supported environment variables in this link. https://cloud.google.com/dataflow/docs/guides/templates/configuring-flex-templates#setting_required_dockerfile_environment_variables.
This flag argument must be specified if any of the other arguments in this group are specified.
- --flex-template-base-image=FLEX_TEMPLATE_BASE_IMAGE
Flex template base image to be used while building the container image. Allowed choices are JAVA8, JAVA11 or gcr.io path of the specific version of the base image. For JAVA8 and JAVA11 option, we use the latest base image version to build the container. You can also provide a specific version from this link https://gcr.io/dataflow-templates-base/
This flag argument must be specified if any of the other arguments in this group are specified.
- --image-gcr-path=IMAGE_GCR_PATH
The Google Container Registry or Google Artifact Registry location to store the flex template image to be built.
This flag argument must be specified if any of the other arguments in this group are specified.
- Exactly one of these must be specified:
- --go-binary-path=GO_BINARY_PATH
Local path to your compiled dataflow pipeline Go binary. The binary should be compiled to run on the target worker architecture (usually linux-amd64). See https://beam.apache.org/documentation/sdks/go-cross-compilation/ for more information.
- --jar=[JAR,...]
Local path to your dataflow pipeline jar file and all their dependent jar files required for the flex template classpath. You can pass them as a comma separated list or repeat individually with --jar flag. Ex: --jar="code.jar,dep.jar" or --jar code.jar, --jar dep.jar.
- --py-path=[PY_PATH,...]
Local path to your dataflow pipeline python files and all their dependent files required for the flex template classpath. You can pass them as a comma separated list or repeat individually with --py-path flag. Ex: --py-path="path/pipleline/,path/dependency/" or --py-path path/pipleline/, --py-path path/dependency/.
- --additional-experiments=[ADDITIONAL_EXPERIMENTS,...]
Default experiments to pass to the job.
- --additional-user-labels=[ADDITIONAL_USER_LABELS,...]
Default user labels to pass to the job.
- --dataflow-kms-key=DATAFLOW_KMS_KEY
Default Cloud KMS key to protect the job resources.
- --disable-public-ips
Cloud Dataflow workers must not use public IP addresses. Overrides the default dataflow/disable_public_ips property value for this command invocation.
- --enable-streaming-engine
Enable Streaming Engine for the streaming job by default. Overrides the default dataflow/enable_streaming_engine property value for this command invocation.
- --gcs-log-dir=GCS_LOG_DIR
Google Cloud Storage directory to save build logs.(Must be a URL beginning with 'gs://'.)
- --image-repository-cert-path=IMAGE_REPOSITORY_CERT_PATH
The full URL to self-signed certificate of private registry in Cloud Storage. For example, gs://mybucket/mycerts/selfsigned.crt. The certificate provided in Cloud Storage must be DER-encoded and may be supplied in binary or printable (Base64) encoding. If the certificate is provided in Base64 encoding, it must be bounded at the beginning by -----BEGIN CERTIFICATE-----, and must be bounded at the end by -----END CERTIFICATE-----. If this parameter is provided, the docker daemon in the template launcher will be instructed to trust that certificate.
- --image-repository-password-secret-id=IMAGE_REPOSITORY_PASSWORD_SECRET_ID
Secret Manager secret id for the password to authenticate to private registry. Should be in the format projects/{project}/secrets/{secret}/versions/{secret_version} or projects/{project}/secrets/{secret}. If the version is not provided latest version will be used.
- --image-repository-username-secret-id=IMAGE_REPOSITORY_USERNAME_SECRET_ID
Secret Manager secret id for the username to authenticate to private registry. Should be in the format projects/{project}/secrets/{secret}/versions/{secret_version} or projects/{project}/secrets/{secret}. If the version is not provided latest version will be used.
- --max-workers=MAX_WORKERS
Default maximum number of workers to run.
- --metadata-file=METADATA_FILE
Local path to the metadata json file for the flex template.
- --network=NETWORK
Default Compute Engine network for launching instances to run your pipeline.
- --num-workers=NUM_WORKERS
Initial number of workers to use by default.
- --print-only
Prints the container spec to stdout. Does not save in Google Cloud Storage. Overrides the default dataflow/print_only property value for this command invocation.
- --service-account-email=SERVICE_ACCOUNT_EMAIL
Default service account to run the workers as.
- --staging-location=STAGING_LOCATION
Default Google Cloud Storage location to stage local files.(Must be a URL beginning with 'gs://'.)
- --subnetwork=SUBNETWORK
Default Compute Engine subnetwork for launching instances to run your pipeline.
- --temp-location=TEMP_LOCATION
Default Google Cloud Storage location to stage temporary files. If not set, defaults to the value for --staging-location.(Must be a URL beginning with 'gs://'.)
- --worker-machine-type=WORKER_MACHINE_TYPE
Default type of machine to use for workers. Defaults to server-specified.
- At most one of these can be specified:
- --worker-region=WORKER_REGION
Default region to run the workers in.
- --worker-zone=WORKER_ZONE
Default zone to run the workers in.
These flags are available to all commands: --access-token-file, --account, --billing-project, --configuration, --flags-file, --flatten, --format, --help, --impersonate-service-account, --log-http, --project, --quiet, --trace-token, --user-output-enabled, --verbosity.
Run $ gcloud help for details.
This variant is also available:
$ gcloud beta dataflow flex-template build