NAME

gcloud beta dataproc jobs submit - submit Dataproc jobs to execute on a cluster

SYNOPSIS

gcloud beta dataproc jobs submit COMMAND [--async] [--bucket=BUCKET] [--region=REGION] [GCLOUD_WIDE_FLAG ...]

DESCRIPTION

(BETA) Submit Dataproc jobs to execute on a cluster.

EXAMPLES

To submit a Hadoop MapReduce job, run:

$ gcloud beta dataproc jobs submit hadoop --cluster my-cluster \ --jar my_jar.jar -- arg1 arg2

To submit a Spark Scala or Java job, run:

$ gcloud beta dataproc jobs submit spark --cluster my-cluster \ --jar my_jar.jar -- arg1 arg2

To submit a PySpark job, run:

$ gcloud beta dataproc jobs submit pyspark --cluster my-cluster \ my_script.py -- arg1 arg2

To submit a Spark SQL job, run:

$ gcloud beta dataproc jobs submit spark-sql --cluster my-cluster \ --file my_queries.q

To submit a Pig job, run:

$ gcloud beta dataproc jobs submit pig --cluster my-cluster \ --file my_script.pig

To submit a Hive job, run:

$ gcloud beta dataproc jobs submit hive --cluster my-cluster \ --file my_queries.q

FLAGS

--async

Return immediately, without waiting for the operation in progress to complete.

--bucket=BUCKET

The Cloud Storage bucket to stage files in. Defaults to the cluster's configured bucket.

--region=REGION

Dataproc region to use. Each Dataproc region constitutes an independent resource namespace constrained to deploying instances into Compute Engine zones inside the region. Overrides the default dataproc/region property value for this command invocation.

GCLOUD WIDE FLAGS

These flags are available to all commands: --help.

Run $ gcloud help for details.

COMMANDS

COMMAND is one of the following:

hadoop

(BETA) Submit a Hadoop job to a cluster.

hive

(BETA) Submit a Hive job to a cluster.

pig

(BETA) Submit a Pig job to a cluster.

presto

(BETA) Submit a Presto job to a cluster.

pyspark

(BETA) Submit a PySpark job to a cluster.

spark

(BETA) Submit a Spark job to a cluster.

spark-r

(BETA) Submit a SparkR job to a cluster.

spark-sql

(BETA) Submit a Spark SQL job to a cluster.

trino

(BETA) Submit a Trino job to a cluster.

NOTES

This command is currently in beta and might change without notice. These variants are also available:

$ gcloud dataproc jobs submit

$ gcloud alpha dataproc jobs submit