Getting batch predictions  |  AI Platform Prediction  |  Google Cloud (2024)

• Home
• Docs
• AI Platform
• AI Platform Prediction
• Documentation

When you don't need your predictions right away, or when you have a large numberof instances to get predictions for, you can use the batch prediction service.This page describes how to start AI Platform Prediction batch prediction jobs.AI Platform Prediction only supports getting batch predictions from TensorFlowmodels.

Learn aboutonline versus batch predictionor read anoverview of prediction concepts.

Before you begin

In order to request predictions, you must first:

• Create a model resource and a versionresource or put a TensorFlow SavedModel ina Cloud Storage location that your project can access.

  • If you choose to use a version resource for batch prediction, you mustcreate the version with the mls1-c1-m2 machinetype.

• Set up a Cloud Storage location that your project has access to for:

  • Input data files. This can be multiple locations, and your project mustbe authorized to read from each.

  • Output files. You can only specify one output path and your project mustbe authorized to write data to it.

• Verify that your input file is in thecorrect format for batch prediction.

Configuring a batch prediction job

To start your batch prediction job, you'll need to gather some configurationdata. This is the same data that is contained in thePredictionInputobject you use when calling the API directly:

Data format

The type of input format you use for your input files. All of your inputfiles for a given job must use the same data format. Set to one of thesevalues:

JSON

Your input files are plain text with an instance on each line. This istheformatdescribed on the prediction concepts page.

TF_RECORD

Your input files use the TensorFlowTFRecord format.

TF_RECORD_GZIP

Your input files are GZIP-compressed TFRecord files.

Input paths

The URIs of your input data files, which must be in Cloud Storagelocations. You can specify:

• Paths to specific files: 'gs://path/to/my/input/file.json'.

• Paths to directories with a single asterisk wildcard, to indicate allfiles in that directory: 'gs://path/to/my/input/*'.

• Paths to partial filenames with a single asterisk wildcard at the end,to indicate all files that start with the provided sequence:'gs://path/to/my/input/file*'.

You can combine multiple URIs. In Python you make a list of them. If you usethe Google Cloud CLI, or call the API directly, you can listmultiple URIs, separated by commas, but with no space in between them. Thisis the right format for the --input-paths flag:

 --input-paths gs://a/directory/of/files/*,gs://a/single/specific/file.json,gs://a/file/template/data*

Output path

The path to the Cloud Storage location where you want the predictionservice to save your results. Your project must have permissions to writeto this location.

Model name and version name

The name of the model and, optionally, the version you want to getpredictions from. If you don't specify a version, the model's defaultversion is used. For batch prediction, the version must use themls1-c1-m2 machinetype.

If you provide a Model URI (see the following section), omit these fields.

Model URI

You can get predictions from a model that isn't deployed onAI Platform Prediction by specifying the URI of the SavedModel you want touse. The SavedModel must be stored on Cloud Storage.

To summarize, you have three options for specifying the model to use forbatch prediction. You can use:

• The model name by itself to use the model's default version.

• The model and version names to use a specific model version.

  See Also

  Batch, barm, cob or bap: what do you call a bread roll?

• The model URI to use a SavedModel that is on Cloud Storage, butnot deployed to AI Platform Prediction.

Region

The Google Compute Engine region where you want to run your job.For best performance, you should run your prediction job and store yourinput and output data in the same region, especially for very largedatasets. AI Platform Prediction batch prediction is available inthe following regions:

 - us-central1 - us-east1 - europe-west1 - asia-east1

To fully understand the available regions forAI Platform Prediction services, including model training and onlineprediction, read the guide to regions.

Job name

A name for your job, which must:

• Contain only mixed-case (case sensitive) letters, digits, andunderscores.
• Start with a letter.
• Contain no more than 128 characters.
• Be unique among all training and batch prediction job names ever used inyour project. This includes all jobs that you created in your project,regardless of their success or status.

Batch size (optional)

The number of records per batch. The service will buffer batch_size number of records in memory before invoking your model. Defaults to 64 if not specified.

Labels (optional)

You can add labels to your job to organize and sort jobs into categories when viewing or monitoring resources. For example, you could sort jobs by team (by adding labels like engineering or research) or by development phase (prod or test). To add labels to your prediction job, provide a list of KEY=VALUE pairs.

Maximum worker count (optional)

The maximum number of prediction nodes to use in the processing cluster forthis job. This is your way to put an upper limit on the automatic scalingfeature of batch prediction. If you don't specify a value, it defaultsto 10. Regardless of the value you specify, scaling is limited by theprediction node quota.

Runtime version (optional)

The AI Platform Prediction versionto use for the job. This option is included so that you can specify aruntime version to use with models that aren't deployed onAI Platform Prediction. You should always omit this value for deployedmodel versions, which signals the service to use the same version that wasspecified when the model version was deployed.

Signature name (optional)

If your saved model has multiple signatures, use this option to specifya custom TensorFlow signature name, which allows you to select analternative input/output map defined in the TensorFlow SavedModel.See the TensorFlow documentation onSavedModelfor a guide to using signatures, and the guide to specifying theoutputs of a custom model.The default isDEFAULT_SERVING_SIGNATURE_DEF_KEY,which has the value serving_default.

The following examples define variables to hold configuration data.

DATA_FORMAT="text" # JSON data formatINPUT_PATHS='gs://path/to/your/input/data/*'OUTPUT_PATH='gs://your/desired/output/location'MODEL_NAME='census'VERSION_NAME='v1'REGION='us-east1'now=$(date +"%Y%m%d_%H%M%S")JOB_NAME="census_batch_predict_$now"MAX_WORKER_COUNT="20"BATCH_SIZE="32"LABELS="team=engineering,phase=test,owner=sara"

Submitting a batch prediction job

Submitting your job is a simple call toprojects.jobs.createor its command-line tool equivalent,gcloud ai-platform jobs submit prediction.

gcloud ai-platform jobs submit prediction $JOB_NAME \ --model $MODEL_NAME \ --input-paths $INPUT_PATHS \ --output-path $OUTPUT_PATH \ --region $REGION \ --data-format $DATA_FORMAT

 import googleapiclient.discovery as discovery project_id = 'projects/{}'.format(project_name) ml = discovery.build('ml', 'v1') request = ml.projects().jobs().create(parent=project_id, body=batch_predict_body) try: response = request.execute() print('Job requested.') # The state returned will almost always be QUEUED. print('state : {}'.format(response['state'])) except errors.HttpError as err: # Something went wrong, print out some information. print('There was an error getting the prediction results.' + 'Check the details:') print(err._get_reason())

Monitoring your batch prediction job

A batch prediction job can take a long time to finish. You can monitor yourjob's progress using Google Cloud console:

1. Go to the AI Platform Prediction Jobs page in the Google Cloud console:

   Go to the Google Cloud console Jobs page

2. Click on your job's name in the Job ID list. This opens theJob details page.

3. The current status is shown with the job name at the top of the page.

4. If you want more details, you can click View logs to see your job'sentry in Cloud Logging.

There are other ways to track the progress of your batch prediction job. Theyfollow the same patterns as monitoring training jobs. You'll find moreinformation on the page describinghow to monitor your training jobs.You may need to adjust the instructions there slightly to work with predictionjobs, but the mechanisms are the same.

Getting prediction results

The service writes predictions to the Cloud Storage location you specify.There are two types of files output that might include interesting results:

• Files named prediction.errors_stats-NNNNN-of-NNNNN contain informationabout any problems encountered during the job.

• JSON Lines files namedprediction.results-NNNNN-of-NNNNN contain the predictionsthemselves, as defined by your model's output.

The filenames include index numbers (shown above as an 'N' for each digit) thatcapture how many file in total you should find. For example a job that has sixresults files includes prediction.results-00000-of-00006 throughprediction.results-00005-of-00006.

Every line of each prediction file is a JSON object representing a singleprediction result. You can open the prediction files with your choice of texteditor. For a quick look on the command line youcan use gsutil cat:

gsutil cat $OUTPUT_PATH/prediction.results-NNNNN-of-NNNNN|less

Remember that your prediction results are not typically output in the same orderas your input instances, even if you use only a single input file. You can findthe prediction for an instance by matching theinstance keys.

What's Next

• Use online prediction.
• Get more details about the prediction process.
• Troubleshoot problems that arisewhen you request online predictions.
• Learn about using labels to organize yourjobs.