Skip To Content

Deep learning in Raster Analysis

ArcGIS Image Server allows you to use statistical or machine learning classification methods to classify remote sensing imagery. Deep learning is a type of machine learning that relies on multiple layers of nonlinear processing for feature identification and pattern recognition described in a model. Deep learning models can be integrated with ArcGIS Image Server for object detection and image classification. Training samples of features or objects of interest are generated in ArcGIS Pro with classification training sample manager tools, then converted to a format for use in the deep learning framework. These training samples are used to train the model using a third-party deep learning framework by a data scientist or image scientist. The trained model, together with the model definition file (and optionally a python raster function script file), is packed and shared as a deep learning package (dlpk) item which is then used to run the raster analysis inference tools. The inference tools either extract specific features or classify the pixels in the imagery. The dlpk item can be used multiple times as inputs to the raster analysis tools, making it easy to assess multiple images over different locations and time periods, after the model has been trained.

Deep learning workflow

Workflow item 1

Create training samples with the Training Samples Manager in ArcGIS Pro and use the Export Training Data for Deep Learning raster analysis tool in ArcGIS REST API, or ArcGIS API for Python to prepare the data for the data scientist.

Workflow item 2

The data scientist uses the training data to develop models using a third-party deep learning framework.

Workflow item 3

The data scientist delivers a deep learning package, consisting of the model definition file and the trained model and/or Python script, back to the ArcGIS Image Server raster analysis environment.

Workflow item 4

Create a deep learning package (.dlpk) item and share it to Portal. With the dlpk item, run the Detect Objects Using Deep Learning tool or the Classify Pixels Using Deep Learning tool to generate an output.

workflow item 5

You may leverage the ArcGIS API for Python to train the model with a deep learning framework.

Features and capabilities

Deep learning raster analysis tools enable you to use more than the standard machine learning classification techniques.

  • Use convolutional neural networks for image classification.
  • Use a deep learning model to either classify image pixels or detect objects such as airplanes, trees, vehicles, water bodies, and oil well pads.
  • Integrate external deep learning model frameworks, including Keras, TensorFlow, CNTK and PyTorch.
  • Use a model definition file multiple times to detect change over time or detect objects in different areas of interest.
  • Generate a polygon feature class showing detected objects to be used for additional analysis or workflows.
  • The deep learning inference tools can use the CPU for distributed processing, or use the powerful GPU on each server node if available.

Get started with deep learning

The creation and export of training samples are done within ArcGIS by a knowledgeable image analyst because they require strong knowledge of image classification workflows. The deep learning model is trained outside of ArcGIS using a deep learning framework by a data scientist, or someone experienced in training deep learning models. Once the model is trained and defined in an Esri model definition file .emd, you share a dlpk item to Portal for ArcGIS and run the raster analysis tools to detect objects or classify pixels from Map Viewer, ArcGIS API for Python, ArcGIS REST API, or ArcGIS Pro. In some cases, all three steps may be completed by one analyst that has experience with deep learning models and ArcGIS image classification.

You will need to install the proper deep learning framework ArcGIS API for Python (TensorFlow, CNTK, PyTorch or Keras) in the ArcGIS Server Python 3 environment; otherwise, an error will occur when you add the dlpk item into the tool. Obtain the appropriate framework information from the creator of the Esri Model Definition file

Refer to the framework's documentation for help with installation. Refer to the help documentation for help configuring and deploying ArcGIS Enterprise for raster analytics.

  1. Create and export training samples.
    1. Use the Training Samples Manager in ArcGIS Pro to select or create a classification schema.
    2. Create training site samples for the class categories or features of interest. Save the training sample file.
    3. Run the Export Training Data for Deep Learning raster analysis tool to convert the source imagery and training samples into deep learning training data. The source imagery may be an image service. The output of the tool is the data store string where the output image chips, labels and metadata files will be stored. An additional output of the tool is a template .emd file to be populated by the data scientist.
  2. Train the deep learning model.
    1. Use the image chips generated in step 1 above to train a model using a deep learning framework such as TensorFlow, CNTK, PyTorch or Keras.
    2. Update the .emd file with the name of the deep learning framework, the model configuration, and other optional properties. For information on required and optional parameters in the .emd file, see the Esri model definition file section below.
  3. Run the inference raster analysis tools from Map Viewer, ArcGIS API for Python, ArcGIS REST API or ArcGIS Pro.
    1. Use the Detect Objects Using Deep Learning or the Classify Pixels Using Deep Learning raster analysis tools to process your imagery. If your trained model incorporated custom Python raster functions with additional variables such as padding or confidence threshold for fine-tuning the sensitivity, these variables will appear on the ArcGIS Pro or Map Viewer raster analysis tool's dialog box for user input. The data type, such as string or numeric, is specified in the Python raster function. Ideally, additional inference parameters should be limited to two.

      The output of the Detect Objects Using Deep Learning tool is a feature class showing the objects detected by the model, and the Classify Pixels Using Deep Learning tool outputs a classified raster.

      Note:

      The deep learning model framework must be installed on the Raster Analysis service machines in order to run the inference tools because the inference logic is built into the framework Python API and is required for computation.

Esri model definition file

The .emd file is a JSON file that describes the trained deep learning model. It contains model definition parameters that are required to run the inference tools, and it should be modified by the data scientist that trained the model. There are required and optional parameters in the file as described in the table below.

Model definition file parameterExplanation

Framework

The name of a deep learning framework used to train your model.

Three deep learning frameworks are supported:

  • TensorFlow
  • CNTK
  • Keras
  • PyTorch

InferenceFunction

ModelConfiguration

The name of the model configuration.

The model configuration defines the model inputs and outputs, the inferencing logic, and the assumptions made about the model inputs and outputs. There are existing open source deep learning workflows that define standard input and output configuration and inferencing logic. ArcGIS supports a set of predefined configurations:

TensorFlow

CNTK

Keras

PyTorch

If you used one of the predefined configurations, type the name of the configuration in the .emd file. If you trained your deep learning model using a custom configuration, you will need to describe the inputs and outputs in the .emd file or in the custom Python file, in full.

ModelFile

The path to a trained deep learning model file. Many model framework file formats are based on Protocol Buffers, so the trained deep learning model is a .pb file, but other file formats are supported.

ModelType

The type of model. The type can be one of the following:

  • ImageClassification—For classifying pixels
  • ObjectDetection—For detecting objects or features

InferenceFunction

(Optional)

The path of the inference function.

An inference function understands the trained model data file and provides the inferencing logic. There are six inference functions that are supported in the ArcGIS deep learning raster analysis tools:

  • Detect Objects for TensorFlow
  • Classify Pixels for Tensor Flow
  • Detect Objects for CNTK
  • Classify Pixels for CNTK
  • Detect Objects for Keras
  • Detect Objects for Pytorch
If you used one of the inference functions above, there is no need to specify it in the .emd file. If your model is trained using a deep learning model configuration that is not yet supported, or it requires special inferencing logic, a custom inference function (a Python module) is required together with the trained model. In this case, set InferenceFunction to the Python module path.

ImageHeight

(Optional)

The number of rows in the image being classified or processed.

ImageWidth

(Optional)

The number of columns in the image being classified or processed.

ExtractBands

(Optional)

The band indexes or band names to extract from the input imagery.

Classes

(Optional)

Information about the output class categories or objects.

Below is an example of a model definition file (.emd) that uses a standard model configuration:

{
    "Framework": "TensorFlow",
    "ModelConfiguration": "ObjectDetectionAPI",
    "ModelFile":"tree_detection.pb",
    "ModelType":"ObjectionDetection",
    "ImageHeight":850,
    "ImageWidth":850,
    "ExtractBands":[0,1,2],
    "Classes" : [
      {
        "Value": 0,
        "Name": "Tree",
        "Color": [0, 255, 0]
      }
    ]
}

Deep learning package (dlpk) item

Deep learning raster analysis tools require a deep learning model package (dlpk) as input. A deep learning model package is composed of the Esri model definition JSON file (.emd), the deep learning binary model file, and optionally, the Python raster function to be used.

When you have all components ready, you can compress all files into a .zip file, and upload the .zip file to your portal as a dlpk item. For more details, please refer to Add files from your computer. You can also change your .zip file extension to .dlpk so that when adding item, the dlpk item type will be automatically detected.