Install and Compile C++ Inference Library on Windows¶

Hardware Environment¶

Hardware Configuration of the experimental environment:

The operating system is win10 family version in the experimental environment.

Hardware Environment¶

Hardware Configuration of the experimental environment:

Steps to Configure Environment¶

Please strictly follow the subsequent steps to install, otherwise the installation may fail

Install Visual Studio 2017

Install Visual Studio 2017. Please choose “customize” for the options of contents to be installed and choose to install all functions relevant to c, c++ and vc++.

Other requirements¶

1. You need to download the Windows inference library or compile the inference library from Paddle source code.

2. You need to run the command to get the Paddle source code.

git clone https://github.com/PaddlePaddle/Paddle.git

Usage of Inference demo¶

# Path is the directory of Paddle you downloaded.
cd path\Paddle\paddle\fluid\inference\api\demo_ci
run_windows_demo.bat

gpu_inference=Y # Use gpu_inference_lib or not(Y/N), default: N.
use_mkl=Y # Use MKL or not(Y/N), default: Y.
use_gpu=Y  # Whether to use GPU for prediction, defalut: N.

paddle_inference_lib=path\fluid_inference_install_dir # Set the path of paddle inference library.
cuda_lib_dir=path\lib\x64  # Set the path of cuda library.
vcvarsall_dir=path\vc\vcvarsall.bat  # Set the path of visual studio command prompt.

Implementing a simple inference demo¶

Complete code example

This example uses Analysisconfig to manage the Analysispredictor prediction configuration. The configuration method is as follows:

1. Create AnalysisConfig

   AnalysisConfig config;
   config->SwitchUseFeedFetchOps(false);  // Turn off the use of feed and fetch OP, this must be set when using the ZeroCopy interface.
   // config->EnableUseGpu(100 /*Set the GPU initial memory pool to 100MB*/,  0 /*Set GPU ID to 0*/); // Turn on GPU prediction
   

2. Set path of models and parameters

   • When there is a model file and multiple parameter files under the model folder model_dir, the model folder path is passed in, and the model file name defaults to __model__.

   config->SetModel("path\\model_dir\\__model__", "path\\model_dir\\__params__");
   

   • When there is only one model file __model__ and one parameter file __params__ in the model folder model_dir, the model file and parameter file path are passed in.

   config->SetModel("path\\model_dir\\__model__", "path\\model_dir\\__params__");
   

3. Create predictor and prepare input data

   std::unique_ptr<PaddlePredictor> predictor = CreatePaddlePredictor(config);
   int batch_size = 1;
   int channels = 3; // The parameters of channels, height, and width must be the same as those required by the input in the model.
   int height = 300;
   int width = 300;
   int nums = batch_size * channels * height * width;
   
   float* input = new float[nums];
   for (int i = 0; i < nums; ++i) input[i] = 0;
   

4. Manage input with ZeroCopyTensor

   auto input_names = predictor->GetInputNames();
   auto input_t = predictor->GetInputTensor(input_names[0]);
   
   // Reshape the input tensor, copy the prepared input data from the CPU to ZeroCopyTensor
   input_t->Reshape({batch_size, channels, height, width});
   input_t->copy_from_cpu(input);
   

5. Run prediction engine

   predictor->ZeroCopyRun();
   

6. Manage input with ZeroCopyTensor

auto output_names = predictor->GetOutputNames();
auto output_t = predictor->GetOutputTensor(output_names[0]);
std::vector<int> output_shape = output_t->shape();
int out_num = std::accumulate(output_shape.begin(), output_shape.end(), 1,
                              std::multiplies<int>());

out_data.resize(out_num);
output_t->copy_to_cpu(out_data.data()); // Copy data from ZeroCopyTensor to cpu
delete[] input;

Note: For more introduction to AnalysisPredictor, please refer to the introduction of C++ Prediction API.