How to Build TensorFlow Models for Mobile and Embedded devices

TensorFlow models can be used in applications running on mobile and embedded platforms. TensorFlow Lite and TensorFlow Mobile are two flavors of TensorFlow for resource-constrained mobile devices. TensorFlow Lite supports a subset of the functionality compared to TensorFlow Mobile. It results in better performance due to smaller binary size with fewer dependencies.

The article covers topics for training a model to integrate TensorFlow into an application. The model can then be saved and used for inference and prediction in the mobile application.

[box type=”note” align=”” class=”” width=””]This article is an excerpt from the book Mastering TensorFlow 1.x written by Armando Fandango. This book will help you leverage the power of TensorFlow and Keras to build deep learning models, using concepts such as transfer learning, generative adversarial networks, and deep reinforcement learning.[/box]

To learn how to use TensorFlow models on mobile devices, following topics are covered:

• TensorFlow on mobile platforms
• TF Mobile in Android apps
• TF Mobile demo on Android
• TF Mobile demo on iOS
• TensorFlow Lite
• TF Lite demo on Android
• TF Lite demo on iOS

TensorFlow on mobile platforms

TensorFlow can be integrated into mobile apps for many use cases that involve one or more of the following machine learning tasks:

• Speech recognition
• Image recognition
• Gesture recognition
• Optical character recognition
• Image or text classification
• Image, text, or speech synthesis
• Object identification

To run TensorFlow on mobile apps, we need two major ingredients:

• A trained and saved model that can be used for predictions
• A TensorFlow binary that can receive the inputs, apply the model, produce the predictions, and send the predictions as output

The high-level architecture looks like the following figure:

The mobile application code sends the inputs to the TensorFlow binary, which uses the trained model to compute predictions and send the predictions back.

TF Mobile in Android apps

The TensorFlow ecosystem enables it to be used in Android apps through the interface class  TensorFlowInferenceInterface, and the TensorFlow Java API in the jar file libandroid_tensorflow_inference_java.jar. You can either use the jar file from the JCenter, download a precompiled jar from ci.tensorflow.org, or build it yourself.

The inference interface has been made available as a JCenter package and can be included in the Android project by adding the following code to the build.gradle file:

allprojects  {

repositories  {

jcenter()

}

}

dependencies  {

compile  'org.tensorflow:tensorflow-android:+'

}

Note : Instead of using the pre-built binaries from the JCenter, you can also build them yourself using Bazel or Cmake by following the instructions at this link: https://github.com/tensorflow/tensorflow/blob/r1.4/ tensorflow/contrib/android/README.md

Once the TF library is configured in your Android project, you can call the TF model with the following four steps:

1.  Load the model:

TensorFlowInferenceInterface  inferenceInterface  =

new  TensorFlowInferenceInterface(assetManager,  modelFilename);

2.  Send the input data to the TensorFlow binary:

inferenceInterface.feed(inputName, floatValues,  1,  inputSize,  inputSize,  3);

3.  Run the prediction or inference:

inferenceInterface.run(outputNames,  logStats);

4.  Receive the output from the TensorFlow binary:

inferenceInterface.fetch(outputName,  outputs);

TF Mobile demo on Android

In this section, we shall learn about recreating the Android demo app provided by the TensorFlow team in their official repo. The Android demo will install the following four apps on your Android device:

• TF  Classify: This is an object identification app that identifies the images in the input from the device camera and classifies them in one of the pre-defined classes. It does not learn new types of pictures but tries to classify them into one of the categories that it has already learned. The app is built using the inception model pre-trained by Google.
• TF  Detect: This is an object detection app that detects multiple objects in the input from the device camera. It continues to identify the objects as you move the camera around in continuous picture feed mode.
• TF  Stylize: This is a style transfer app that transfers one of the selected predefined styles to the input from the device camera.
• TF  Speech: This is a speech recognition app that identifies your speech and if it matches one of the predefined commands in the app, then it highlights that specific command on the device screen.

Note: The sample demo only works for Android devices with an API level greater than 21 and the device must have a modern camera that supports FOCUS_MODE_CONTINUOUS_PICTURE. If your device camera does not have this feature supported, then you have to add the path submitted to TensorFlow by the author: https://github.com/ tensorflow/tensorflow/pull/15489/files.

The easiest way to build and deploy the demo app on your device is using Android Studio. To build it this way, follow these steps:

1.  Install Android Studio. We installed Android Studio on Ubuntu 16.04 from the instructions at the following link: https://developer.android.com/studio/ install.html
2.  Check out the TensorFlow repository, and apply the patch mentioned in the previous tip. Let’s assume you checked out the code in the tensorflow folder in your home directory.
3.  Using Android Studio, open the Android project in the path ~/tensorflow/tensorflow/examples/Android.    

Your screen will look similar to this:

4.  Expand the Gradle Scripts option from the left bar and then open the  build.gradle file.
5.  In the build.gradle file, locate the def  nativeBuildSystem definition and set it to ‘none’. In the version of  the code we checked out, this definition is at line 43:

def  nativeBuildSystem  =  'none'

6.  Build the demo and run it on either a real or simulated device. We tested the app on these devices:

7.  You can also build the apk and install the apk file on the virtual or actual connected device. Once the app installs on the device, you will see the four apps we discussed earlier:

You can also build the whole demo app from the source using Bazel or Cmake by following the instructions at this link: https://github.com/tensorflow/tensorflow/tree/r1.4/tensorflow/examples/android

TF Mobile in iOS apps

TensorFlow enables support for iOS apps by following these steps:

1.  Include TF Mobile in your app by adding a file named Profile in the root directory of your project. Add the following content to the Profile:

target  'Name-Of-Your-Project'

pod  'TensorFlow-experimental'

2.  Run the pod  install command to download and install the TensorFlow Experimental pod.
3.  Run the myproject.xcworkspace command to open the workspace so you can add the      prediction code to your application logic.

Note: To create your own TensorFlow binaries for iOS projects, follow the instructions at this link: https://github.com/tensorflow/tensorflow/ tree/master/tensorflow/examples/ios

Once the TF library is configured in your iOS project, you can call the TF model with the following four steps:

1.  Load the model:

PortableReadFileToProto(file_path,  &tensorflow_graph);

2.  Create a session:

tensorflow::Status  s  =  session->Create(tensorflow_graph);

3.  Run the prediction or inference and get the outputs:

std::string  input_layer  =  "input"; std::string  output_layer  =  "output"; std::vector<tensorflow::Tensor>  outputs; tensorflow::Status  run_status  =  session->Run(

{{input_layer,  image_tensor}},

{output_layer},  {},  &outputs);

4.  Fetch the output data:

tensorflow::Tensor*  output  =  &outputs[0];

TF Mobile demo on iOS

In order to build the demo on iOS, you need Xcode 7.3 or later. Follow these steps to build the iOS demo apps:

1.  Check out the TensorFlow code in a tensorflow folder in your home directory.
2.  Open a terminal window and execute the following commands from your home folder to download the Inception V1 model, extract the label and graph files, and move these files into the data folders inside the sample app code:

$ mkdir -p ~/Downloads

$ curl -o ~/Downloads/inception5h.zip  https://storage.googleapis.com/download.tensorflow.org/models/incep tion5h.zip 

&& unzip ~/Downloads/inception5h.zip -d ~/Downloads/inception5h

$ cp ~/Downloads/inception5h/* 

~/tensorflow/tensorflow/examples/ios/benchmark/data/

$ cp ~/Downloads/inception5h/* 

~/tensorflow/tensorflow/examples/ios/camera/data/

$ cp ~/Downloads/inception5h/* 

~/tensorflow/tensorflow/examples/ios/simple/data/

3.  Navigate to one of the sample folders and download the experimental pod:

$ cd ~/tensorflow/tensorflow/examples/ios/camera

$ pod install

4.  Open the Xcode workspace:

$ open tf_simple_example.xcworkspace

5.  Run the sample app in the device simulator. The sample app will appear with a Run Model button. The camera app requires an Apple device to be connected, while the other two can run in a simulator too.

TensorFlow Lite

TF Lite is the new kid on the block and still in the developer view at the time of writing this book. TF Lite is a very small subset of TensorFlow Mobile and TensorFlow, so the binaries compiled with TF Lite are very small in size and deliver superior performance. Apart from reducing the size of binaries, TensorFlow employs various other techniques, such as:

• The kernels are optimized for various device and mobile architectures
• The values used in the computations are quantized
• The activation functions are pre-fused
• It leverages specialized machine learning software or hardware available on the device, such as the Android NN API

The workflow for using the models in TF Lite is as follows:

1.  Get the model: You can train your own model or pick a pre-trained model available from different sources, and use the pre-trained as is or retrain it with your own data, or retrain after modifying some parts of the model. As long as you have a trained model in the file with an extension .pb or .pbtxt, you are good to proceed to the next step. We learned how to save the models in the previous chapters.
2.  Checkpoint the model: The model file only contains the structure of the graph, so you need to save the checkpoint file. The checkpoint file contains the serialized variables of the model, such as weights and biases. We learned how to save a checkpoint in the previous chapters.

3.  Freeze the model: The checkpoint and the model files are merged, also known as freezing the graph. TensorFlow provides the freeze_graph tool for this step, which can be executed as follows:

$ freeze_graph

--input_graph=mymodel.pb

--input_checkpoint=mycheckpoint.ckpt

--input_binary=true

--output_graph=frozen_model.pb

--output_node_name=mymodel_nodes

4.  Convert the model: The frozen model from step 3 needs to be converted to TF Lite format with the toco tool provided by TensorFlow:

$ toco

--input_file=frozen_model.pb

--input_format=TENSORFLOW_GRAPHDEF

--output_format=TFLITE

--input_type=FLOAT

--input_arrays=input_nodes

--output_arrays=mymodel_nodes

--input_shapes=n,h,w,c

5.  The .tflite model saved in step 4 can now be used inside an Android or iOS app that employs the TFLite binary for inference. The process of including the TFLite binary in your app is continuously evolving, so we recommend the reader follows the information at this link to include the TFLite binary in your Android or iOS app: https://github.com/tensorflow/tensorflow/tree/master/ tensorflow/contrib/lite/g3doc

Generally, you would use the graph_transforms:summarize_graph tool to prune the model obtained in step 1. The pruned model will only have the paths that lead from input to output at the time of inference or prediction. Any other nodes and paths that are required only for training or for debugging purposes, such as saving checkpoints, are removed, thus making the size of the final model very small.

The official TensorFlow repository comes with a TF Lite demo that uses a pre-trained mobilenet to classify the input from the device camera in the 1001 categories. The demo app displays the probabilities of the top three categories.

TF Lite Demo on Android

To build a TF Lite demo on Android, follow these steps:

1. Install Android Studio. We installed Android Studio on Ubuntu 16.04 from the instructions at the following link: https://developer.android.com/studio/ install.html
2. Check out the TensorFlow repository, and apply the patch mentioned in the previous tip. Let’s assume you checked out the code in the tensorflow folder in your home directory.
3. Using Android Studio, open the Android project from the path ~/tensorflow/tensorflow/contrib/lite/java/demo. If it complains about a missing SDK or Gradle components, please install those components and sync Gradle.
4. Build the project and run it on a virtual device with API > 21.

We received the following warnings, but the build succeeded. You may want to resolve the warnings if the build fails:

Warning:The  Jack  toolchain  is  deprecated  and  will  not run.  To  enable  support  for  Java  8 language  features  built into  the  plugin,  remove  ‘jackOptions  {  …  }’  from  your build.gradle  file, and  add android.compileOptions.sourceCompatibility  1.8 android.compileOptions.targetCompatibility  1.8

Note:  Future  versions  of  the  plugin  will  not  support  usage ‘jackOptions’  in  build.gradle. To learn  more,  go  to https://d.android.com/r/tools/java-8-support-message.html

Warning:The  specified  Android  SDK  Build  Tools  version (26.0.1)  is  ignored,  as  it  is  below  the minimum  supported version  (26.0.2)  for  Android  Gradle  Plugin  3.0.1. Android  SDK  Build  Tools 26.0.2  will  be  used.

To  suppress  this  warning,  remove  “buildToolsVersion ‘26.0.1’”  from  your  build.gradle  file,  as  each  version  of the  Android  Gradle  Plugin  now  has  a  default  version  of the  build  tools.

TF Lite demo on iOS

In order to build the demo on iOS, you need Xcode 7.3 or later. Follow these steps to build the iOS demo apps:

1.  Check out the TensorFlow code in a  tensorflow folder in your home directory.
2.  Build the TF Lite binary for iOS from the instructions at this link: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite

3.  Navigate to the sample folder and download the pod:

$ cd ~/tensorflow/tensorflow/contrib/lite/examples/ios/camera

$ pod install

4.  Open the Xcode workspace:

$ open tflite_camera_example.xcworkspace

5.  Run the sample app in the device simulator.

We learned about using TensorFlow models on mobile applications and devices. TensorFlow provides two ways to run on mobile devices: TF Mobile and TF Lite. We learned how to build TF Mobile and TF Lite apps for iOs and Android. We used TensorFlow demo apps as an example.  

If you found this post useful, do check out the book Mastering TensorFlow 1.x  to skill up for building smarter, faster, and efficient machine learning and deep learning systems.

Read Next:

The 5 biggest announcements from TensorFlow Developer Summit 2018

Getting started with Q-learning using TensorFlow

Implement Long-short Term Memory (LSTM) with TensorFlow