The Coding Notebook
Memorable coding moments of a software engineer
Cross Platform Object Detection with TensorFlow Lite - Part I
TF Lite comes with dedicated modules for Andoird and iOS, but what if we want to write the code once and run on both platform? we use C++ of course...




# Overview
In a recent project I worked on we had to incorporate object detection into our apps, the object detection was not a standalone feature but part of a bigger computer vision pipeline, since we didn't want to write this pipeline twice we decided to write everything in C++, so we can use the same code for both Android and iOS. In this post I will go over on how todo object detection in C++ and run it on both platforms.

This is a 3 posts series:
Part I - Build `libtensorflowlite` (as shared lib) for both Android and iOS
[Part II](https://www.thecodingnotebook.com/2020/04/cross-platform-object-detection-with.html) - Write the actual object detection code in C++
[Part III](https://www.thecodingnotebook.com/2020/04/cross-platform-object-detection-with_26.html) - Develop the Android and iOS apps that will use the native detector

There are also notes on [converting TF object detection model to tflite model](https://www.thecodingnotebook.com/2020/05/converting-to-tensorflow-lite-models.html)

## What is not covered in this guide?
- Training an Object Detection model and converting it to a .tflite model (too big of an issue to address in this post)
- Creating the Android/iOS app and all the camera integration code, I will do go over the "bridging" code between Kotlin/Swift and C++.

# Part I - TL;DR
Instead of compiling the libraries and gathering the headers yourself, I have created a repo with the required distribution, you can [download from here](https://github.com/ValYouW/tflite-dist/releases)

# Setup
In order to work with TF Lite we need to build the C++ library (libtensorflowlite). As of today, there is no pre-build distribution available from Tensorflow. We will also have to collect all the relevant header files as well.
In order to build the Android libs you'll have to work on Linux or run Ubuntu from Windows WSL.
In order to build the iOS libs you'll have to work on Mac.

**Note:** The tensorflow project is progressing quite rapidly, and any guide written today might not be exact next week, so this guide is good for tensorflow v2.0.0

**NOTE:** This setup procedure should be done on both Mac (for iOS) and Ubuntu (or WSL) for Android

Start by cloning tensorflow and switching to tag v2.0.0
```sh
git clone git@github.com:tensorflow/tensorflow.git
cd tensorflow
git checkout v2.0.0
```
Next download dependencies:
```sh
cd [tf-repo-root]
./tensorflow/lite/tools/make/download_dependencies.sh
```
If successful all deps were extracted to: `./tensorflow/lite/tools/make/downloads`

## Get TF Lite headers files
Create a folder that will include all the needed files, for me I created a folder named `tflite-dist`, in this guide we'll refer to it as `[dist-root]`
We'll grab all the headers files using a combination of "find" and "tar" as follow:
```sh
cd [tf-repo-root]/tensorflow
find ./lite -name "*.h" | tar -cf headers.tar -T -
```
Extract the tar file into your dist folder into the following path: `[dist-root]/include/tensorflow` folder.
```sh
mv headers.tar [dist-root]
cd [dist-root]
mkdir -p include/tensorflow
tar xvf headers.tar -C include/tensorflow
rm headers.tar
```

## Get faltbuffers headers
The flatbuffers headers are under `[tf-repo-root]/tensorflow/lite/tools/make/downloads/flatbuffers/include/flatbuffers`, copy the headers to `[dist-root]/include/flatbuffers`
```sh
cd [dist-root]
mkdir -p include/flatbuffers
cp [tf-repo-root]/tensorflow/lite/tools/make/downloads/flatbuffers/include/flatbuffers/* include/flatbuffers/
```

## Prepare folders for libs
```sh
cd [dist-root]
mkdir -p libs/arm64-v8a
mkdir -p libs/armeabi-v7a
mkdir -p libs/x86_64
mkdir -p libs/x86
```

# Building libtensorflowlite for Android
Building for Android requires lots for steps, I created a special [blog post](https://www.thecodingnotebook.com/2019/11/building-tensorflow-lite-for-android-on.html) just for that. Note that at the very end of the post you don't have to "Build tensorflow-lite AAR" but only "Build libtensorflow C++ shared object".
You should run the build command 4 times, one per architecture (android_arm64, android_arm, android_x86_64, android_x86), after each build copy the output lib to the relevant folder under `[dist-root]/libs`, for example:
```sh
# Build for arm64
bazel build //tensorflow/lite:libtensorflowlite.so --config=android_arm64 --cxxopt='--std=c++11' -c opt
# And copy the output to the [dist-root]
cp bazel-bin/tensorflow/lite/libtensorflowlite.so [dist-root]/libs/arm64-v8a/
```

Side note: It is also possible to build the GPU delegate as follow:
```sh
# Build for arm64
bazel build -c opt --config android_arm64 --copt -Os --copt -DTFLITE_GPU_BINARY_RELEASE --copt  -s --strip always //tensorflow/lite/delegates/gpu:libtensorflowlite_gpu_gl.so
```

# Building libtensorflowlite for iOS
For iOS we will build a single static library that will cover all architectures, after you completed the **setup** step above (specifically running `download_dependencies.sh`) run the following build script:
```sh
cd [tf-repo-root]
./tensorflow/lite/tools/make/build_ios_universal_lib.sh
```
If successful the output is lib is: `./tensorflow/lite/tools/make/gen/lib/libtensorflow-lite.a`, now copy it to the `dist-root/libs` folder.

# Checkpoint
Your [dist-root] folder for Android should look something like this:
![tflite-dist android](https://1.bp.blogspot.com/-14hjiQRjg20/XcFi9tV1WaI/AAAAAAAAA0Y/h38pEAEaHEgPc1OVsRq3iuXnK1vaQJVywCNcBGAsYHQ/s320/2019-11-05%2B13_54_00-SocketSniff.jpg)

And for iOS:
![tflite-dist ios](https://1.bp.blogspot.com/-Mpl-j6zbCJI/Xk2tydspw1I/AAAAAAAAA6k/Zt11_-q690UHnZawDawyqED9SMXpnCIiwCNcBGAsYHQ/s1600/tflite-dist-ios.jpg)

That's it! We will start coding the object detection in the next post