Dynamic Backend Loading

Updated 1,779 Days AgoPublic

Actions

Technical Contact: Matteo Martincigh

Pre-Implementation Review

Post-Implementation Review

Abstract

This is a design note to add to ability to dynamically load a backend in the ARmNN's runtime

Overview

Enable ArmNN to discover all the backends available on a system and dynamically load any it might find during the runtime's startup. It should be possible for the same compiled libarmnn.so deployed on different systems to load different backends. The same backend can be loaded simultaneously by different runtimes.

The current way of statically adding a backend at compile time must be retained.

Detailed Design

Main specifications:

1. A "backend object" or "dynamic backend object" is a shared object that conforms to ArmNN's IBackendInternal interface
2. Load the backends as shared libraries from a "backends" subdirectory in the path where libarmnn.so is, or make the backends path an argument that can be passed to the runtime via CreationOptions (this second option being the easiest)
3. Use dlopen, dlclose and dlsym to load, unload and obtain the address of a symbol in the shared object respectively (see code snippet below)
4. Create a new "LoadDynamicBackends" method in the Runtime class. The method must be private so that it can't be called outside the normal intended use. Call the new method in the IRunTime::Create factory method LoadDynamicBackends must be called before the Runtime constructor, as the Runtime constructor need the BackendRegistry to be already populated
5. To be loaded properly, a shared object must implement all the methods defined by the IBackendInternal interface
6. If any backend fails to load, the error must be reported but the runtime should ignore the failing backend and continue to the next one, if any, and then ultimately start as usual
7. Provide detailed documentation for the users who want to create their own (dynamic) backend

Implementation Plan

Load the backends as shared libraries from a specific path

During the creation of the Runtime object, ArmNN should scan a given path searching for suitable backend objects. The (absolute) path can be specific through the CreationOptions class, that is passed to the Runtime for its construction.
A default path should be provided as a fallback, for example a subdirectory named "backends" respect where the ArmNN shared object (libarmnn.so) lives on the system.
Getting the path of a shared library from itself can be tricky on Linux system, as it requires to parse /proc/self/maps where the full path of every loaded shared object is listed.

Use the dlfcn API to load the backend objects

Once the path to the backend objects has been located, proceed to load them using the standard dlopen, dlclose and dlsym functions.

Create a new LoadDynamicBackends method in the Runtime class

Add a new private LoadDynamicBackends method to the Runtime class that makes use of the utilities developed in the previous point.
The LoadDynamicBackends should attempt to load any shared object that's considered suitable (i.e. complient to the IBackendInternal interface) regardless of its filename.
If a valid object is found, that backend should be registered in the BackendRegistry

Add to the Backends documentation

Update the docs with information and examples about adding a custom dynamic backend object to an ArmNN's deployment.
Detail all the steps necessary to create a dynamic backend from scratch, and how to use make use of it in ArmNN.
Mention possible/common errors (where to get the error log, the object is not found, found but not loaded, loaded but not registered, registered but not used, etc.) and give basic troubleshooting advice on what the possible causes of the problems could be and suggest a valid fix.
Mention the possibility of having dynamic backends in the existing static backends documentation as well.