Page MenuHomePhabricator

Dynamic Backend Loading
Updated 1,751 Days AgoPublic

Version 9 of 51: You are viewing an older version of this document, as it appeared on Jun 11 2019, 2:08 PM.

Technical Contact: Matteo Martincigh

Pre-Implementation Review

Post-Implementation Review

Abstract

This is a design note to add to ability to dynamically load a backend in the ARmNN's runtime

VersionDateChangesAuthor
0.110/06/2016Initial draftMatteo Martincigh

Overview

Enable ArmNN to discover all the backends available on a system and dynamically load any it might find during the runtime's startup. It should be possible for the same compiled libarmnn.so deployed on different systems to load different backends. The same backend can be loaded simultaneously by different runtimes.

The current way of statically adding a backend at compile time must be retained.

Detailed Design

Main specifications:

  1. A "backend object" or "dynamic backend object" is a shared object that conforms to ArmNN's IBackendInternal interface
  2. Load the backends as shared libraries from the hard-coded path "/usr/shared/armnn/backends", and make the backends path an argument that can be passed to the runtime via CreationOptions (can be used for testing purposes)
  3. Use dlopen, dlclose and dlsym to load, unload and obtain the address of a symbol in the shared object respectively
  4. Create a new "LoadDynamicBackends" method in the Runtime class. The method must be private so that it can't be called outside the normal intended use. Call the new method in the IRunTime::Create factory method LoadDynamicBackends must be called before the Runtime constructor, as the Runtime constructor need the BackendRegistry to be already populated
  5. To be loaded properly, a shared object must implement all the methods defined by the IBackendInternal interface
  6. If any backend fails to load, the error must be reported but the runtime should ignore the failing backend and continue to the next one, if any, and then ultimately start as usual
  7. Provide detailed documentation for the users who want to create their own (dynamic) backend

Implementation Plan

Create a new DynamicBackendUtils helper class to hold all the methods required by the implementation. Make all the methods static, DynamicBackendUtils should not have class members.

Load the backends as shared libraries from a specific path

During the creation of the Runtime object, ArmNN should scan a given path searching for suitable backend objects. The (absolute) path can be specified through the CreationOptions class, that is passed to the Runtime for its construction.
A default path should be provided as a fallback, for example a subdirectory named "backends" respect where the ArmNN shared object (libarmnn.so) lives on the system.
Getting the path of a shared library from itself can be tricky on Linux system, as it requires to parse /proc/self/maps where the full path of every loaded shared object is listed.
Create the following utility functions:

  • GetBackendsPath: returns the absolute path of where to search for dynamic backends

Can be either what the user passed via CreationOptions, or the fallback value (the "backends" subdirectory)

if (!CreationOptions.m_DynamicBackendsPath.empty())
{
  if (!CheckIfPathIsValid(CreationOptions.m_DynamicBackendsPath))
  {
    throw RuntimeException("The given dynamic backends path is not valid: " + CreationOptions.m_DynamicBackendsPath);
  }
  return CreationOptions.m_DynamicBackendsPath;

  string fallbackPath = GetBackendsFallbackPath();
  if (!CheckIfPathIsValid(fallbackPath))
  {
    throw RuntimeException("The fallback dynamic backends path is not valid: " + fallbackPath);
  }
  return fallbackPath;
}

Create a new DynamicBackend class

A new DynamicBackend class can be used to store all the function pointers for a loaded dynamic backend.

Use the dlfcn API to load the backend objects

Once the path to the backend objects has been located, proceed to load them using the standard dlopen, dlclose and dlsym functions.
Create the following utility functions:

  • GetSharedObjects: gets a list of all the shared objects (as absolute path strings) at the path returned by GetBackendsPath
  • LoadBackend: try to load each shared object returned by GetSharedObjects as a dynamic backend
void* sharedObjectHandle = dlopen(sharedObjectPath, RTLD_LAZY | RTLD_LOCAL);
if (!sharedObjectHandle)
{
  BOOST_LOG_TRIVIAL(error) << "An error occurred trying to open the shared object \"" << sharedObjectPath << "\": " << dlerror() << "\n";
  continue; // Continue to the next shared object
}

// Try to load all methods in IBackendInternal
OptimizationViews (*optimizeSubgraphViewPtr)(SubgraphView&);
optimizeSubgraphViewPtr = (OptimizationViews(*)(SubgraphView&))dlsym(sharedObjectHandle, "OptimizeSubgraphView");
if (!optimizeSubgraphViewPtr)
{
  BOOST_LOG_TRIVIAL(error) << "An error occurred trying to open the shared object \"" << sharedObjectPath << "\": " << dlerror() << "\n";
  dlclose(sharedObjectHandle); // Close the current shared object and move onto the next
  continue; // Continue to the next shared object
}

// Store the function pointer in a DynamicBackend instance
dynamicBackend.SetOptimizeSubgraphViewPtr(optimizeSubgraphViewPtr);

... // Repeat for all the other methods in IBackendInternal
  • UnloadBackend: release the backend once the runtime gets destroyed
for (DynamicBackend dynamicBackend : dynamicBackends)
{
  dlclose(dynamicBackend.GetSharedObjectHandle); // Ignore any error
}

The shared object handles will have to be members of the DynamicBackendUtils class.

Create a new LoadDynamicBackends method in the Runtime class

Add a new private LoadDynamicBackends method to the Runtime class that makes use of the utilities developed in the previous point.
The LoadDynamicBackends should attempt to load any shared object that's considered suitable (i.e. complient to the IBackendInternal interface) regardless of its filename.
If a valid object is found, that backend should be registered in the BackendRegistry
Create the following function in the Runtime class:

  • LoadDynamicBackends: applies the entire procedure of getting the backend objects, load them, and register the dynamic backends in the runtime (throws an exception in case of backend registration error)
string backendsPath = GetBackendsPath();
vector<string> sharedObjects = GetSharedObjects(backendsPath);
vector<DynamicBackend> dynamicBackends;
for (string sharedObject : sharedObjects)
{
  DynamicBackend dynamicBackend = LoadBackend(sharedObject);
  if (success)
  {
    dynamicBackends.push_back(dynamicBackend);
  }
}
for (DynamicBackend dynamicBackend : dynamicBackends)
{
  BackendRegistry.Register(dynamicBackend.GetId(), ...); // throws in case of error
}

Use a dynamic backend

Once a backend (static or dynamic) is registered in the BackendRegistry, its factory method is called automatically by the runtime as per the current implementation, no changes should be required.

Extend the Backends documentation

Update the docs with information and examples about adding a custom dynamic backend object to an ArmNN's deployment.
Detail all the steps necessary to create a dynamic backend from scratch, and how to use make use of it in ArmNN.
Mention possible/common errors (where to get the error log, the object is not found, found but not loaded, loaded but not registered, registered but not used, etc.) and give basic troubleshooting advice on what the possible causes of the problems could be and suggest a valid fix.
Mention the possibility of having dynamic backends in the existing static backends documentation as well.

Test Strategy

Tests to implement:

  1. Test that you must be able load a valid backend
  2. Test that you must not load an invalid shared object (e.g. a file that is not a shared object)
  3. Test that you must not load an invalid backend (e.g. a shared object that si not a backend)
  4. Test that you must be able to load multiple backends
  5. Test that you must be able to load multiple backends with the same id, but the backend registrator must fail for all but the first one
  6. Test that all the loaded backend are unloaded when the runtime is destroyed
  7. Test that all the methods of a loaded backend can be executed (use dummy backend methods)
  8. Test that you cannot start a runtime with no backends loaded
  9. Test that you can start multiple runtimes with the same backends (by calling the backend factory)
Last Author
MatteoArm
Last Edited
Jun 11 2019, 2:08 PM

Event Timeline

MatteoArm created this document.Jun 10 2019, 3:17 PM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jun 10 2019, 3:24 PM
MatteoArm published a new version of this document.
MatteoArm edited the content of this document. (Show Details)Jun 10 2019, 4:34 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 10:46 AM
MatteoArm published a new version of this document.Jun 11 2019, 12:03 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 1:37 PM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 1:46 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 2:02 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 2:08 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 2:11 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 2:31 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 2:50 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 3:11 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 3:32 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 3:37 PM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 3:40 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 3:45 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 3:53 PM
MatteoArm added a subscriber: MatteoArm.
MatteoArm edited the content of this document. (Show Details)Jun 12 2019, 10:10 AM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jun 13 2019, 3:54 PM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jun 14 2019, 11:02 AM
MatteoArm edited the content of this document. (Show Details)Jun 14 2019, 4:24 PM
MatteoArm edited the content of this document. (Show Details)Jun 26 2019, 8:01 AM
MatteoArm edited the content of this document. (Show Details)Jun 26 2019, 1:06 PM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jun 26 2019, 2:07 PM
MatteoArm edited the content of this document. (Show Details)Jun 26 2019, 2:27 PM
MatteoArm published a new version of this document.
MatteoArm edited the content of this document. (Show Details)Jun 26 2019, 2:52 PM
MatteoArm edited the content of this document. (Show Details)Jun 26 2019, 2:55 PM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jun 26 2019, 3:18 PM
MatteoArm edited the content of this document. (Show Details)Jun 27 2019, 7:58 AM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jul 3 2019, 1:54 PM
MatteoArm edited the content of this document. (Show Details)Jul 4 2019, 9:59 AM
MatteoArm edited the content of this document. (Show Details)Jul 4 2019, 10:30 AM
MatteoArm edited the content of this document. (Show Details)Jul 4 2019, 4:57 PM
MatteoArm edited the content of this document. (Show Details)Jul 5 2019, 3:40 PM
MatteoArm edited the content of this document. (Show Details)Jul 5 2019, 3:53 PM

I am happy to sign off on this document

MatteoArm changed the edit policy from "Custom Policy" to "Custom Policy".Jul 9 2019, 4:06 PM
MatthewARM edited the content of this document. (Show Details)Jul 11 2019, 3:07 PM
MatthewARM edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jul 11 2019, 3:31 PM
MatteoArm edited the content of this document. (Show Details)Jul 11 2019, 3:39 PM
brianjoycearm edited the content of this document. (Show Details)Jul 12 2019, 8:37 AM
MatteoArm edited the content of this document. (Show Details)Jul 22 2019, 12:06 PM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jul 22 2019, 12:59 PM
MatthewARM changed the visibility from "All Users" to "Public (No Login Required)".Jun 18 2020, 9:00 AM