Page MenuHomePhabricator

Dynamic Backend Loading
Updated 1,737 Days AgoPublic

Version 33 of 51: You are viewing an older version of this document, as it appeared on Jun 26 2019, 2:56 PM.

Technical Contact: Matteo Martincigh (matteo.martincigh@arm.com)

Pre-Implementation Review

DateReviewersCommentsActions
_

Post-Implementation Review

DateReviewersCommentsActions
_

Abstract

This is a design note to add to ability to dynamically load a backend in the ArmNN's runtime

VersionDateChangesAuthor
0.110/06/2016Initial draftMatteo Martincigh
0.211/06/2016Overall reworkingMatteo Martincigh
0.312/06/2016Updated backend loading designMatteo Martincigh
0.413/06/2016Added implementation hintsMatteo Martincigh
0.526/06/2016Overall reworking and added sectionsMatteo Martincigh

Intended Users

  1. End users (wants use an app with Arm NN)
  2. App developers (wants to develop an app with Arm NN)
  3. OEM developer (wants to develop an app with Arm NN to be shipped as part of an OEM bundle)
  4. Backend author (wants to develop a backend to be consumed by Arm NN)

Intended use cases

  1. As an end user, I want the easiest, smoothest user-experience when using an app with ArmNN (i.e. no performance impact due to dynamic backend loading).
  2. As an app developer, I would like to be able to control whether backends are linked statically or loaded dynamically. It should be possible to have some statically linked and some dynamically loaded. Or one or the other.
  3. As an app developer, I would like to be able to use ArmNN without having to know or care about which backends it will execute on.
  4. As an app developer, I would like to control which backends get loaded into the runtime during development, which may be different when deployed.
  5. As an app developer, I would like ArmNN to be robust when dealing with malformed/buggy backend implementations.
  6. As an OEM developer, I would like to explicitly control which backends execute my model. I want to be able to specify this at the model level, but with overrides at the per-layer level (i.e. it must not be possible to leverage the changes introduced to implement dynamic backend loading to disrupt a model execution).
  7. As a backend author, I would like to make available a new version of my backend with performance improvements and bug fixes. End users would download and install the new version and existing apps would make use of the updated code.

Overview

Enable ArmNN to discover all the backends available on a system and dynamically load any it might find during the runtime's startup. It should be possible for the same compiled libarmnn.so deployed on different systems to load different backends. The same backend can be loaded simultaneously by different runtimes.

The current way of statically adding a backend at compile time must be retained. It should be possible to have some backends statically linked and/or some dynamically loaded. It should be possible to disable the dynamic backend loaded if required.

Detailed Design

Main specifications:

  1. A "backend object" or "dynamic backend object" is a shared object that conforms to ArmNN's IBackendInternal interface, and to a spefic naming scheme
  2. Load the backends as shared libraries from the hard-coded path "/usr/lib/armnn/backends"
  3. For testing purposes also make the backends path an optional argument that can be passed to the runtime via CreationOptions (can be used for testing purposes)
  4. Use dlopen, dlclose and dlsym to load, unload and obtain the address of a symbol in the shared object respectively. Use dlerror to get a human readable error description for error reporting
  5. Create a new "LoadDynamicBackends" method in the Runtime class. The method must be private so that it won't be visible from outside the class (and thus not be called outside the normal intended use). Call the new method in the IRunTime::Create factory method LoadDynamicBackends must be called before the Runtime constructor, as the Runtime constructor needs the BackendRegistry to be already populated
  6. To be loaded properly, a shared object must implement all the methods defined by the IBackendInternal interface
  7. If any backend fails to load, the error must be reported but the runtime should ignore the failing backend and continue to the next one, if any, and then ultimately start as usual
  8. Provide detailed documentation for the users who want to create their own (dynamic) backend

Implementation Plan

Create a new DynamicBackendUtils helper class to hold all the methods required by the implementation.
Make all the methods static, DynamicBackendUtils should not have class members. The implementation is based on the dlfcn API.

Implementation hint:

class DynamicBackendUtils
{
  static std::string GetBackendsPath();
  static bool IsPathValid(const std::string& path);
  static bool IsBackendSupported(const BackendVersion& backendVersion);

  template<typename EntryPointType>
  static EntryPointType GetEntryPoint(const void* sharedObjectHandle, const char* symbolName)
  {
    if (sharedObjectHandle == nullptr)
    {
        throw RuntimeException("Called DynamicBackend::GetEntryPoint on an invalid module");
    }
    if (symbolName == nullptr)
    {
        throw RuntimeException("Called DynamicBackend::GetEntryPoint for an invalid symbol");
    }
    auto entryPoint = reinterpret_cast<EntryPointType>(dlsym(const_cast<void*>(sharedObjectHandle), symbolName));
    if (!entryPoint)
    {
        throw RuntimeException("Invalid entry point");
    }
    return entryPoint;
  }

  static void* OpenHandle(const std::string& sharedObjectPath)
  {
    if (sharedObjectPath.empty())
    {
        throw RuntimeException("Invalid shared object path");
    }

    void* handle = dlopen(sharedObjectPath.c_str(), RTLD_LAZY | RTLD_GLOBAL);
    if (!handle)
    {
      throw RuntimeException(DynamicBackendUtils::GetDlError());
    }

    return handle;
  }

  static void CloseHandle(const void* handle)
  {
    if (!handle)
    {
      return;
    }

    dlclose(const_cast<void*>(handle)); // Ignore errors when closing the handle
  }

  static std::string GetDlError()
  {
    const char* dl_error = dlerror();
    if (!dl_error)
    {
      return "";
    }

    return std::string(dl_error);
  }

  ...
};

Load the backends from shared libraries in a specific path

During the creation of the Runtime object, ArmNN should scan a given path searching for suitable backend objects. The (absolute) path can be specified through the CreationOptions class, that is passed to the Runtime for its construction.
A default path should be provided as a hard-coded fallback, for example "/usr/lib/armnn/backends".
A single path should be considered for dynamically loading the backends, to either load the "official" backends from a system path, or to load any test backends from another (possibly non-system) path. That is, paths are mutually esclusive, so no multiple paths will be considered during the loading phase.
ArmNN will try to load only the files that match the accepted naming scheme(s):

  1. <vendor>_<name>_<version>_backend.so (e.g. "Arm_GpuAcc_0102_backend.so", version format MMmm)
  2. <vendor>_<name>_backend.so<version> (e.g. "Arm_GpuAcc_backend.so.1.2")
  3. <vendor>_<name>_backend.so (e.g. "Arm_GpuAcc_backend.so")

For security reasons, symbolic links should not be followed.

Create the following utility functions:

  • GetBackendsPath: returns the absolute path of where to search for dynamic backends. Can be either what the user passed via CreationOptions, or the fallback default value. A single path is retuned or an exception is throw and the loading phase is interrupted.
  • IsPathValid: checks that the given path is valid (it exists, it's readable, etc.)
  • IsBackendSupported: checks that the given backend version is supported (check strategy detailed below)

Implementation hint:

std::string GetBackendsPath()
{
  if (!CreationOptions.m_DynamicBackendsPath.empty())
  {
    if (!IsPathValid(CreationOptions.m_DynamicBackendsPath))
    {
      throw RuntimeException("The given dynamic backends path is not valid: " + CreationOptions.m_DynamicBackendsPath);
    }
    return CreationOptions.m_DynamicBackendsPath;
  }

  string fallbackPath = GetBackendsFallbackPath();
  if (!IsPathValid(fallbackPath))
  {
    throw RuntimeException("The fallback dynamic backends path is not valid: " + fallbackPath);
  }
  return fallbackPath;
}

Backend versioning

A backend compatibilty with the ArmNN's runtime will be verified immediately upon loading by only inspecting the backend's version.

Version format:

struct BackendVersion
{
  uint32_t m_Major;
  uint32_t m_Minor;
};

It'll be useful to have some utility function(s) to compare versions, for example:

int compare(const BackendVersion& first, const BackendVersion& second)
{
  if (first is later than second) return 1;
  if (first is earlier than second) return -1;
  return 0;
}

The compatibility check is done by the DynamicBackendUtils::IsBackendSupported method.
A backend is considered compatible when it has the same major version of ArmnNN's Backend API, and an equal to or greater than minor version.

Examples:

  • Backend v2.4 is compatible with ArmNN's Backend API v2.1
  • Backend v2.4 is not compatible with ArmNN's Backend API v2.5
  • Backend v2.4 is not compatible with ArmNN's Backend API v1.0

Create a new DynamicBackend class

A new DynamicBackend class will be used to represent a loaded dynamic backend in ArmNN. Malformed backend shared object should not have a corresponding DynamicBackend in ArmNN's runtime.
The DynamicBackend class will need to hold (and manage) the shared object handle as well as provide the factory method for creating an instance of the backend, which will be used to create and register the backend instance in the backend registry. Once the new backend instance will be added to the backend registry, AmNN will proceed to making use of the new backend as usual, with to further changes required to the runtime.
The DynamicBackend class will need to get the backend id of the backend for registration purposes.
The DynamicBackend class will need to get the version of the dynamic backend in the shared object so that ArmNN could verify its compatibility against the current runtime version. The shared object must then export a function for checking its version.
A new list of DynamicBackend instances should be maintained in the Runtime class.

A new BackendIdFunction type should be added to the BackendRegistry to define the signature of the backend id getter method.
Proposed name and signature:

const char* GetBackenId() // A BackendId object could be later created by ArmNN from the given string

A new VersionFunction type should be added to the BackendRegistry to define the signature of the version getter method.
Proposed name and signature:

void GetVersion(uint32_t* outMajor, uint32_t* outMinor) // A BackendVersion object could be later created by ArmNN with the given int values

The BackendRegistry already defines a type for the FactoryFunction that expects a std::unique_ptr<IBackendInternal> to be returned, however the proposed name and signature for the BackendFactory function exported by the shared object is:

IBackendInternal* BackendFactory() // The function returns a raw pointer that ArmNN will later manage using a smart pointer

ArmNN will try to load the functions by their name without mangling, so external C linkage must be specified when exporting the shared object symbols:

extern "C"
{
const char* GetBackenId();
void GetVersion(uint32_t* major, uint32_t* minor);
IBackendInternal* BackendFactory();
}

Implementation hint:

using HandleCloser = std::function<void(const void*)>;
using HandlePtr = std::unique_ptr<void, HandleCloser>;

class DynamicBackend
{
public:
  explicit DynamicBackend(const void* handle)
    : m_GetBackendIdFunction(nullptr)
    , m_GetVersionFunction(nullptr)
    , m_BackendFactoryFunction(nullptr)
    , m_Handle(const_cast<void*>(handle), &DynamicBackendUtils::CloseHandle)
  {
    if (m_Handle == nullptr)
    {
      throw InvalidArgumentException("Cannot create a DynamicBackend from an invalid shared object")
    }

    // These calls may throw
    m_GetBackendIdFunction = DynamicBackendUtils::GetEntryPoint<BackendRegistry::BackendIdFunction>("GetBackenId");
    m_GetVersionFunction = DynamicBackendUtils::GetEntryPoint<BackendRegistry::VersionFunction>("GetVersion");
    m_BackendFactoryFunction = DynamicBackendUtils::GetEntryPoint<BackendRegistry::FactoryFunction>("BackendFactory");
  }

  // These are probably not necessary, mentioned here just in case
  bool IsLoaded() const { return m_Handle != nullptr; }
  void* Get() const { return m_Handle.get(); }
  void Close() { m_Handle.reset(); }
  void* Release()
  {
    void* handle = Get();
    Close();
    return handle;
  }
  ...

  // Public dybnamic backend functions
  BackendId GetBackendId()
  {
    assert(m_GetBackendIdFunction);
    return BackendId(m_GetBackendIdFunction);
  }
  BackendVersion GetBackendVersion()
  {
    assert(m_GetVersionFunction);
    uint32_t major = 0;
    uint32_t minor = 0;
    m_GetVersionFunction(&major, &minor);
    return BackendVersion{ major, minor };
  }
  IBackendInternalUniquePtr GetBackend()
  {
    assert(m_BackendFactoryFunction);
    return std::move(std::unique_ptr<IBackendInternal>(m_BackendFactoryFunction));
  }

private:
  // Factory function pointers
  BackendRegistry::BackendIdFunction m_GetBackendIdFunction;
  BackendRegistry::VersionFunction m_GetVersionFunction;
  BackendRegistry::FactoryFunction m_BackendFactoryFunction;

  // Shared object handle
  HandlePtr m_Handle;
};

Load the backend objects

Once the path to the backend objects has been located, proceed to load them using the utility functions in DynamicBackendUtils.

Create the following utility functions:

  • GetSharedObjects: gets a list of all the shared objects (as absolute path strings) at the path returned by GetBackendsPath
  • LoadDynamicBackend: try to load each shared object returned by GetSharedObjects as a dynamic backend. An already loaded backend cannot be loaded twice in the same runtime instance

Implemetation hint:

// While looping through a list of shared objects

void* sharedObjectHandle = nullptr;
try
{
  sharedObjectHandle = DynamicBackendUtils::OpenHandle(sharedObjectPath);
}
catch exceptions
{
  // Report exception error
  continue; // Continue to the next shared object
}
if (!sharedObjectHandle)
{
  BOOST_LOG_TRIVIAL(error) << "An error occurred trying to open the shared object \"" << sharedObjectPath << "\": " << DynamicBackendUtils::GetDlError() << "\n";
  continue; // Continue to the next shared object
}

std::unique_ptr<DynamicBackend> dynamicBackend;
try
{
  dynamicBackend.reset(new DynamicBackend(sharedObjectHandle));
}
catch exceptions
{
  // Dynamic backend possible malformed, report exception error
  continue; // Continue to the next shared object
}

// Get the backend id
BackendId backendId = dynamicBackend.GetBackendId();

if (backendId already exists in the backend registry)
{
  BOOST_LOG_TRIVIAL(error) << "A backend with id \"" << backendId << " has already been loaded\n";
  dynamicBackend.reset(); // Destroy the dynamic backend
  continue; // Continue to the next shared object
}

BackendVersion backendVersion = dynamicBackend.GetBackendVersion();

if (backendVersion is incompatible with the current Backend API version) // Use DynamicBackendUtils::IsBackendSupported
{
  BOOST_LOG_TRIVIAL(error) << "Backend \"" << backendId << " with version " << backendVersion <<  " is not compatible with the current Backend API " << backendApiVersion << "\n";
  dynamicBackend.reset(); // Destroy the dynamic backend
  continue; // Continue to the next shared object
}

// Append the DynamicBackend object to the list of dynamic backends

// Continue to the next shared object

Create a new LoadDynamicBackends method in the Runtime class

Add a new private LoadDynamicBackends method to the Runtime class that makes use of the utilities developed in the previous point.
The LoadDynamicBackends should attempt to load any shared object that's considered suitable (i.e. compliant to the IBackendInternal interface) regardless of its filename.
If a valid object is found, that backend should be registered in the BackendRegistry.

Create the following function in the Runtime class:

  • LoadDynamicBackends: applies the entire procedure of getting the backend objects, load them, and register the dynamic backends in the runtime (throws an exception in case of backend registration error)

Implementation hint:

string backendsPath = GetBackendsPath();
vector<string> sharedObjects = GetSharedObjects(backendsPath);
vector<DynamicBackend> dynamicBackends;
for (string sharedObject : sharedObjects)
{
  DynamicBackend dynamicBackend = LoadBackend(sharedObject);
  if (success)
  {
    dynamicBackends.push_back(dynamicBackend);
  }
}
for (DynamicBackend dynamicBackend : dynamicBackends)
{
  // Get the backend id
  BackendId backendId = dynamicBackend.GetBackendId();

  // Get the backend factory function
  FactoryFunction backendFactoryFunction = dynamicBackend.GetFactoryFunction();

  // Register the backend
  BackendRegistryInstance().Register(backendId, backendFactoryFunction);
}

Use a dynamic backend

Once a backend (static or dynamic) is registered in the BackendRegistry, its factory method is called automatically by the runtime as per the current implementation, no further changes should be required.

Extend the Backends documentation

Update the docs with information and examples about adding a custom dynamic backend object to an ArmNN's deployment.
Detail all the steps necessary to create a dynamic backend from scratch, and how to use make use of it in ArmNN.
Mention possible/common errors (where to get the error log, the object is not found, found but not loaded, loaded but not registered, registered but not used, etc.) and give basic troubleshooting advice on what the possible causes of the problems could be and suggest a valid fix.
Mention the possibility of having dynamic backends in the existing static backends documentation as well.

Test Strategy

Unit tests to implement:

  1. Test that you must be able load a valid backend (create a mock dynamic backend for testing)
  2. Test that you must not be able to load an invalid shared object (e.g. a file that is not a shared object, no symlinks, no files that don't match the given naming scheme)
  3. Test that you must not be able to load an invalid backend (e.g. a shared object that does not comply to the given interface)
  4. Test that you must not be able to load an unsupported backend (e.g. a backend with an incompatible version)
  5. Test that you must be able to load multiple different backends
  6. Test that you must not be able to load a backend with the same id of a backend already loaded
  7. Test that you must not be be able to load the same backend (same shared object) more than once per runtime instance
  8. Test that all the loaded backend are unloaded when the runtime is destroyed
  9. Test that all the methods of a loaded backend can be executed (use the mock dynamic backend for testing)
  10. Test that you cannot start a runtime with no backends loaded
  11. Test that you can start multiple runtimes with the same backend(s) (a different instance of the backend has to be created per runtime by calling the backend factory)
  12. Test all the utlity functions created for this feature (all methods in the DynamicBackendUtils class)

End to end tests:

  1. Run ExecuteNetwork with at least one dynamic backend (perhaps wrapping an existing backend in a shared object)
Last Author
MatteoArm
Last Edited
Jun 26 2019, 2:56 PM

Event Timeline

MatteoArm created this document.Jun 10 2019, 3:17 PM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jun 10 2019, 3:24 PM
MatteoArm published a new version of this document.
MatteoArm edited the content of this document. (Show Details)Jun 10 2019, 4:34 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 10:46 AM
MatteoArm published a new version of this document.Jun 11 2019, 12:03 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 1:37 PM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 1:46 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 2:02 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 2:08 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 2:11 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 2:31 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 2:50 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 3:11 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 3:32 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 3:37 PM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 3:40 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 3:45 PM
MatteoArm edited the content of this document. (Show Details)Jun 11 2019, 3:53 PM
MatteoArm added a subscriber: MatteoArm.
MatteoArm edited the content of this document. (Show Details)Jun 12 2019, 10:10 AM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jun 13 2019, 3:54 PM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jun 14 2019, 11:02 AM
MatteoArm edited the content of this document. (Show Details)Jun 14 2019, 4:24 PM
MatteoArm edited the content of this document. (Show Details)Jun 26 2019, 8:01 AM
MatteoArm edited the content of this document. (Show Details)Jun 26 2019, 1:06 PM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jun 26 2019, 2:07 PM
MatteoArm edited the content of this document. (Show Details)Jun 26 2019, 2:27 PM
MatteoArm published a new version of this document.
MatteoArm edited the content of this document. (Show Details)Jun 26 2019, 2:52 PM
MatteoArm edited the content of this document. (Show Details)Jun 26 2019, 2:55 PM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jun 26 2019, 3:18 PM
MatteoArm edited the content of this document. (Show Details)Jun 27 2019, 7:58 AM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jul 3 2019, 1:54 PM
MatteoArm edited the content of this document. (Show Details)Jul 4 2019, 9:59 AM
MatteoArm edited the content of this document. (Show Details)Jul 4 2019, 10:30 AM
MatteoArm edited the content of this document. (Show Details)Jul 4 2019, 4:57 PM
MatteoArm edited the content of this document. (Show Details)Jul 5 2019, 3:40 PM
MatteoArm edited the content of this document. (Show Details)Jul 5 2019, 3:53 PM

I am happy to sign off on this document

MatteoArm changed the edit policy from "Custom Policy" to "Custom Policy".Jul 9 2019, 4:06 PM
MatthewARM edited the content of this document. (Show Details)Jul 11 2019, 3:07 PM
MatthewARM edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jul 11 2019, 3:31 PM
MatteoArm edited the content of this document. (Show Details)Jul 11 2019, 3:39 PM
brianjoycearm edited the content of this document. (Show Details)Jul 12 2019, 8:37 AM
MatteoArm edited the content of this document. (Show Details)Jul 22 2019, 12:06 PM
MatteoArm edited the content of this document. (Show Details)
MatteoArm edited the content of this document. (Show Details)Jul 22 2019, 12:59 PM
MatthewARM changed the visibility from "All Users" to "Public (No Login Required)".Jun 18 2020, 9:00 AM