Phriction Machine Learning Platform Arm NN Arm NN Design Notes Dynamic Backend Loading History Version 18 vs 51
Version 18 vs 51
Version 18 vs 51
Edits
Edits
- Edit by • MatteoArm, Version 51
- Jul 22 2019 12:59 PM
- Edit by • MatteoArm, Version 18
- Jun 11 2019 3:45 PM
| Edit Older Version 18... | Edit Current Version 51... |
Content Changes
Content Changes
**Technical Contact:** Matteo Martincigh (matteo.martincigh@arm.com)
== Pre-Implementation Review ==
| Date | Reviewers | Comments | Actions |
|---|---|---|---|
| _ | | | |
== Post-Implementation Review ==
| Date | Reviewers | Comments | Actions|
|---|---|---|---|
| _ | | | |
= Abstract =
This is a design note to add to ability to dynamically load a backend in the ARmNN's runtime
| Version | Date | Changes | Author|
|---|---|---|---|
| 0.1 | 10/06/2016 | Initial draft | Matteo Martincigh |
| 0.2 | 11/06/2016 | Overall reworking | Matteo Martincigh |
= Overview =
Enable ArmNN to discover all the backends available on a system and dynamically load any it might find during the runtime's startup. It should be possible for the same compiled libarmnn.so deployed on different systems to load different backends. The same backend can be loaded simultaneously by different runtimes.
The current way of statically adding a backend at compile time must be retained.
= Detailed Design =
Main specifications:
# A "backend object" or "dynamic backend object" is a shared object that conforms to ArmNN's IBackendInternal interface
# Load the backends as shared libraries from the hard-coded path "/usr/shared/armnn/backends", and make the backends path an argument that can be passed to the runtime via CreationOptions (can be used for testing purposes). As a security consideration, we should check that the given path is a system path, perhaps limiting the root of the given path to be "/usr/lib" or "/usr/shared", to prevent an attacker from loading and running malicious code on the system through ArmNN
# Use dlopen, dlclose and dlsym to load, unload and obtain the address of a symbol in the shared object respectively. Use dlerror to get a human readable error description for error reporting
# Create a new "LoadDynamicBackends" method in the Runtime class. The method must be private so that it won't be visible from outside the class (and thus not be called outside the normal intended use). Call the new method in the IRunTime::Create factory method
LoadDynamicBackends must be called before the Runtime constructor, as the Runtime constructor need the BackendRegistry to be already populated
# To be loaded properly, a shared object must implement all the methods defined by the IBackendInternal interface
# If any backend fails to load, the error must be reported but the runtime should ignore the failing backend and continue to the next one, if any, and then ultimately start as usual
# Provide detailed documentation for the users who want to create their own (dynamic) backend
= Implementation Plan =
Create a new DynamicBackendUtils helper class to hold all the methods required by the implementation.
Make all the methods static, DynamicBackendUtils should not have class members.
== Load the backends as shared libraries from a specific path ==
During the creation of the Runtime object, ArmNN should scan a given path searching for suitable backend objects. The (absolute) path can be specified through the CreationOptions class, that is passed to the Runtime for its construction.
A default path should be provided as a hard-coded fallback, for example "/usr/shared/armnn/backends".
Create the following utility function:
* GetBackendsPath: returns the absolute path of where to search for dynamic backends. Can be either what the user passed via CreationOptions, or the fallback value.
```
if (!CreationOptions.m_DynamicBackendsPath.empty())
{
if (!CheckIfPathIsValid(CreationOptions.m_DynamicBackendsPath))
{
throw RuntimeException("The given dynamic backends path is not valid: " + CreationOptions.m_DynamicBackendsPath);
}
return CreationOptions.m_DynamicBackendsPath;
string fallbackPath = GetBackendsFallbackPath();
if (!CheckIfPathIsValid(fallbackPath))
{
throw RuntimeException("The fallback dynamic backends path is not valid: " + fallbackPath);
}
return fallbackPath;
}
```
== Create a new DynamicBackend class ==
A new DynamicBackend class can be used to store all the function pointers for a loaded dynamic backend.
A new list of DynamicBackend instances should be maintained in the Runtime class.
== Use the dlfcn API to load the backend objects ==
Once the path to the backend objects has been located, proceed to load them using the standard dlopen, dlclose and dlsym functions.
Create the following utility functions:
* GetSharedObjects: gets a list of all the shared objects (as absolute path strings) at the path returned by GetBackendsPath
* LoadBackend: try to load each shared object returned by GetSharedObjects as a dynamic backend. An already loaded backend cannot be loaded twice in the same runtime instance
```
// While looping through a list of shared objects
void* sharedObjectHandle = dlopen(sharedObjectPath, RTLD_LAZY | RTLD_LOCAL);
if (!sharedObjectHandle)
{
BOOST_LOG_TRIVIAL(error) << "An error occurred trying to open the shared object \"" << sharedObjectPath << "\": " << dlerror() << "\n";
continue; // Continue to the next shared object
}
if (sharedObjectHandle already exists in the list of DynamicBackend)
{
BOOST_LOG_TRIVIAL(error) << "An handle already exists for the shared object \"" << sharedObjectPath << "\n";
continue; // Continue to the next shared object
}
// Try to load all methods in IBackendInternal
OptimizationViews (*optimizeSubgraphViewPtr)(SubgraphView&);
optimizeSubgraphViewPtr = (OptimizationViews(*)(SubgraphView&))dlsym(sharedObjectHandle, "OptimizeSubgraphView");
if (!optimizeSubgraphViewPtr)
{
BOOST_LOG_TRIVIAL(error) << "An error occurred trying to open the shared object \"" << sharedObjectPath << "\": " << dlerror() << "\n";
dlclose(sharedObjectHandle); // Close the current shared object and move onto the next
continue; // Continue to the next shared object
}
// Store the function pointer in a DynamicBackend instance
dynamicBackend.SetOptimizeSubgraphViewPtr(optimizeSubgraphViewPtr);
... // Repeat for all the other methods in IBackendInternal
```
* UnloadBackend: release the backend once the runtime gets destroyed
```
for (DynamicBackend dynamicBackend : dynamicBackends)
{
dlclose(dynamicBackend.GetSharedObjectHandle); // Ignore any error
}
```
The shared object handles will have to be members of the DynamicBackendUtils class.
== Create a new LoadDynamicBackends method in the Runtime class ==
Add a new private LoadDynamicBackends method to the Runtime class that makes use of the utilities developed in the previous point.
The LoadDynamicBackends should attempt to load any shared object that's considered suitable (i.e. complient to the IBackendInternal interface) regardless of its filename.
If a valid object is found, that backend should be registered in the BackendRegistry.
Create the following function in the Runtime class:
* LoadDynamicBackends: applies the entire procedure of getting the backend objects, load them, and register the dynamic backends in the runtime (throws an exception in case of backend registration error)
```
string backendsPath = GetBackendsPath();
vector<string> sharedObjects = GetSharedObjects(backendsPath);
vector<DynamicBackend> dynamicBackends;
for (string sharedObject : sharedObjects)
{
DynamicBackend dynamicBackend = LoadBackend(sharedObject);
if (success)
{
dynamicBackends.push_back(dynamicBackend);
}
}
for (DynamicBackend dynamicBackend : dynamicBackends)
{
BackendRegistryInstance().Register(dynamicBackend.GetId(), ...); // throws in case of error
}
```
== Use a dynamic backend ==
Once a backend (static or dynamic) is registered in the BackendRegistry, its factory method is called automatically by the runtime as per the current implementation, no changes should be required.
== Extend the Backends documentation ==
Update the docs with information and examples about adding a custom dynamic backend object to an ArmNN's deployment.
Detail all the steps necessary to create a dynamic backend from scratch, and how to use make use of it in ArmNN.
Mention possible/common errors (where to get the error log, the object is not found, found but not loaded, loaded but not registered, registered but not used, etc.) and give basic troubleshooting advice on what the possible causes of the problems could be and suggest a valid fix.
Mention the possibility of having dynamic backends in the existing static backends documentation as well.
== Test Strategy ==
Tests to implement:
# Test that you must be able load a valid backend (create a mock dynamic backend for testing)
# Test that you must not be able to load an invalid shared object (e.g. a file that is not a shared object)
# Test that you must not be able to load an invalid backend (e.g. a shared object that si not a backend)
# Test that you must be able to load multiple (different) backends
# Test that you must be able to load multiple backends with the same id, but the backend registration must fail for all but the first one
# Test that you must not be be able to load the same backend (same shared object) more than once per runtime instance
# Test that all the loaded backend are unloaded when the runtime is destroyed
# Test that all the methods of a loaded backend can be executed (use dummy backend methods)
# Test that you cannot start a runtime with no backends loaded
# Test that you can start multiple runtimes with the same backend(s) (a different instance of the backend has to be created by calling the backend factory)
**Technical Contact:** Matteo Martincigh (matteo.martincigh@arm.com)
== Pre-Implementation Review ==
| Date | Reviewers | Comments | Actions |
|---|---|---|---|
| 11/07/2019 | Matthew Bentham | Cool | |
| 11/07/2019 | Brian Joyce | Looks good to me. | |
== Post-Implementation Review ==
| Date | Reviewers | Comments | Actions|
|---|---|---|---|
| _ | | | |
= Abstract =
This is a design note to add to ability to dynamically load a backend in the ArmNN's runtime
| Version | Date | Changes | Author|
|---|---|---|---|
| 0.1 | 10/06/2019 | Initial draft | Matteo Martincigh |
| 0.2 | 11/06/2019 | Overall reworking | Matteo Martincigh |
| 0.3 | 12/06/2019 | Updated backend loading design | Matteo Martincigh |
| 0.4 | 13/06/2019 | Added implementation hints | Matteo Martincigh |
| 0.5 | 26/06/2019 | Overall reworking and added sections | Matteo Martincigh |
| 0.6 | 03/07/2019 | Minor changes | Matteo Martincigh |
| 1.0 | 05/07/2019 | Changes based on feedback collected during the design note review | Matteo Martincigh |
= Intended Users =
# End users (want use an app with ArmNN)
# App developers (want to develop an app with ArmNN)
# OEM developer (want to develop an app with ArmNN to be shipped as part of an OEM bundle)
# Backend author (want to develop a backend to be consumed by ArmNN)
= Intended use cases =
# As an end user, I want the easiest, smoothest user-experience when using an app with ArmNN (i.e. no performance impact due to dynamic backend loading).
# As an app developer, I would like to be able to control whether backends are linked statically or loaded dynamically. It should be possible to have some statically linked and some dynamically loaded.
# As an app developer, I would like to be able to use ArmNN without having to know or care about which backends it will execute on.
# As an app developer, I would like to control which backends get loaded into the runtime during development, which may be different when deployed.
# As an app developer, I would like ArmNN to be robust when dealing with malformed/buggy backend implementations.
# As an OEM developer, I would like to explicitly control which backends execute my model. I want to be able to specify this at the model level, but with overrides at the per-layer level (i.e. it must not be possible to leverage the changes introduced to implement dynamic backend loading to disrupt a model execution).
# As a backend author, I would like to make available a new version of my backend with performance improvements and bug fixes. End users would download and install the new version and existing apps would make use of the updated code.
= Overview =
Enable ArmNN to discover all the backends available on a system and dynamically load any it might find during the runtime's startup. It should be possible for the same compiled libarmnn.so deployed on different systems to load different backends. The same backend can be loaded simultaneously by different runtimes.
The current way of statically adding a backend at compile time must be retained. It should be possible to have some backends statically linked and/or some dynamically loaded. It should be possible to disable the dynamic backend loaded if required.
= Detailed Design =
Main specifications:
# A "backend object" or "dynamic backend object" is a shared object that conforms to ArmNN's IBackendInternal interface, and to a specific naming scheme
# Load the backends as shared libraries from a set of paths passed via the compiler by means of "define" directives. The list of paths can be left empty, effectively preventing any dynamic backend from loading
# For testing purposes also make the backends' path an optional argument that can be passed to the runtime via CreationOptions
# Use dlopen, dlclose and dlsym to load, unload and obtain the address of a symbol in the shared object respectively. Use dlerror to get a human readable error description for error reporting
# Create a new "LoadDynamicBackends" method in the Runtime class. The method must be private so that it won't be visible from outside the class (and thus not be called outside the normal intended use). Call the new method in the IRunTime::Create factory method
LoadDynamicBackends must be called before the Runtime constructor, as the Runtime constructor needs the BackendRegistry to be already populated
# To be loaded properly, a shared object must declare a version that is compatible with the current version of the IBackendInternal interface
# If any backend fails to load, the error must be reported but the runtime should ignore the failing backend and continue to the next one, if any, and then ultimately start as usual
# The feature can be disable entirely, so that no dynamic backends will be used by the runtime
# Provide detailed documentation and at least one example for the users who want to create their own (dynamic) backend
= Implementation Plan =
Create a new DynamicBackendUtils helper class to hold all the methods required by the implementation.
Make all the methods static, DynamicBackendUtils should not have class members. The implementation is based on the dlfcn API.
//Implementation hint//:
```
class DynamicBackendUtils
{
public:
static std::vector<std::string> GetBackendPaths();
static bool IsPathValid(const std::string& path);
static bool IsBackendSupported(const BackendVersion& backendVersion);
template<typename EntryPointType>
static EntryPointType GetEntryPoint(const void* sharedObjectHandle, const char* symbolName)
{
if (sharedObjectHandle == nullptr)
{
throw RuntimeException("Called DynamicBackend::GetEntryPoint on an invalid module");
}
if (symbolName == nullptr)
{
throw RuntimeException("Called DynamicBackend::GetEntryPoint for an invalid symbol");
}
auto entryPoint = reinterpret_cast<EntryPointType>(dlsym(const_cast<void*>(sharedObjectHandle), symbolName));
if (!entryPoint)
{
throw RuntimeException("Invalid entry point");
}
return entryPoint;
}
static void* OpenHandle(const std::string& sharedObjectPath)
{
if (sharedObjectPath.empty())
{
throw RuntimeException("Invalid shared object path");
}
void* handle = dlopen(sharedObjectPath.c_str(), RTLD_LAZY | RTLD_GLOBAL);
if (!handle)
{
throw RuntimeException(DynamicBackendUtils::GetDlError());
}
return handle;
}
static void CloseHandle(const void* handle)
{
if (!handle)
{
return;
}
dlclose(const_cast<void*>(handle)); // Ignore errors when closing the handle
}
private:
static std::string GetDlError() // Private as it should be only needed by methods in this class
{
const char* dl_error = dlerror();
if (!dl_error)
{
return "";
}
return std::string(dl_error);
}
...
};
```
== Load the backends from shared libraries in a specific path ==
During the creation of the Runtime object, ArmNN should scan a given path searching for suitable backend objects. The (absolute) path can be specified through the CreationOptions class, that is passed to the Runtime for its construction.
A default list of paths should be provided via the compiler as a hard-coded fallback.
ArmNN will try to load only the files that match the following accepted naming scheme:
<vendor>_<name>_backend.so (e.g. "Arm_GpuAcc_backend.so")
Symlinks to other files are allowed to support the standard linux shared object versioning:
```
Arm_GpuAcc_backend.so -> Arm_GpuAcc_backend.so.1.0.0
Arm_GpuAcc_backend.so.1 -> Arm_GpuAcc_backend.so.1.0.0
Arm_GpuAcc_backend.so.1.0 -> Arm_GpuAcc_backend.so.1.0.0
Arm_GpuAcc_backend.so.1.0.0
```
Create the following utility functions:
* GetBackendPaths: returns a set of absolute paths where to search for dynamic backends. Can be either what the user passed via CreationOptions (a single path), or the default value (set via the compiler)
* IsPathValid: checks that the given path is valid (it exists, it's readable, etc.)
* IsBackendSupported: checks that the given backend version is supported (version strategy detailed below)
//Implementation hint//:
```
std::vector<std::string> GetBackendPaths()
{
if (!CreationOptions.m_DynamicBackendsPath.empty())
{
if (!IsPathValid(CreationOptions.m_DynamicBackendsPath))
{
Report warning: "The given dynamic backends path is not valid: " + CreationOptions.m_DynamicBackendsPath;
return {};
}
return { CreationOptions.m_DynamicBackendsPath };
}
std::vector<std::string> defaultPaths = GetDefaultBackendPaths();
std::vector<std::string> validDefaultPaths;
for (const std::string& path : defaultPaths)
{
if (!IsPathValid(path))
{
Report warning: "One of the default dynamic backends paths is not valid: " + path;
continue;
}
validDefaultPaths.push_back(path);
}
return validDefaultPaths;
}
```
== Backend versioning ==
A backend compatibility with the ArmNN's runtime will be verified immediately upon loading by only inspecting the backend's version.
Version format:
```
struct BackendVersion
{
uint32_t m_Major;
uint32_t m_Minor;
};
```
The compatibility check is done by the DynamicBackendUtils::IsBackendSupported method.
A backend is guaranteed to be compatible when it has been compiled with the same major version of ArmNN's Backend API, and an equal to or greater than minor version.
However, any minor increment in the ArmNN's Backend API will be carried out so that it won't disrupt the backends compiled against a previous minor version. Still in this case the backend is not guaranteed to benefit from the latest changes introduced in the Backend API.
Examples:
* Backend built with v2.1 API is compatible with ArmNN's Backend API v2.4 (same major version, backend built against earlier compatible API)
* Backend built with v2.5 API is **not** compatible with ArmNN's Backend API v2.4 (same major version, backend built against later incompatible API, backend might require update to the latest compatible backend API)
* Backend built with v2.4 API is **not** compatible with ArmNN's Backend API v1.0 (backend requires completely new API version)
* Backend built with v2.0 API is **not** compatible with ArmNN's Backend API v3.0 (backwards compatibility in the Backend API is broken)
It'll be useful to have a utility function to check whether a backend is compatible with the current Backend API for example:
```
bool IsBackendCompatible(const BackendVersion& backendVersion)
{
return backendVersion.m_Major == backendApiVersion.m_Major && backendVersion.m_Minor <= backendApiVersion.m_Minor;
}
```
== Create a new DynamicBackend class ==
A new DynamicBackend class will be used to represent a loaded dynamic backend in ArmNN. Malformed backend shared object should not have a corresponding DynamicBackend in ArmNN's runtime.
The DynamicBackend class will need to hold (and manage) the shared object handle as well as provide the factory method for creating an instance of the backend, which will be used to create and register the backend instance in the backend registry. Once the new backend instance will be added to the backend registry, ArmNN will proceed to making use of the new backend as usual, with to further changes required to the runtime.
The DynamicBackend class will need to get the backend id of the backend for registration purposes.
The DynamicBackend class will need to get the version of the dynamic backend in the shared object so that ArmNN could verify its compatibility against the current runtime version. The shared object must then export a function for checking its version.
A new list of DynamicBackend instances should be maintained in the Runtime class.
A new BackendIdFunction type should be added to the BackendRegistry to define the signature of the backend id getter method.
Proposed name and signature:
```
const char* GetBackendId() // A BackendId object could be later created by ArmNN from the given string
```
A new VersionFunction type should be added to the BackendRegistry to define the signature of the version getter method.
Proposed name and signature:
```
void GetVersion(uint32_t* outMajor, uint32_t* outMinor) // A BackendVersion object could be later created by ArmNN with the given int values
```
The BackendRegistry already defines a type for the FactoryFunction that expects a std::unique_ptr<IBackendInternal> to be returned, however the proposed name and signature for the BackendFactory function exported by the shared object is:
```
IBackendInternal* BackendFactory() // The function returns a raw pointer that ArmNN will later manage using a smart pointer
```
ArmNN will try to load the functions by their name without mangling, so external C linkage must be specified when exporting the shared object symbols:
```
extern "C"
{
const char* GetBackendId();
void GetVersion(uint32_t* outMajor, uint32_t* outMinor);
IBackendInternal* BackendFactory();
}
```
//Implementation hint//:
```
using HandleCloser = std::function<void(const void*)>;
using HandlePtr = std::unique_ptr<void, HandleCloser>;
class DynamicBackend
{
public:
explicit DynamicBackend(const void* handle)
: m_GetBackendIdFunction(nullptr)
, m_GetVersionFunction(nullptr)
, m_BackendFactoryFunction(nullptr)
, m_Handle(const_cast<void*>(handle), &DynamicBackendUtils::CloseHandle)
{
if (m_Handle == nullptr)
{
throw InvalidArgumentException("Cannot create a DynamicBackend from an invalid shared object")
}
// These calls may throw
m_GetBackendIdFunction = DynamicBackendUtils::GetEntryPoint<BackendRegistry::BackendIdFunction>("GetBackendId");
m_GetVersionFunction = DynamicBackendUtils::GetEntryPoint<BackendRegistry::VersionFunction>("GetVersion");
m_BackendFactoryFunction = DynamicBackendUtils::GetEntryPoint<BackendRegistry::FactoryFunction>("BackendFactory");
}
// These are probably not necessary, mentioned here just in case
bool IsLoaded() const { return m_Handle != nullptr; }
void* Get() const { return m_Handle.get(); }
void Close() { m_Handle.reset(); }
void* Release()
{
void* handle = Get();
Close();
return handle;
}
...
// Public dynamic backend functions
BackendId GetBackendId()
{
assert(m_GetBackendIdFunction);
return BackendId(m_GetBackendIdFunction());
}
BackendVersion GetBackendVersion()
{
assert(m_GetVersionFunction);
uint32_t major = 0;
uint32_t minor = 0;
m_GetVersionFunction(&major, &minor);
return BackendVersion{ major, minor };
}
IBackendInternalUniquePtr GetBackend()
{
assert(m_BackendFactoryFunction);
return std::move(std::unique_ptr<IBackendInternal>(m_BackendFactoryFunction()));
}
private:
// Backend function pointers
BackendRegistry::BackendIdFunction m_GetBackendIdFunction;
BackendRegistry::VersionFunction m_GetVersionFunction;
BackendRegistry::FactoryFunction m_BackendFactoryFunction;
// Shared object handle
HandlePtr m_Handle;
};
```
== Load the backend objects ==
Once the path to the backend objects has been located, proceed to load them using the utility functions in DynamicBackendUtils.
Create the following utility functions:
* GetSharedObjects: gets a list of all the shared objects (as absolute path strings) at the paths returned by GetBackendPaths
* LoadDynamicBackend: try to load each shared object returned by GetSharedObjects as a dynamic backend. An already loaded backend cannot be loaded twice in the same runtime instance
//Implementation hint://
```
// While looping through a list of shared objects
void* sharedObjectHandle = nullptr;
try
{
sharedObjectHandle = DynamicBackendUtils::OpenHandle(sharedObjectPath);
}
catch exceptions
{
// Report exception error
continue; // Continue to the next shared object
}
if (!sharedObjectHandle)
{
BOOST_LOG_TRIVIAL(error) << "An error occurred trying to open the shared object \"" << sharedObjectPath << "\": " << DynamicBackendUtils::GetDlError() << "\n";
continue; // Continue to the next shared object
}
std::unique_ptr<DynamicBackend> dynamicBackend;
try
{
dynamicBackend.reset(new DynamicBackend(sharedObjectHandle));
}
catch exceptions
{
// Dynamic backend possible malformed, report exception error
continue; // Continue to the next shared object
}
// Get the backend id
BackendId backendId = dynamicBackend.GetBackendId();
if (backendId already exists in the backend registry)
{
BOOST_LOG_TRIVIAL(error) << "A backend with id \"" << backendId << " has already been loaded\n";
dynamicBackend.reset(); // Destroy the dynamic backend
continue; // Continue to the next shared object
}
BackendVersion backendVersion = dynamicBackend.GetBackendVersion();
if (backendVersion is incompatible with the current Backend API version) // Use DynamicBackendUtils::IsBackendSupported
{
BOOST_LOG_TRIVIAL(error) << "Backend \"" << backendId << " with version " << backendVersion << " is not compatible with the current Backend API " << backendApiVersion << "\n";
dynamicBackend.reset(); // Destroy the dynamic backend
continue; // Continue to the next shared object
}
// Append the DynamicBackend object to the list of dynamic backends
// Continue to the next shared object
```
== Create a new LoadDynamicBackends method in the Runtime class ==
Add a new private LoadDynamicBackends method to the Runtime class that makes use of the utilities developed in the previous point.
The LoadDynamicBackends should attempt to load any shared object that's considered suitable (i.e. compliant to the IBackendInternal interface) regardless of its filename.
If a valid object is found, that backend should be registered in the BackendRegistry.
Create the following function in the Runtime class:
* LoadDynamicBackends: applies the entire procedure of getting the backend objects, load them, and register the dynamic backends in the runtime (throws an exception in case of backend registration error)
//Implementation hint//:
```
string backendsPath = GetBackendsPath();
vector<string> sharedObjects = GetSharedObjects(backendsPath);
vector<DynamicBackend> dynamicBackends;
for (string sharedObject : sharedObjects)
{
DynamicBackend dynamicBackend = LoadBackend(sharedObject);
if (success)
{
dynamicBackends.push_back(dynamicBackend);
}
}
for (DynamicBackend dynamicBackend : dynamicBackends)
{
// Get the backend id
BackendId backendId = dynamicBackend.GetBackendId();
// Get the backend factory function
FactoryFunction backendFactoryFunction = dynamicBackend.GetFactoryFunction();
// Register the backend
BackendRegistryInstance().Register(backendId, backendFactoryFunction);
}
```
== Use a dynamic backend ==
Once a backend (static or dynamic) is registered in the BackendRegistry, its factory method is called automatically by the runtime as per the current implementation, no further changes should be required.
== Extend the Backends documentation ==
Update the docs with information and examples about adding a custom dynamic backend object to an ArmNN's deployment.
Detail all the steps necessary to create a dynamic backend from scratch, and how to consume it in ArmNN.
Mention possible/common errors (where to get the error log, the object is not found, found but not loaded, loaded but not registered, registered but not used, etc.) and give basic troubleshooting advice on what the possible causes of the problems could be and suggest a valid fix.
Mention the possibility of having dynamic backends in the existing static backends documentation as well.
== Test Strategy ==
Unit tests to implement:
# Test that you must be able load a valid backend (create a mock dynamic backend for testing)
# Test that you must **not** be able to load an invalid shared object (e.g. a file that is not a shared object, no symlinks, no files that don't match the given naming scheme)
# Test that you must **not** be able to load an invalid backend (e.g. a shared object that does not comply to the given interface)
# Test that you must **not** be able to load an unsupported backend (e.g. a backend with an incompatible version)
# Test that you must be able to load multiple different backends
# Test that you must **not** be able to load a backend with the same id of a backend already loaded
# Test that you must **not** be be able to load the same backend (same shared object) more than once per runtime instance
# Test that all the loaded backend are unloaded when the runtime is destroyed
# Test that all the methods of a loaded backend can be executed (use the mock dynamic backend for testing)
# Test that you **cannot** start a runtime with no backends loaded
# Test that you can start multiple runtimes with the same backend(s) (a different instance of the backend has to be created per runtime by calling the backend factory)
# Test all the utility functions created for this feature (all methods in the DynamicBackendUtils class)
End to end tests:
# Run ExecuteNetwork with at least one dynamic backend (perhaps wrapping an existing backend in a shared object)
**Technical Contact:** Matteo Martincigh (matteo.martincigh@arm.com)
== Pre-Implementation Review ==
| Date | Reviewers | Comments | Actions |
|---|---|---|---|
| _ | | | || 11/07/2019 | Matthew Bentham | Cool | |
| 11/07/2019 | Brian Joyce | Looks good to me. | |
== Post-Implementation Review ==
| Date | Reviewers | Comments | Actions|
|---|---|---|---|
| _ | | | |
= Abstract =
This is a design note to add to ability to dynamically load a backend in the ARArmNN's runtime
| Version | Date | Changes | Author|
|---|---|---|---|
| 0.1 | 10/06/2016 | Initial draft | Matteo Martincigh |10/06/2019 | Initial draft | Matteo Martincigh |
| 0.2 | 11/06/2019 | Overall reworking | Matteo Martincigh |
| 0.3 | 12/06/2019 | Updated backend loading design | Matteo Martincigh |
| 0.4 | 13/06/2019 | Added implementation hints | Matteo Martincigh |
| 0.5 | 26/06/2019 | Overall reworking and added sections | Matteo Martincigh |
| 0.6 | 03/07/2019 | Minor changes | Matteo Martincigh |
| 1.0 | 05/07/2019 | Changes based on feedback collected during the design note review | Matteo Martincigh |
= Intended Users =
# End users (want use an app with ArmNN)
# App developers (want to develop an app with ArmNN)
# OEM developer (want to develop an app with ArmNN to be shipped as part of an OEM bundle)
# Backend author (want to develop a backend to be consumed by ArmNN)
= Intended use cases =
# As an end user, I want the easiest, smoothest user-experience when using an app with ArmNN (i.e. no performance impact due to dynamic backend loading).
# As an app developer, I would like to be able to control whether backends are linked statically or loaded dynamically. It should be possible to have some statically linked and some dynamically loaded.
# As an app developer, I would like to be able to use ArmNN without having to know or care about which backends it will execute on.
# As an app developer, I would like to control which backends get loaded into the runtime during development, which may be different when deployed.
| 0.2 | 11/06/2016 | Overall reworking | Matteo Martincigh |# As an app developer, I would like ArmNN to be robust when dealing with malformed/buggy backend implementations.
# As an OEM developer, I would like to explicitly control which backends execute my model. I want to be able to specify this at the model level, but with overrides at the per-layer level (i.e. it must not be possible to leverage the changes introduced to implement dynamic backend loading to disrupt a model execution).
# As a backend author, I would like to make available a new version of my backend with performance improvements and bug fixes. End users would download and install the new version and existing apps would make use of the updated code.
= Overview =
Enable ArmNN to discover all the backends available on a system and dynamically load any it might find during the runtime's startup. It should be possible for the same compiled libarmnn.so deployed on different systems to load different backends. The same backend can be loaded simultaneously by different runtimes.
The current way of statically adding a backend at compile time must be retained. It should be possible to have some backends statically linked and/or some dynamically loaded. It should be possible to disable the dynamic backend loaded if required.
= Detailed Design =
Main specifications:
# A "backend object" or "dynamic backend object" is a shared object that conforms to ArmNN's IBackendInternal interface, and to a specific naming scheme
# Load the backends as shared libraries from a set of paths passed via the compiler by means of "define" directives. The list of paths can be left empty, effectively preventing any dynamic backend from loading
# Load the backends as shared libraries from the hard-coded path "/usr/shared/armnn/# For testing purposes also make the backends", and make the backends path an' path an optional argument that can be passed to the runtime via CreationOptions (can be used for testing purposes). As a security consideration, we should check that the given path is a system path, perhaps limiting the root of the given path to be "/usr/lib" or "/usr/shared", to prevent an attacker from loading and running malicious code on the system through ArmNN
# Use dlopen, dlclose and dlsym to load, unload and obtain the address of a symbol in the shared object respectively. Use dlerror to get a human readable error description for error reporting
# Create a new "LoadDynamicBackends" method in the Runtime class. The method must be private so that it won't be visible from outside the class (and thus not be called outside the normal intended use). Call the new method in the IRunTime::Create factory method
LoadDynamicBackends must be called before the Runtime constructor, as the Runtime constructor needs the BackendRegistry to be already populated
# To be loaded properly, a shared object must implement alldeclare a version that is compatible with the methods defined bycurrent version of the IBackendInternal interface
# If any backend fails to load, the error must be reported but the runtime should ignore the failing backend and continue to the next one, if any, and then ultimately start as usual
# Provide detailed documentationThe feature can be disable entirely, so that no dynamic backends will be used by the runtime
# Provide detailed documentation and at least one example for the users who want to create their own (dynamic) backend
= Implementation Plan =
Create a new DynamicBackendUtils helper class to hold all the methods required by the implementation.
Make all the methods static, DynamicBackendUtils should not have class members. The implementation is based on the dlfcn API.
//Implementation hint//:
```
class DynamicBackendUtils
{
public:
static std::vector<std::string> GetBackendPaths();
static bool IsPathValid(const std::string& path);
static bool IsBackendSupported(const BackendVersion& backendVersion);
template<typename EntryPointType>
static EntryPointType GetEntryPoint(const void* sharedObjectHandle, const char* symbolName)
{
if (sharedObjectHandle == nullptr)
{
throw RuntimeException("Called DynamicBackend::GetEntryPoint on an invalid module");
}
if (symbolName == nullptr)
{
throw RuntimeException("Called DynamicBackend::GetEntryPoint for an invalid symbol");
}
auto entryPoint = reinterpret_cast<EntryPointType>(dlsym(const_cast<void*>(sharedObjectHandle), symbolName));
if (!entryPoint)
{
throw RuntimeException("Invalid entry point");
}
return entryPoint;
}
static void* OpenHandle(const std::string& sharedObjectPath)
{
if (sharedObjectPath.empty())
{
throw RuntimeException("Invalid shared object path");
}
void* handle = dlopen(sharedObjectPath.c_str(), RTLD_LAZY | RTLD_GLOBAL);
if (!handle)
{
throw RuntimeException(DynamicBackendUtils::GetDlError());
}
return handle;
}
static void CloseHandle(const void* handle)
{
if (!handle)
{
return;
}
dlclose(const_cast<void*>(handle)); // Ignore errors when closing the handle
}
private:
static std::string GetDlError() // Private as it should be only needed by methods in this class
{
const char* dl_error = dlerror();
if (!dl_error)
{
return "";
}
return std::string(dl_error);
}
...
};
```
== Load the backends asfrom shared libraries fromin a specific path ==
During the creation of the Runtime object, ArmNN should scan a given path searching for suitable backend objects. The (absolute) path can be specified through the CreationOptions class, that is passed to the Runtime for its construction.
A default path should be provided as a hard-coded fallback, for example "/usr/shared/armnn/backends".list of paths should be provided via the compiler as a hard-coded fallback.
ArmNN will try to load only the files that match the following accepted naming scheme:
Create the following utility function:<vendor>_<name>_backend.so (e.g. "Arm_GpuAcc_backend.so")
Symlinks to other files are allowed to support the standard linux shared object versioning:
```
Arm_GpuAcc_backend.so -> Arm_GpuAcc_backend.so.1.0.0
* GetBackendsPath: returns the absolute path of where to search for dynamic backends. Can be either what the user passed via CreationOptions, or the fallback value.Arm_GpuAcc_backend.so.1 -> Arm_GpuAcc_backend.so.1.0.0
Arm_GpuAcc_backend.so.1.0 -> Arm_GpuAcc_backend.so.1.0.0
Arm_GpuAcc_backend.so.1.0.0
```
Create the following utility functions:
if (!CreationOptions.m_DynamicBackendsPath.empty()* GetBackendPaths: returns a set of absolute paths where to search for dynamic backends. Can be either what the user passed via CreationOptions (a single path), or the default value (set via the compiler)
* IsPathValid: checks that the given path is valid (it exists, it's readable, etc.)
* IsBackendSupported: checks that the given backend version is supported (version strategy detailed below)
//Implementation hint//:
```
std::vector<std::string> GetBackendPaths()
{
if (!CheckIfPathIsValid(CCreationOptions.m_DynamicBackendsPath.empty())
{
throw RuntimeException("The given dynamic backends path is not vif (!IsPathValid: " + (CreationOptions.m_DynamicBackendsPath);)
{
Report warning: "The given dynamic backends path is not valid: " + CreationOptions.m_DynamicBackendsPath;
return {};
}
return { CreationOptions.m_DynamicBackendsPath };
return CreationOptions.m_DynamicBackendsPath;}
ststd::vector<std::string fallback> defaultPaths = GeetDefaultBackendsFallbackdPaths();
std::vector<std::string> validDefaultPath();s;
if (!CheckIfPathIsValid(fallbackfor (const std::string& path : defaultPath)s)
{
throw RuntimeException("The fallbackif (!IsPathValid(path))
{
Report warning: "One of the default dynamic backends paths is not valid: " + fallpath;
continue;
}
validDefaultPaths.push_backP(path);
}
return fallbackPath;validDefaultPaths;
}
```
== Backend versioning ==
A backend compatibility with the ArmNN's runtime will be verified immediately upon loading by only inspecting the backend's version.
Version format:
```
struct BackendVersion
{
uint32_t m_Major;
uint32_t m_Minor;
};
```
The compatibility check is done by the DynamicBackendUtils::IsBackendSupported method.
A backend is guaranteed to be compatible when it has been compiled with the same major version of ArmNN's Backend API, and an equal to or greater than minor version.
However, any minor increment in the ArmNN's Backend API will be carried out so that it won't disrupt the backends compiled against a previous minor version. Still in this case the backend is not guaranteed to benefit from the latest changes introduced in the Backend API.
Examples:
* Backend built with v2.1 API is compatible with ArmNN's Backend API v2.4 (same major version, backend built against earlier compatible API)
* Backend built with v2.5 API is **not** compatible with ArmNN's Backend API v2.4 (same major version, backend built against later incompatible API, backend might require update to the latest compatible backend API)
* Backend built with v2.4 API is **not** compatible with ArmNN's Backend API v1.0 (backend requires completely new API version)
* Backend built with v2.0 API is **not** compatible with ArmNN's Backend API v3.0 (backwards compatibility in the Backend API is broken)
It'll be useful to have a utility function to check whether a backend is compatible with the current Backend API for example:
```
bool IsBackendCompatible(const BackendVersion& backendVersion)
{
return backendVersion.m_Major == backendApiVersion.m_Major && backendVersion.m_Minor <= backendApiVersion.m_Minor;
}
```
== Create a new DynamicBackend class ==
A new DynamicBackend class can be used to store all the function pointers for a loaded dynamic backendwill be used to represent a loaded dynamic backend in ArmNN. Malformed backend shared object should not have a corresponding DynamicBackend in ArmNN's runtime.
The DynamicBackend class will need to hold (and manage) the shared object handle as well as provide the factory method for creating an instance of the backend, which will be used to create and register the backend instance in the backend registry. Once the new backend instance will be added to the backend registry, ArmNN will proceed to making use of the new backend as usual, with to further changes required to the runtime.
The DynamicBackend class will need to get the backend id of the backend for registration purposes.
The DynamicBackend class will need to get the version of the dynamic backend in the shared object so that ArmNN could verify its compatibility against the current runtime version. The shared object must then export a function for checking its version.
A new list of DynamicBackend instances should be maintained in the Runtime class.
== Use the dlfcn API to load the backend objects ==A new BackendIdFunction type should be added to the BackendRegistry to define the signature of the backend id getter method.
Proposed name and signature:
```
const char* GetBackendId() // A BackendId object could be later created by ArmNN from the given string
```
A new VersionFunction type should be added to the BackendRegistry to define the signature of the version getter method.
Proposed name and signature:
```
void GetVersion(uint32_t* outMajor, uint32_t* outMinor) // A BackendVersion object could be later created by ArmNN with the given int values
```
The BackendRegistry already defines a type for the FactoryFunction that expects a std::unique_ptr<IBackendInternal> to be returned, however the proposed name and signature for the BackendFactory function exported by the shared object is:
```
IBackendInternal* BackendFactory() // The function returns a raw pointer that ArmNN will later manage using a smart pointer
```
Once the path to the backend objects has been located, proceed to load them using the standard dlopenArmNN will try to load the functions by their name without mangling, dlclose and dlsym functions.so external C linkage must be specified when exporting the shared object symbols:
Create the following utility functions:```
* GetSharedObjects: gets a list of all the shared objects (as absolute path strings) at the path returned by GetBackendsPathextern "C"
* LoadBackend: try to load each shared object returned by GetSharedObjects as a dynamic backend.{
const char* GetBackendId();
void GetVersion(uint32_t* outMajor, An already loaded buint32_t* outMinor);
IBackend cannot be loaded twice in the same runtime instanceInternal* BackendFactory();
}
```
//Implementation hint//:
```
// While looping through a list of shared objectsusing HandleCloser = std::function<void(const void*)>;
using HandlePtr = std::unique_ptr<void, HandleCloser>;
void* sharedObjectHandle = dlopen(sharedObjectPath, RTLD_LAZY | RTLD_LOCAL);class DynamicBackend
if (!sharedObjectHandle{
public:
explicit DynamicBackend(const void* handle)
: m_GetBackendIdFunction(nullptr)
, m_GetVersionFunction(nullptr)
, m_BackendFactoryFunction(nullptr)
, m_Handle(const_cast<void*>(handle), &DynamicBackendUtils::CloseHandle)
{
if (m_Handle == nullptr)
{
throw InvalidArgumentException("Cannot create a DynamicBackend from an invalid shared object")
}
// These calls may throw
m_GetBackendIdFunction = DynamicBackendUtils::GetEntryPoint<BackendRegistry::BackendIdFunction>("GetBackendId");
m_GetVersionFunction = DynamicBackendUtils::GetEntryPoint<BackendRegistry::VersionFunction>("GetVersion");
m_BackendFactoryFunction = DynamicBackendUtils::GetEntryPoint<BackendRegistry::FactoryFunction>("BackendFactory");
}
BOOST_LOG_TRIVIAL(error) << "An error occurred trying to open the shared object \"" << sharedObjectPath << "\": " << dlerror() << "\n";// These are probably not necessary, mentioned here just in case
continuebool IsLoaded() const { return m_Handle != nullptr; }
void* Get() const { return m_Handle.get(); }
void Close() { m_Handle.reset(); // Continue to the next shared object}
void* Release()
{
void* handle = Get();
Close();
return handle;
}
...
if (sharedObjectHandle already exists in the list of Dynamic // Public dynamic backend functions
BackendId GetBackenddId()
{
assert(m_GetBackendIdFunction);
return BackendId(m_GetBackendIdFunction());
BOOST_LOG_TRIVIAL(error) << "An handle already exists for the shared object \"" << sharedObjectPath << "\n";}
continue;BackendVersion GetBackendVersion()
{
assert(m_GetVersionFunction);
uint32_t major = 0;
uint32_t minor = 0;
m_GetVersionFunction(&major, &minor);
return BackendVersion{ major, // Continue to the next shared objectminor };
}
IBackendInternalUniquePtr GetBackend()
{
assert(m_BackendFactoryFunction);
return std::move(std::unique_ptr<IBackendInternal>(m_BackendFactoryFunction()));
}
// Try to load all methods in IBackendInternalprivate:
// Backend function pointers
BackendRegistry::BackendIdFunction m_GetBackendIdFunction;
BackendRegistry::VersionFunction m_GetVersionFunction;
BackendRegistry::FactoryFunction m_BackendFactoryFunction;
// Shared object handle
HandlePtr m_Handle;
};
```
== Load the backend objects ==
Once the path to the backend objects has been located, proceed to load them using the utility functions in DynamicBackendUtils.
Create the following utility functions:
OptimizationViews (*optimizeSubgraphViewPtr)(SubgraphView&);* GetSharedObjects: gets a list of all the shared objects (as absolute path strings) at the paths returned by GetBackendPaths
optimizeSubgraphViewPtr = (OptimizationViews(*)(SubgraphView&))dlsym(s* LoadDynamicBackend: try to load each shared object returned by GetSharedObjectHandle,s as a dynamic backend. "OptimizeSubgraphView");An already loaded backend cannot be loaded twice in the same runtime instance
//Implementation hint://
```
if (!optimizeSubgraphViewPtr)// While looping through a list of shared objects
void* sharedObjectHandle = nullptr;
try
{
BOOST_LOG_TRIVIAL(error) << "An error occurred trying to open the shared object \"" << sharedObjectPath << "\": " << dlerror() << "\n";sharedObjectHandle = DynamicBackendUtils::OpenHandle(sharedObjectPath);
}
catch exceptions
{
dlclose(sharedObjectHandle);// Report exception error
continue; // Close the current shared object and move onto the nextontinue to the next shared object
}
if (!sharedObjectHandle)
{
BOOST_LOG_TRIVIAL(error) << "An error occurred trying to open the shared object \"" << sharedObjectPath << "\": " << DynamicBackendUtils::GetDlError() << "\n";
continue; // Continue to the next shared object
}
// Store the function pointer in a DynamicBackend instancestd::unique_ptr<DynamicBackend> dynamicBackend;
try
{
dynamicBackend.reset(new DynamicBackend(sharedObjectHandle));
}
catch exceptions
{
// Dynamic backend possible malformed, report exception error
continue; // Continue to the next shared object
dynamicBackend.SetOptimizeSubgraphViewPtr(optimizeSubgraphViewPtr);}
... // Repeat for all the other methods in IB// Get the backendInternal id
```BackendId backendId = dynamicBackend.GetBackendId();
* UnloadBif (backend: release the backend onceId already exists in the runtime gets destroyedbackend registry)
```{
for (DynamicBackend dynamicBackend : dynamicBackends) BOOST_LOG_TRIVIAL(error) << "A backend with id \"" << backendId << " has already been loaded\n";
dynamicBackend.reset(); // Destroy the dynamic backend
continue; // Continue to the next shared object
}
BackendVersion backendVersion = dynamicBackend.GetBackendVersion();
if (backendVersion is incompatible with the current Backend API version) // Use DynamicBackendUtils::IsBackendSupported
{
dlclose(dynamicBackend.GetSharedObjectHandle)BOOST_LOG_TRIVIAL(error) << "Backend \"" << backendId << " with version " << backendVersion << " is not compatible with the current Backend API " << backendApiVersion << "\n";
dynamicBackend.reset(); // Destroy the dynamic backend
continue; // Ignore any error// Continue to the next shared object
}
// Append the DynamicBackend object to the list of dynamic backends
// Continue to the next shared object
```
The shared object handles will have to be members of the DynamicBackendUtils class.
== Create a new LoadDynamicBackends method in the Runtime class ==
Add a new private LoadDynamicBackends method to the Runtime class that makes use of the utilities developed in the previous point.
The LoadDynamicBackends should attempt to load any shared object that's considered suitable (i.e. complientant to the IBackendInternal interface) regardless of its filename.
If a valid object is found, that backend should be registered in the BackendRegistry.
Create the following function in the Runtime class:
* LoadDynamicBackends: applies the entire procedure of getting the backend objects, load them, and register the dynamic backends in the runtime (throws an exception in case of backend registration error)
//Implementation hint//:
```
string backendsPath = GetBackendsPath();
vector<string> sharedObjects = GetSharedObjects(backendsPath);
vector<DynamicBackend> dynamicBackends;
for (string sharedObject : sharedObjects)
{
DynamicBackend dynamicBackend = LoadBackend(sharedObject);
if (success)
{
dynamicBackends.push_back(dynamicBackend);
}
}
for (DynamicBackend dynamicBackend : dynamicBackends)
{
BackendRegistryInstance().Register(dynamicBackend.GetId(), ...);// Get the backend id
BackendId backendId = dynamicBackend.GetBackendId();
// Get the backend factory function
FactoryFunction backendFactoryFunction = dynamicBackend.GetFactoryFunction();
// Register the backend
BackendRegistryInstance().Register(backendId, // throws in case of errorbackendFactoryFunction);
}
```
== Use a dynamic backend ==
Once a backend (static or dynamic) is registered in the BackendRegistry, its factory method is called automatically by the runtime as per the current implementation, no further changes should be required.
== Extend the Backends documentation ==
Update the docs with information and examples about adding a custom dynamic backend object to an ArmNN's deployment.
Detail all the steps necessary to create a dynamic backend from scratch, and how to use make use ofconsume it in ArmNN.
Mention possible/common errors (where to get the error log, the object is not found, found but not loaded, loaded but not registered, registered but not used, etc.) and give basic troubleshooting advice on what the possible causes of the problems could be and suggest a valid fix.
Mention the possibility of having dynamic backends in the existing static backends documentation as well.
== Test Strategy ==
TUnit tests to implement:
# Test that you must be able load a valid backend (create a mock dynamic backend for testing)
# Test that you must not**not** be able to load an invalid shared object (e.g. a file that is not a shared object, no symlinks, ano files that is not a shared objectdon't match the given naming scheme)
# Test that you must not**not** be able to load an invalid backend (e.g. a shared object that sidoes not a backendcomply to the given interface)
# Test that you must **not** be able to load multiple (different)an unsupported backend (e.g. a backends with an incompatible version)
# Test that you must be able to load multiple backends with the same id, but thedifferent backend registration must fail for all but the first ones
# Test that you must not**not** be able to load a backend with the same id of a backend already loaded
# Test that you must **not** be be able to load the same backend (same shared object) more than once per runtime instance
# Test that all the loaded backend are unloaded when the runtime is destroyed
# Test that all the methods of a loaded backend can be executed (use dummythe mock dynamic backend methodsfor testing)
# Test that you **cannot** start a runtime with no backends loaded
# Test that you can start multiple runtimes with the same backend(s) (a different instance of the backend has to be created per runtime by calling the backend factory)
# Test all the utility functions created for this feature (all methods in the DynamicBackendUtils class)
End to end tests:
# Run ExecuteNetwork with at least one dynamic backend (perhaps wrapping an existing backend in a shared object)