Material Manager — omni.sensors.nv.materials 1.0.0 documentation (2024)

Initialization

The Material Manager and its dependency upon a valid cuda device index requires that the initialization process be done at a timewhen RtxSensor provides the plugin with the cuda device. Within the context of the plugin being updated withthe cuda device (e.g. prepTrace), the following initialization can be performed:

m_materialManager.init( m_materialMap, m_hitShaderPtxString, m_entryPointName, sizeof(UserData), m_cudaDevice);

The m_materialMap is a map containing the material descriptions to be included with the key being the desired material Id.The m_hitShaderPtxString and m_entryPointName are the ptx compiled main hit shader string and the entry point name for the main hit shader respectively.The final arguments are the size of the user data to be input into the main hit shader for its argument and the cuda device index to interface with.As mentioned above, the material manager is designed to be for a single cuda context so a more well rounded solution would be to initialize multiplematerial managers per cuda device:

m_materialManager[m_cudaDevice].init( m_materialMap, m_hitShaderPtxString, m_entryPointName, sizeof(TraceData), m_cudaDevice);

The main difference here is the usage of the cuda device that is the key for resolving a material manager instance. This is done explicitlysince the material manager cannot easily integrate with the GpuMigrationHandler. Therefore, a material manager per cuda device is required.

It is important that the m_materialMap contains valid material name that properly maps to an existing material plugin. The carbonite registered name within that pluginis keyed upon at this stage and must be resolved for proper execution. Failure to ensure this is correct will result in no materials mapped andno meaningful returns for the desired sensor.

Note. The initialization stage is where the BSDF material fully defines a context via getContext. This is how the host side material propertydata is passed into the context that is then written to the device side for kernel execution. This the the important link on how the device side BSDFaccesses the data for computing reflection/transmission functions. See Custom Materials for more details.

std::map<uint32_t, omni::sensors::materials::MaterialDesc> m_materialMap;

using omni::sensors::materials::SpectralRange;using omni::sensors::materials::NvMatConfig;SpectralRange spectralRange {903.0f};m_materialMap[0] = { "CoreMaterial", std::make_shared<NvMatConfig>("RubberStandard", spectralRange) };// Alternativelym_materialMap[1] = std::make_pair("CoreMaterial", NvMatConfig("PlasticStandard", spectralRange));

Initialization Example

The initialization example here can be seen in the RtxSensorSample::setupMaterials() method in the sample_rtxsensor plugin:

if (m_materialManager.empty()){ // Add material manager for device in map m_materialManager.emplace(m_cudaDevice, omni::drivesim::sensors::materials::MaterialManager()); // Add material to the mapping -> can be changed anytime m_materialMap[0] = { "SampleGeneric" }; m_hitShaderPtxString = (const char*)imageBytes; // name of the function in the hitshader m_entryPointName = "ProcessReturns_CUDA"; // init manager m_materialManager[m_cudaDevice].init( m_materialMap, m_hitShaderPtxString, m_entryPointName, sizeof(TraceData), m_cudaDevice);} // MaterialManager cannot be handled by the migration handler -- so do it yourselfelse if (!m_materialManager.empty() && m_materialManager.find(m_cudaDevice) == m_materialManager.end()){ m_materialManager.emplace(m_cudaDevice, omni::drivesim::sensors::materials::MaterialManager()); m_materialManager[m_cudaDevice].init( m_materialMap, m_hitShaderPtxString, m_entryPointName, sizeof(TraceData), m_cudaDevice);}

This code demonstrates the index and material name provided to the material map, In addition, the entry point name for the main hit shader,and the manager creation and initialization is performed with the perscribed input quantities. The imageBytes is the main hit shader datathat has already been compiled to ptx. Compiling the main hit shader and material plugin shaders are required for proper usage of the manager.

Re-initialization

Referencing the same example above for material manager initialization, the code can add additional conditionals for runtime re-initialization.:

if (m_materialManager.empty()){ // Add material manager for device in map m_materialManager.emplace(m_cudaDevice, omni::drivesim::sensors::materials::MaterialManager()); // Add material to the mapping -> can be changed anytime m_materialMap[0] = { "SampleGeneric" }; m_hitShaderPtxString = (const char*)imageBytes; // name of the function in the hitshader m_entryPointName = "ProcessReturns_CUDA"; // init manager m_materialManager[m_cudaDevice].init( m_materialMap, m_hitShaderPtxString, m_entryPointName, sizeof(TraceData), m_cudaDevice);} // MaterialManager cannot be handled by the migration handler -- so do it yourselfelse if (!m_materialManager.empty() && m_materialManager.find(m_cudaDevice) == m_materialManager.end()){ m_materialManager[m_cudaDevice].resetMaterialManager(); m_materialManager[m_cudaDevice].init( m_materialMap, m_hitShaderPtxString, m_entryPointName, sizeof(TraceData), m_cudaDevice);}

The only difference here is that rather creating a new material manager and initializing, it can be reset and then re-initialized.This allows the existing materials within the map to persist but be reprocessed allowing any changes to input data or behaviors.

Material Processing

Within each trace from RtxSensor, either a single or a series of batchBegin and batchEnd calls are made. Since the material manager must dosomething with the returned hits from the ray traced scene, material processing is executed from within the batchEnd notification. Thisprovides the necessary access to the RtxSensorReturns. The function call to process returns performs this action:

omni::sensors::materials::MaterialManager::LaunchCfg launchCfg;UserData m_userData;m_materialManager->processReturns(launchCfg, &m_userData);

The userData is a pointer to some user defined structure that is the input argument to the main hit shader. The init method required thesize of this structure to preallocate the memory for sensor processing within batchEnd. The launchCfg is a structure within the Material Managerthat defines the device side details for launching the main hit shader and subsequent material processing. The LaunchCfg structure has thefollowing layout:

struct LaunchCfg{ dim3 blocksPerGrid; dim3 threadsPerBlock; cudaStream_t cudaStream; unsigned int sharedMemBytes{ 0 }; int cudaDevice;};

Allowing for multiple cuda devices, the material manager map can be leveraged to suffice a multi device solution:

omni::sensors::materials::MaterialManager::LaunchCfg launchCfg;UserData m_userData;m_materialManager[m_cudaDevice]->processReturns(launchCfg, &m_userData);

Each material that is registered with the MaterialManager has a cuda BSDF (BiDirectional Scattering Distribution Function)kernel, compiled to ptx, that determines the reflectance and transmission at the boundary. This unique material BSDF program is resolvedby the MaterialManager and called from within the hit shader through the processMaterialById function:

processMaterialById(void* outBlob, void* inBlob, unsigned int materialId)

The materialId is the index that resolves which BSDF shader to call and the outBlob and inBlob are the output and input structureto the BSDF. The material input and output structures are defined within the following header file:

#include <omni/sensors/materials/MaterialDefines.h>

Though the materials interface has defined structures for input and output, the fact that these arguments are void* types signifiesthat the MaterialManager infrastructure can support any user defined BSDF kernel for material processing. The structureswithin the MaterialDefines.h are the current members for material BSDF computations and have the following definitions:

enum class NvMatInFlags : uint8_t{ CALC_R = 1, CALC_T = 2, CALC_PHASE_AND_POLARIZED = 4, CALC_INTERNAL_TERMS = 8,};

The NvMatInFlags enumeration defines which components within the BSDF to compute. The BSDF will perform a bitwise OR operation to determinewhich additional components to compute. The CALC_R component computes the reflectance along the mirror reflection angle direction. The CALC_Tcomponent computes the transmission component in the direction defined by the input direction and the material properties. The CALC_PHASE_AND_POLARIZEDoption instructs the BSDF to compute component wise transverse electric and transverse magnetic polarized values and phase tracking. It is theresponsibility of the sensor model to define sensor space aperture polarization (aka analyzer) for polarized returns that are phase tracked. FinallyCALC_INTERNAL_TERMS is an option that works in conjunction with paints and coating layering on the base material return. Light can reflect many timeswithin a layer adding reflection and transmission terms to the base reflection and transmission options selected. This provides an additional levelof accuracy when paint and coatings are applied to a base material layer. These options are implemented in the current Nvidia based BSDFfunctionality. User defined BSDF would need to implement the behavior for such flags if desired:

struct NvMatRayProps{ float3 vector;};

The NvMatRayProps structures are members within the input and output structures for the BSDF (shown below) which are filled once the reflection and transmissiondata is calculated. It stores the reflected or transmitted coefficients from the surface boundary and returned to the sensor model hit shader for additionalprocessing.

Given the generic infrastructure behind the material manager and its ability to operate on user defined BSDF behavior types, the NvMatRayProps can be extendedand used in user defined BSDF types. By inheriting from the NvMatRayProps and adding variables unique to any custom BSDF, additional information can be storedand passed back to the sensor model’s hit shader for any user defined additional processing beyond just reflection and transmission functions. User defined materialsis discussed more thoroughly in the Custom Materials section.:

struct NvMatInput{ float3 matNormal; float3 lookupRayDir; float3 hitPoint; uint32_t materialId{ 0 }; float thickness{ 0 }; curandStateType randomState; uint8_t flags{ 0 }; float3 incRayDir; float3 diffuseRefl{ make_float3(-0.5f, -0.5f, -0.5f) }; float roughness{ 0.f }; NvMatRayProps* incRayProps; uint32_t customPropsSize{ 0 }; float distPrevHit{ 0 }; float sourceDivergence{ 0.f };};

The NvMatInput structure contains input details required for computing material reflection and transmission information. The matNormalis the normal of the current hit point from within the hit shader. The lookupRayDir is an optional member that defines a look directionIf this look direction is a zero vector, then no computations for the output lookupRayProps will be computed. If the look vector is a non zeronormalized vector, then the reflection components along that view direction are computed. The hitPoint member is the location where the ray intersectedthe scene. All of these members are in relative sensor coordinate space. The materialId is the material index from the intersected geometry that isreturned from the RtxSensorReturn. This id is passed down to the BSDF via the input structure to provide material index information. The thicknessprovides the at normal incidence depth of the primitive that is hit from the ray traced scene. The randomState is for the cuRand library to compute randomnumbers for injecting some material variances. the flag is a control for the to identify how reflection or transmission components are computed (see NvMatInFlags above).Its value is bitwise ORed with the NvMatInFlags to determine whether to include mirror direction reflection and/or transmission components, phase and polarization, and internal layer contributions.The incRayDir is self explanatory in that it defines the direction of the incident ray in sensor space. The diffuseRefl flag is an optional parameter that definesthe visible material diffuse color from the hit geometry. This information can be used for providing additonal information for the BSDF to use for more advanced processing.One such example is providing paint color variants for a single base material type. The roughness variable is another optional additional information piece that cancome from the visible material data. The roughness can be used to provide additional variances (i.e. perturbed normals) derived from visible band content modeling.The incRayProps allows the vector to define an input starting point. Rather than assuming 1 for the input, it can be in any unit space such that the BSDF behaviorcan augment for specific return spaces (i.e. unitless reflectance/transmission, Power space, Radiance space, etc). Note that the input is a pointer to the NvMatRayProps.This is by design to support generic user defined capabilities. Rather than just using the base vector in the structure to store input states, a user defined derived classtypecan store additional information that the BSDF can leverage. This is one of the primary strengths behind the Material Infrastructure. The customPropsSize supportsthe derived aspect of the NvMatRayProps. It is used to keep track of the size of a derived NvMatRayProps. The distPrevHit stores the previous hit distance foraccumulating distances where continuation rays can be cast from previous ray hit point. Finally, the sourceDivergence is a variable that is optional to define how thesource radiation diverges. This can be important for the sensor model since the radiometry in any BSDF may need to account for sufficiently diverging source illumination

Note that the flags that control the BSDF behavior details are Nvidia BSDF supported features. A user defined BSDF can define different flags with associated functionality.Also, the diffuseRefl and roughness parameters are dependent upon flags to rtxSensor. If the following flags are not set, the rtxSensorReturn struct that storesthese values will be nullptr.:

// result is a rtx::rtxsensor::RtxSensorRequirements* type coming from the rtxSensor call to getModelRequirementsresult->returnDataConfig |= rtx::rtxsensor::RtxSensorReturnData::RTXSENSOR_RETURN_MATERIAL_REFLECTANCE;result->returnDataConfig |= rtx::rtxsensor::RtxSensorReturnData::RTXSENSOR_RETURN_MATERIAL_ADDITIONAL;

The following structure defines the output structure that the BSDF fills for the model to consume:

struct NvMatOutput{ float3 exitPoint; float distThroughCurMat{ 0.f }; uint32_t materialId{ 0 }; NvMatRayProps* lookupRayProps; float3 reflRayDir; NvMatRayProps* reflRayProps; float3 transRayDir; NvMatRayProps* transRayProps;};

The NvMatOutput structure stores necessary elements for reflection and transmission at a boundary surface. The exitPoint details where theray has exited a material in sensor local space. The distanceThroughCurMat holds the length through the material that is transmission angledependent. The materialId is the index from the currently intersected geometry from the traced ray. It is typically the same index as thematerial index from the NvMatInput structure. The lookupRayProps is the view direction reflection components. The reflection values are storedas a float3 to account for both unpolarized and polarized forms of reflection for general sensor model support. The reflRayProps and transRayPropshold the details about the reflected and transmitted ray. The reflRayDir and transRayDir contain idealized reflected direction and materially definedtransmission direction respectively with respect to the normal vector. Additionally, the “vector” member in the NvMatRayPropsstores the reflection and transmission data both in polarized and unpolarized form. If the input view vector has zero length, the reflected ray propscompute the mirror angle reflection components. If the view vector is non zero, it will determine reflectance data as a function of the view vector.This structure can be extended to add more user defined data. Note that the NvMatRayProps are pointers to allow for extended user defined behavior.

Material Processing Example

The following depicts how the material manager is used to process returns within the batchEnd rtxSensor stage:

// Hit shader will be called through the material manager// Fill launchConfig for material managerconst uint32_t numThreadsPerBlock{ std::min((uint32_t)256, m_userData.numRays) };const uint32_t numBlocks{ DivideRoundUp(m_userData.numRays, numThreadsPerBlock) };omni::drivesim::sensors::materials::MaterialManager::LaunchCfg launchCfg;launchCfg.blocksPerGrid.x = numBlocks;launchCfg.blocksPerGrid.y = 1;launchCfg.blocksPerGrid.z = 1;launchCfg.threadsPerBlock.x = numThreadsPerBlock;launchCfg.threadsPerBlock.y = 1;launchCfg.threadsPerBlock.z = 1;launchCfg.cudaDevice = cudaDevice;launchCfg.cudaStream = cudaStream;m_materialManager[m_cudaDevice]->processReturns(launchCfg, &m_userData);

As mentioned earlier, the m_userData structure is a complete structure that contains all necessary data requiredfor the main hit shader. As shown here, it provides information that helps segment number of blocks and number ofthreads per block.

The sample_rtxsensor example illustrates the full scope of the material manager usage.