43. Optical Flow
43.1. Optical Flow Queues
VK_NV_optical_flow adds a optical flow queue type bit
VK_QUEUE_OPTICAL_FLOW_BIT_NV to VkQueueFlagBits.
Optical flow operations are supported by queues with an advertised queue
capability of VK_QUEUE_OPTICAL_FLOW_BIT_NV.
As in the case of other queue types, an application must use
vkGetPhysicalDeviceQueueFamilyProperties to query whether the physical
device has support for the Optical Flow Queue.
When the implementation reports the VK_QUEUE_OPTICAL_FLOW_BIT_NV bit
for a queue family, it advertises general support for Vulkan queue
operations described in Devices and Queues.
43.2. Optical Flow Image Formats
To enumerate the supported image formats for a specific optical flow usage, call:
// Provided by VK_NV_optical_flow
VkResult vkGetPhysicalDeviceOpticalFlowImageFormatsNV(
VkPhysicalDevice physicalDevice,
const VkOpticalFlowImageFormatInfoNV* pOpticalFlowImageFormatInfo,
uint32_t* pFormatCount,
VkOpticalFlowImageFormatPropertiesNV* pImageFormatProperties);
-
physicalDeviceis the physical device being queried. -
pOpticalFlowImageFormatInfois a pointer to a VkOpticalFlowImageFormatInfoNV structure specifying the optical flow usage for which information is returned. -
pFormatCountis a pointer to an integer related to the number of optical flow properties available or queried, as described below. -
pImageFormatPropertiesis a pointer to an array of VkOpticalFlowImageFormatPropertiesNV structures in which supported formats and image parameters are returned.
If pImageFormatProperties is NULL, then the number of optical flow
properties supported for the given physicalDevice is returned in
pFormatCount.
Otherwise, pFormatCount must point to a variable set by the user to
the number of elements in the pImageFormatProperties array, and on
return the variable is overwritten with the number of values actually
written to pImageFormatProperties.
If the value of pFormatCount is less than the number of optical flow
properties supported, at most pFormatCount values will be written to
pImageFormatProperties, and VK_INCOMPLETE will be returned
instead of VK_SUCCESS, to indicate that not all the available values
were returned.
Before creating an image to be used as a optical flow frame, obtain the
supported image creation parameters by querying with
vkGetPhysicalDeviceFormatProperties2 and
vkGetPhysicalDeviceImageFormatProperties2 using one of the reported
formats and adding VkOpticalFlowImageFormatInfoNV to the pNext
chain of VkPhysicalDeviceImageFormatInfo2.
When querying the parameters with
vkGetPhysicalDeviceImageFormatProperties2 for images used for optical
flow operations, the VkOpticalFlowImageFormatInfoNV::usage field
must contain one or more of the bits defined in
VkOpticalFlowUsageFlagBitsNV.
|
Note
|
The VkOpticalFlowImageFormatInfoNV structure is defined as:
// Provided by VK_NV_optical_flow
typedef struct VkOpticalFlowImageFormatInfoNV {
VkStructureType sType;
const void* pNext;
VkOpticalFlowUsageFlagsNV usage;
} VkOpticalFlowImageFormatInfoNV;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
usageis a bitmask of VkOpticalFlowUsageFlagBitsNV describing the intended optical flow usage of the image.
Bits which can be set in VkOpticalFlowImageFormatInfoNV::usage,
controlling optical flow usage, are:
// Provided by VK_NV_optical_flow
typedef enum VkOpticalFlowUsageFlagBitsNV {
VK_OPTICAL_FLOW_USAGE_UNKNOWN_NV = 0,
VK_OPTICAL_FLOW_USAGE_INPUT_BIT_NV = 0x00000001,
VK_OPTICAL_FLOW_USAGE_OUTPUT_BIT_NV = 0x00000002,
VK_OPTICAL_FLOW_USAGE_HINT_BIT_NV = 0x00000004,
VK_OPTICAL_FLOW_USAGE_COST_BIT_NV = 0x00000008,
VK_OPTICAL_FLOW_USAGE_GLOBAL_FLOW_BIT_NV = 0x00000010,
} VkOpticalFlowUsageFlagBitsNV;
-
VK_OPTICAL_FLOW_USAGE_INPUT_BIT_NVspecifies that the image can be used as input or reference frame for an optical flow operation. -
VK_OPTICAL_FLOW_USAGE_OUTPUT_BIT_NVspecifies that the image can be used as output flow vector map for an optical flow operation. -
VK_OPTICAL_FLOW_USAGE_HINT_BIT_NVspecifies that the image can be used as hint flow vector map for an optical flow operation. -
VK_OPTICAL_FLOW_USAGE_COST_BIT_NVspecifies that the image can be used as output cost map for an optical flow operation. -
VK_OPTICAL_FLOW_USAGE_GLOBAL_FLOW_BIT_NVspecifies that the image can be used as global flow vector for an optical flow operation.
// Provided by VK_NV_optical_flow
typedef VkFlags VkOpticalFlowUsageFlagsNV;
VkOpticalFlowUsageFlagsNV is a bitmask type for setting a mask of zero
or more VkOpticalFlowUsageFlagBitsNV.
The VkOpticalFlowImageFormatPropertiesNV structure is defined as:
// Provided by VK_NV_optical_flow
typedef struct VkOpticalFlowImageFormatPropertiesNV {
VkStructureType sType;
const void* pNext;
VkFormat format;
} VkOpticalFlowImageFormatPropertiesNV;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
formatis a VkFormat that specifies the format that can be used with the specified optical flow image usages.
43.3. Optical Flow Session
43.3.1. Optical Flow Session Object
Optical flow session objects are abstracted and represented by VkOpticalFlowSessionNV handles:
// Provided by VK_NV_optical_flow
VK_DEFINE_NON_DISPATCHABLE_HANDLE(VkOpticalFlowSessionNV)
Creating an Optical Flow Session
To create an optical flow session object, call:
// Provided by VK_NV_optical_flow
VkResult vkCreateOpticalFlowSessionNV(
VkDevice device,
const VkOpticalFlowSessionCreateInfoNV* pCreateInfo,
const VkAllocationCallbacks* pAllocator,
VkOpticalFlowSessionNV* pSession);
-
deviceis the logical device that creates the optical flow session object. -
pCreateInfois a pointer to a VkOpticalFlowSessionCreateInfoNV structure containing parameters specifying the creation of the optical flow session. -
pAllocatorcontrols host memory allocation as described in the Memory Allocation chapter. -
pSessionis a pointer to a VkOpticalFlowSessionNV handle specifying the optical flow session object which will be created by this function when it returnsVK_SUCCESS
The VkOpticalFlowSessionCreateInfoNV structure is defined as:
// Provided by VK_NV_optical_flow
typedef struct VkOpticalFlowSessionCreateInfoNV {
VkStructureType sType;
void* pNext;
uint32_t width;
uint32_t height;
VkFormat imageFormat;
VkFormat flowVectorFormat;
VkFormat costFormat;
VkOpticalFlowGridSizeFlagsNV outputGridSize;
VkOpticalFlowGridSizeFlagsNV hintGridSize;
VkOpticalFlowPerformanceLevelNV performanceLevel;
VkOpticalFlowSessionCreateFlagsNV flags;
} VkOpticalFlowSessionCreateInfoNV;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
widthis the width in pixels of the input or reference frame to be bound to this optical flow session. -
heightis the height in pixels of the input or reference frame to be bound to this optical flow session. -
imageFormatis the VkFormat of the input and reference frame to be bound to this optical flow session. -
flowVectorFormatis the VkFormat of the flow vector maps (output or hint) to be bound to this optical flow session. -
costFormatis the VkFormat of the cost maps to be bound to this optical flow session. -
outputGridSizeis exactly one bit of VkOpticalFlowGridSizeFlagsNV specifying the grid size of the output flow and cost maps to be bound to this optical flow session. The size of the output flow and cost maps is determined byVkOpticalFlowSessionCreateInfoNV::widthandVkOpticalFlowSessionCreateInfoNV::heightdivided byVkOpticalFlowSessionCreateInfoNV::outputGridSize. -
hintGridSizeis one exactly bit of VkOpticalFlowGridSizeFlagsNV specifying the grid size of the hint flow vector maps to be bound to this optical flow session. The size of the hint maps is determined byVkOpticalFlowSessionCreateInfoNV::widthandVkOpticalFlowSessionCreateInfoNV::heightdivided byVkOpticalFlowSessionCreateInfoNV::hintGridSize. -
performanceLevelis the VkOpticalFlowPerformanceLevelNV used for this optical flow session. -
flagsare the VkOpticalFlowSessionCreateFlagsNV used for this optical flow session.
The VkOpticalFlowSessionCreatePrivateDataInfoNV structure is for NV internal use only and is defined as:
// Provided by VK_NV_optical_flow
typedef struct VkOpticalFlowSessionCreatePrivateDataInfoNV {
VkStructureType sType;
void* pNext;
uint32_t id;
uint32_t size;
const void* pPrivateData;
} VkOpticalFlowSessionCreatePrivateDataInfoNV;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
idis an identifier for data which is passed at a memory location specified inVkOpticalFlowSessionCreatePrivateDataInfoNV::pPrivateData. -
sizeis the size of data in bytes which is passed at a memory location specified inVkOpticalFlowSessionCreatePrivateDataInfoNV::pPrivateData. -
pPrivateDatais a pointer to NV internal data.
Optical flow vectors are generated block-wise, one vector for each block of NxN pixels (referred to as grid).
Bits which can be set in
VkOpticalFlowSessionCreateInfoNV::outputGridSize and
VkOpticalFlowSessionCreateInfoNV::hintGridSize, or which are
returned in
VkPhysicalDeviceOpticalFlowPropertiesNV::supportedOutputGridSizes
and
VkPhysicalDeviceOpticalFlowPropertiesNV::supportedHintGridSizes
controlling optical flow grid sizes, are:
// Provided by VK_NV_optical_flow
typedef enum VkOpticalFlowGridSizeFlagBitsNV {
VK_OPTICAL_FLOW_GRID_SIZE_UNKNOWN_NV = 0,
VK_OPTICAL_FLOW_GRID_SIZE_1X1_BIT_NV = 0x00000001,
VK_OPTICAL_FLOW_GRID_SIZE_2X2_BIT_NV = 0x00000002,
VK_OPTICAL_FLOW_GRID_SIZE_4X4_BIT_NV = 0x00000004,
VK_OPTICAL_FLOW_GRID_SIZE_8X8_BIT_NV = 0x00000008,
} VkOpticalFlowGridSizeFlagBitsNV;
-
VK_OPTICAL_FLOW_GRID_SIZE_1X1_BIT_NVspecifies that grid is 1x1 pixel. -
VK_OPTICAL_FLOW_GRID_SIZE_2X2_BIT_NVspecifies that grid is 2x2 pixel. -
VK_OPTICAL_FLOW_GRID_SIZE_4X4_BIT_NVspecifies that grid is 4x4 pixel. -
VK_OPTICAL_FLOW_GRID_SIZE_8X8_BIT_NVspecifies that grid is 8x8 pixel.
// Provided by VK_NV_optical_flow
typedef VkFlags VkOpticalFlowGridSizeFlagsNV;
VkOpticalFlowGridSizeFlagsNV is a bitmask type for setting a mask of
zero or more VkOpticalFlowGridSizeFlagBitsNV.
Optical flow exposes performance levels which the user can choose based on the desired performance and quality requirement.
The optical flow performance level types are defined with the following:
// Provided by VK_NV_optical_flow
typedef enum VkOpticalFlowPerformanceLevelNV {
VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_UNKNOWN_NV = 0,
VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_SLOW_NV = 1,
VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_MEDIUM_NV = 2,
VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_FAST_NV = 3,
} VkOpticalFlowPerformanceLevelNV;
-
VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_SLOW_NVis a level with slower performance but higher quality. -
VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_MEDIUM_NVis a level with medium performance and medium quality. -
VK_OPTICAL_FLOW_PERFORMANCE_LEVEL_FAST_NVis a preset with higher performance but lower quality.
Bits which can be set in
VkOpticalFlowSessionCreateInfoNV::flags, controlling optical
flow session operations, are:
// Provided by VK_NV_optical_flow
typedef enum VkOpticalFlowSessionCreateFlagBitsNV {
VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_HINT_BIT_NV = 0x00000001,
VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_COST_BIT_NV = 0x00000002,
VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_GLOBAL_FLOW_BIT_NV = 0x00000004,
VK_OPTICAL_FLOW_SESSION_CREATE_ALLOW_REGIONS_BIT_NV = 0x00000008,
VK_OPTICAL_FLOW_SESSION_CREATE_BOTH_DIRECTIONS_BIT_NV = 0x00000010,
} VkOpticalFlowSessionCreateFlagBitsNV;
-
VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_HINT_BIT_NVspecifies that a VkImageView with external flow vectors will be used as hints in performing the motion search and must be bound toVK_OPTICAL_FLOW_SESSION_BINDING_POINT_HINT_NV. -
VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_COST_BIT_NVspecifies that the cost for the forward flow is generated in a VkImageView which must be bound toVK_OPTICAL_FLOW_SESSION_BINDING_POINT_COST_NV. Additionally, ifVK_OPTICAL_FLOW_SESSION_CREATE_BOTH_DIRECTIONS_BIT_NVis also set, the cost for backward flow is generated in a VkImageView which must be bound toVK_OPTICAL_FLOW_SESSION_BINDING_POINT_BACKWARD_COST_NV. The cost is the confidence level of the flow vector for each grid in the frame. The Cost implies how (in)accurate the flow vector is. Higher cost value implies the flow vector to be less accurate and vice-versa. -
VK_OPTICAL_FLOW_SESSION_CREATE_ENABLE_GLOBAL_FLOW_BIT_NVspecifies that a global flow vector is estimated from forward flow in a single pixel VkImageView which must be bound toVK_OPTICAL_FLOW_SESSION_BINDING_POINT_GLOBAL_FLOW_NV. -
VK_OPTICAL_FLOW_SESSION_CREATE_ALLOW_REGIONS_BIT_NVspecifies that regions of interest can be specified in VkOpticalFlowExecuteInfoNV. -
VK_OPTICAL_FLOW_SESSION_CREATE_BOTH_DIRECTIONS_BIT_NVspecifies that backward flow is generated in addition to forward flow which is always generated.
VkOpticalFlowSessionCreateFlagsNV is a bitmask type for setting a mask
of zero or more VkOpticalFlowSessionCreateFlagBitsNV.
// Provided by VK_NV_optical_flow
typedef VkFlags VkOpticalFlowSessionCreateFlagsNV;
Destroying an Optical Flow Session
To destroy a optical flow session object, call:
// Provided by VK_NV_optical_flow
void vkDestroyOpticalFlowSessionNV(
VkDevice device,
VkOpticalFlowSessionNV session,
const VkAllocationCallbacks* pAllocator);
-
deviceis the device that was used for the creation of the optical flow session. -
sessionis the optical flow session to be destroyed. -
pAllocatorcontrols host memory allocation as described in the Memory Allocation chapter.
43.3.2. Binding Vulkan Image Views to an Optical Flow Session
To bind a vulkan image to an optical flow session object, call:
// Provided by VK_NV_optical_flow
VkResult vkBindOpticalFlowSessionImageNV(
VkDevice device,
VkOpticalFlowSessionNV session,
VkOpticalFlowSessionBindingPointNV bindingPoint,
VkImageView view,
VkImageLayout layout);
-
deviceis the device which owns the optical flow session objectsession. -
sessionis the optical flow session object to which the image view is to be bound. -
bindingPointspecifies the binding point VkOpticalFlowSessionBindingPointNV to which the image view is bound. -
viewis a VkImageView to be bound. -
layout must specify the layout that the image subresources accessible from
viewwill be in at the time the optical flow vectors are calculated with vkCmdOpticalFlowExecuteNV on aVkDevice.
The optical flow session binding points are defined with the following:
// Provided by VK_NV_optical_flow
typedef enum VkOpticalFlowSessionBindingPointNV {
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_UNKNOWN_NV = 0,
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_INPUT_NV = 1,
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_REFERENCE_NV = 2,
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_HINT_NV = 3,
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_FLOW_VECTOR_NV = 4,
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_BACKWARD_FLOW_VECTOR_NV = 5,
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_COST_NV = 6,
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_BACKWARD_COST_NV = 7,
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_GLOBAL_FLOW_NV = 8,
} VkOpticalFlowSessionBindingPointNV;
-
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_INPUT_NVspecifies the binding point for the input frame. -
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_REFERENCE_NVspecifies the binding point for the input reference frame. -
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_HINT_NVspecifies the binding point for the optional external hint flow vectors. -
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_FLOW_VECTOR_NVspecifies the binding point for output flow vectors of default forward flow calcution. -
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_BACKWARD_FLOW_VECTOR_NVspecifies the binding point for the optional output flow vector map of optional backward flow calcution. -
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_COST_NVspecifies the binding point for the optional output cost map of default forward flow calcution. -
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_BACKWARD_COST_NVspecifies the binding point for the optional output cost map of optional backward flow calcution. -
VK_OPTICAL_FLOW_SESSION_BINDING_POINT_GLOBAL_FLOW_NVspecifies the binding point for the optional global flow value of default forward flow calcution.
43.3.3. Optical Flow Execution
Default direction of flow estimation is forward which calculates the optical flow from input frame to reference frame. Optionally backward flow estimation can be additionally calculated. An output flow vector (Vx, Vy) means that current pixel (x, y) of input frame can be found at location (x+Vx, y+Vy) in reference frame. A backward flow vector (Vx, Vy) means that current pixel (x, y) of reference frame can be found at location (x+Vx, y+Vy) in input frame.
To calculate optical flow vectors from two input frames, call:
// Provided by VK_NV_optical_flow
void vkCmdOpticalFlowExecuteNV(
VkCommandBuffer commandBuffer,
VkOpticalFlowSessionNV session,
const VkOpticalFlowExecuteInfoNV* pExecuteInfo);
-
commandBufferis the command buffer into which the command will be recorded. -
sessionis the optical flow session object on which this command is operating. -
pExecuteInfoInfo is a pointer to a VkOpticalFlowExecuteInfoNV.
The VkOpticalFlowExecuteInfoNV structure is defined as:
// Provided by VK_NV_optical_flow
typedef struct VkOpticalFlowExecuteInfoNV {
VkStructureType sType;
void* pNext;
VkOpticalFlowExecuteFlagsNV flags;
uint32_t regionCount;
const VkRect2D* pRegions;
} VkOpticalFlowExecuteInfoNV;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
flagsare the VkOpticalFlowExecuteFlagsNV used for this command. -
regionCountis the number of regions of interest specified inpRegions. -
pRegionsis a pointer toregionCountVkRect2Dregions of interest.
Bits which can be set in VkOpticalFlowExecuteInfoNV::flags,
controlling optical flow execution, are:
// Provided by VK_NV_optical_flow
typedef enum VkOpticalFlowExecuteFlagBitsNV {
VK_OPTICAL_FLOW_EXECUTE_DISABLE_TEMPORAL_HINTS_BIT_NV = 0x00000001,
} VkOpticalFlowExecuteFlagBitsNV;
-
VK_OPTICAL_FLOW_EXECUTE_DISABLE_TEMPORAL_HINTS_BIT_NVspecifies that temporal hints from previously generated flow vectors are not used. If temporal hints are enabled, optical flow vectors from previous vkCmdOpticalFlowExecuteNV call are automatically used as hints for the current vkCmdOpticalFlowExecuteNV call, to take advantage of temporal correlation in a video sequence. Temporal hints should be disabled if there is a-priori knowledge of no temporal correlation (e.g. a scene change, independent successive frame pairs).
VkOpticalFlowExecuteFlagsNV is a bitmask type for setting a mask of
zero or more VkOpticalFlowExecuteFlagBitsNV.
// Provided by VK_NV_optical_flow
typedef VkFlags VkOpticalFlowExecuteFlagsNV;