42. Video Coding
Vulkan implementations may expose one or more queue families supporting video coding operations. These operations are performed by recording them into a command buffer within a video coding scope, and submitting them to queues with compatible video coding capabilities.
The Vulkan video functionalities are designed to be made available through a set of APIs built on top of each other, consisting of:
-
A core API providing common video coding functionalities,
-
APIs providing codec-independent video decode and video encode related functionalities, respectively,
-
Additional codec-specific APIs built on top of those.
This chapter details the fundamental components and operations of these.
42.1. Video Picture Resources
In the context of video coding, multidimensional arrays of image data that can be used as the source or target of video coding operations are referred to as video picture resources. They may store additional metadata that includes implementation-private information used during the execution of video coding operations, as discussed later.
Video picture resources are backed by VkImage objects. Individual subregions of VkImageView objects created from such resources can be used as decode output pictures, encode input pictures, reconstructed pictures, and/or reference pictures.
The parameters of a video picture resource are specified using a
VkVideoPictureResourceInfoKHR structure.
The VkVideoPictureResourceInfoKHR structure is defined as:
// Provided by VK_KHR_video_queue
typedef struct VkVideoPictureResourceInfoKHR {
VkStructureType sType;
const void* pNext;
VkOffset2D codedOffset;
VkExtent2D codedExtent;
uint32_t baseArrayLayer;
VkImageView imageViewBinding;
} VkVideoPictureResourceInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
codedOffsetis the offset in texels of the image subregion to use. -
codedExtentis the size in pixels of the coded image data. -
baseArrayLayeris the array layer of the image view specified inimageViewBindingto use as the video picture resource. -
imageViewBindingis an image view representing the video picture resource.
The image subresource referred to by such a structure is defined as the
image array layer index specified in baseArrayLayer relative to the
image subresource range the image view specified in imageViewBinding
was created with.
The meaning of the codedOffset and codedExtent depends on the
command and context the video picture resource is used in, as well as on the
used video profile and corresponding codec-specific
semantics, as described later.
A video picture resource is uniquely defined by the image subresource
referred to by an instance of this structure, together with the
codedOffset and codedExtent members that identify the image
subregion within the image subresource referenced corresponding to the video
picture resource according to the particular codec-specific semantics.
Accesses to image data within a video picture resource happen at the
granularity indicated by
VkVideoCapabilitiesKHR::pictureAccessGranularity, as returned by
vkGetPhysicalDeviceVideoCapabilitiesKHR for the used video profile.
As a result, given an effective image subregion corresponding to a video
picture resource, the actual image subregion accessed may be larger than
that as it may include additional padding texels due to the picture access
granularity.
Any writes performed by video coding operations to such padding texels will
result in undefined texel values.
Two video picture resources match if they refer to the same image
subresource and they specify identical codedOffset and
codedExtent values.
42.2. Decoded Picture Buffer
An integral part of video coding pipelines is the reconstruction of pictures from a compressed video bitstream. A reconstructed picture is a video picture resource resulting from this process.
Such reconstructed pictures can be used as reference pictures in subsequent video coding operations to provide predictions of the values of samples of subsequently decoded or encoded pictures. The correct use of such reconstructed pictures as reference pictures is driven by the video compression standard, the implementation, and the application-specific use cases.
The list of reference pictures used to provide such predictions within a single video coding operation is referred to as the list of active reference pictures.
The decoded picture buffer (DPB) is an indexed data structure that
maintains the set of reference pictures available to be used in video coding
operations.
Individual indexed entries of the DPB are referred to as the
decoded picture buffer (DPB) slots.
The range of valid DPB slot indices is between zero and
N-1, where N is the capacity of the DPB.
Each DPB slot can refer to a reference picture containing a video frame
or can refer to up to two reference pictures containing the top and/or
bottom fields that, when both present, together represent a full video
frame
.
In Vulkan, the state and the backing store of the DPB is separated as follows:
-
The state of individual DPB slots is maintained by video session objects.
-
The backing store of DPB slots is provided by subregions of VkImage objects used as video picture resources.
In addition, the implementation may also maintain opaque metadata associated with DPB slots, including:
Such metadata may be stored by the implementation as part of the DPB slot state maintained by the video session, or as part of the video picture resource backing the DPB slot.
Any metadata stored in the video picture resources backing DPB slots are independent of the video session used to store it, hence such video picture resources can be shared with other video sessions. Correspondingly, any metadata that is dependent on the video session will always be stored as part of the DPB slot state maintained by that video session.
The responsibility of managing the DPB is split between the application and the implementation as follows:
-
The application maintains the association between DPB slot indices and corresponding video picture resources.
-
The implementation maintains global and per-slot opaque reference picture metadata.
In addition, the application is also responsible for managing the mapping between the codec-specific picture IDs and DPB slots, and any other codec-specific states unless otherwise specified.
42.2.1. DPB Slot States
At a given time, each DPB slot is either in active or inactive state. Initially, all DPB slots managed by a video session are in inactive state.
A DPB slot can be activated by using it as the target of picture reconstruction in a video coding operation with the reconstructed picture requested to be set up as a reference picture, according to the codec-specific semantics, changing its state to active and associating it with a picture reference to the reconstructed pictures.
Some video coding standards allow multiple picture references to be associated with a single DPB slot. In this case the state of the individual picture references can be independently updated.
|
Note
As an example, H.264 decoding allows associating a separate top field and bottom field picture with the same DPB slot. |
As part of reference picture setup, the implementation may also generate reference picture metadata. Such reference picture metadata is specific to each picture reference associated with the DPB slot.
If such a video coding operation completes successfully, the activated DPB slot will have a valid picture reference and the reconstructed picture is associated with the DPB slot. This is true even if the DPB slot is used as the target of a picture reconstruction that only sets up a top field or bottom field reference picture and thus does not yet refer to a complete frame. However, if any data provided as input to such a video coding operation is not compliant to the video compression standard used, that video coding operation may complete unsuccessfully, in which case the activated DPB slot will have an invalid picture reference. This is true even if the DPB slot previously had a valid picture reference to a top field or bottom field reference picture, but the reconstruction of the other field corresponding to the DPB slot failed.
The application can use queries to get feedback about the outcome of video coding operations and use the resulting VkQueryResultStatusKHR value to determine whether the video coding operation completed successfully (result status is positive) or unsuccessfully (result status is negative).
Using a reference picture associated with a DPB slot that has an invalid picture reference as an active reference picture in subsequent video coding operations is legal, however, the contents of the outputs of such operations are undefined, and any DPB slots activated by such video coding operations will also have an invalid picture reference. This is true even if such video coding operations may otherwise complete successfully.
A DPB slot can also be deactivated by the application, changing its state to inactive and invalidating any picture references and reference picture metadata associated with the DPB slot.
If an already active DPB slot is used as the target of picture reconstruction in a video coding operation, but the decoded picture is not requested to be set up as a reference picture, according to the codec-specific semantics, no reference picture setup happens and the corresponding picture reference and reference picture metadata is invalidated within the DPB slot. If the DPB slot no longer has any associated picture references after such an operation, the DPB slot is implicitly deactivated.
If an already active DPB slot is used as the target of picture reconstruction when decoding a field picture that is not marked as reference, then the behavior is as follows:
-
If the DPB slot is currently associated with a frame, then the DPB slot is deactivated.
-
If the DPB slot is not currently associated with a top field picture and the decoded picture is a top field picture, or if the DPB slot is not currently associated with a bottom field picture and the decoded picture is a bottom field picture, then the other field picture association of the DPB slot, if any, is not disturbed.
-
If the DPB slot is currently associated with a top field picture and the decoded picture is a top field picture, or if the DPB slot is currently associated with a bottom field picture and the decoded picture is a bottom field picture, then that picture association is invalidated, without disturbing the other field picture association, if any. If the DPB slot no longer has any associated picture references after such an operation, the DPB slot is implicitly deactivated.
A DPB slot can be activated with a new frame even if it is already active. In this case all previous associations of the DPB slots with reference pictures are replaced with an association with the reconstructed picture used to activate it.
If an already active DPB slot is activated with a reconstructed field picture, then the behavior is as follows:
-
If the DPB slot is currently associated with a frame, then that association is replaced with an association with the reconstructed field picture used to activate it.
-
If the DPB slot is not currently associated with a top field picture and the DPB slot is activated with a top field picture, or if the DPB slot is not currently associated with a bottom field picture and the DPB slot is activated with a bottom field picture, then the DPB slot is associated with the reconstructed field picture used to activate it, without disturbing the other field picture association, if any.
-
If the DPB slot is currently associated with a top field picture and the DPB slot is activated with a new top field picture, or if the DPB slot is currently associated with a bottom field picture and the DPB slot is activated with a new bottom field picture, then that association is replaced with an association with the reconstructed field picture used to activate it, without disturbing the other field picture association, if any.
42.3. Video Profiles
The VkVideoProfileInfoKHR structure is defined as follows:
// Provided by VK_KHR_video_queue
typedef struct VkVideoProfileInfoKHR {
VkStructureType sType;
const void* pNext;
VkVideoCodecOperationFlagBitsKHR videoCodecOperation;
VkVideoChromaSubsamplingFlagsKHR chromaSubsampling;
VkVideoComponentBitDepthFlagsKHR lumaBitDepth;
VkVideoComponentBitDepthFlagsKHR chromaBitDepth;
} VkVideoProfileInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
videoCodecOperationis a VkVideoCodecOperationFlagBitsKHR value specifying a video codec operation. -
chromaSubsamplingis a bitmask of VkVideoChromaSubsamplingFlagBitsKHR specifying video chroma subsampling information. -
lumaBitDepthis a bitmask of VkVideoComponentBitDepthFlagBitsKHR specifying video luma bit depth information. -
chromaBitDepthis a bitmask of VkVideoComponentBitDepthFlagBitsKHR specifying video chroma bit depth information.
Video profiles are provided as input to video capability queries such as vkGetPhysicalDeviceVideoCapabilitiesKHR or vkGetPhysicalDeviceVideoFormatPropertiesKHR, as well as when creating resources to be used by video coding operations such as images, buffers, query pools, and video sessions.
The full description of a video profile is specified by an instance of this
structure, and the codec-specific and auxiliary structures provided in its
pNext chain.
When this structure is specified as an input parameter to
vkGetPhysicalDeviceVideoCapabilitiesKHR, or through the
pProfiles member of an VkVideoProfileListInfoKHR structure in
the pNext chain of the input parameter of a query command such as
vkGetPhysicalDeviceVideoFormatPropertiesKHR or
vkGetPhysicalDeviceImageFormatProperties2, the following error codes
indicate specific causes of the failure of the query operation:
-
VK_ERROR_VIDEO_PICTURE_LAYOUT_NOT_SUPPORTED_KHRindicates that the requested video picture layout (e.g. through thepictureLayoutmember of a VkVideoDecodeH264ProfileInfoKHR structure included in thepNextchain ofVkVideoProfileInfoKHR) is not supported. -
VK_ERROR_VIDEO_PROFILE_OPERATION_NOT_SUPPORTED_KHRindicates that a video profile operation specified byvideoCodecOperationis not supported. -
VK_ERROR_VIDEO_PROFILE_FORMAT_NOT_SUPPORTED_KHRindicates that video format parameters specified bychromaSubsampling,lumaBitDepth, orchromaBitDepthare not supported. -
VK_ERROR_VIDEO_PROFILE_CODEC_NOT_SUPPORTED_KHRindicates that the codec-specific parameters corresponding to the video codec operation are not supported.
Possible values of VkVideoProfileInfoKHR::videoCodecOperation,
specifying the type of video coding operation and video compression standard
used by a video profile, are:
// Provided by VK_KHR_video_queue
typedef enum VkVideoCodecOperationFlagBitsKHR {
VK_VIDEO_CODEC_OPERATION_NONE_KHR = 0,
// Provided by VK_KHR_video_encode_h264
VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR = 0x00010000,
// Provided by VK_KHR_video_encode_h265
VK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR = 0x00020000,
// Provided by VK_KHR_video_decode_h264
VK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR = 0x00000001,
// Provided by VK_KHR_video_decode_h265
VK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR = 0x00000002,
} VkVideoCodecOperationFlagBitsKHR;
-
VK_VIDEO_CODEC_OPERATION_NONE_KHRindicates no support for any video codec operations. -
VK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHRspecifies support for H.264 decode operations. -
VK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHRspecifies support for H.265 decode operations. -
VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHRspecifies support for H.264 encode operations. -
VK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHRspecifies support for H.265 encode operations.
// Provided by VK_KHR_video_queue
typedef VkFlags VkVideoCodecOperationFlagsKHR;
VkVideoCodecOperationFlagsKHR is a bitmask type for setting a mask of
zero or more VkVideoCodecOperationFlagBitsKHR.
The video format chroma subsampling is defined with the following enums:
// Provided by VK_KHR_video_queue
typedef enum VkVideoChromaSubsamplingFlagBitsKHR {
VK_VIDEO_CHROMA_SUBSAMPLING_INVALID_KHR = 0,
VK_VIDEO_CHROMA_SUBSAMPLING_MONOCHROME_BIT_KHR = 0x00000001,
VK_VIDEO_CHROMA_SUBSAMPLING_420_BIT_KHR = 0x00000002,
VK_VIDEO_CHROMA_SUBSAMPLING_422_BIT_KHR = 0x00000004,
VK_VIDEO_CHROMA_SUBSAMPLING_444_BIT_KHR = 0x00000008,
} VkVideoChromaSubsamplingFlagBitsKHR;
-
VK_VIDEO_CHROMA_SUBSAMPLING_MONOCHROME_BIT_KHRspecifies that the format is monochrome. -
VK_VIDEO_CHROMA_SUBSAMPLING_420_BIT_KHRspecified that the format is 4:2:0 chroma subsampled, i.e. the two chroma components are sampled horizontally and vertically at half the sample rate of the luma component. -
VK_VIDEO_CHROMA_SUBSAMPLING_422_BIT_KHR- the format is 4:2:2 chroma subsampled, i.e. the two chroma components are sampled horizontally at half the sample rate of luma component. -
VK_VIDEO_CHROMA_SUBSAMPLING_444_BIT_KHR- the format is 4:4:4 chroma sampled, i.e. all three components of the Y′CBCR format are sampled at the same rate, thus there is no chroma subsampling.
Chroma subsampling is described in more detail in the Chroma Reconstruction section.
// Provided by VK_KHR_video_queue
typedef VkFlags VkVideoChromaSubsamplingFlagsKHR;
VkVideoChromaSubsamplingFlagsKHR is a bitmask type for setting a mask
of zero or more VkVideoChromaSubsamplingFlagBitsKHR.
Possible values for the video format component bit depth are:
// Provided by VK_KHR_video_queue
typedef enum VkVideoComponentBitDepthFlagBitsKHR {
VK_VIDEO_COMPONENT_BIT_DEPTH_INVALID_KHR = 0,
VK_VIDEO_COMPONENT_BIT_DEPTH_8_BIT_KHR = 0x00000001,
VK_VIDEO_COMPONENT_BIT_DEPTH_10_BIT_KHR = 0x00000004,
VK_VIDEO_COMPONENT_BIT_DEPTH_12_BIT_KHR = 0x00000010,
} VkVideoComponentBitDepthFlagBitsKHR;
-
VK_VIDEO_COMPONENT_BIT_DEPTH_8_BIT_KHRspecifies a component bit depth of 8 bits. -
VK_VIDEO_COMPONENT_BIT_DEPTH_10_BIT_KHRspecifies a component bit depth of 10 bits. -
VK_VIDEO_COMPONENT_BIT_DEPTH_12_BIT_KHRspecifies a component bit depth of 12 bits.
// Provided by VK_KHR_video_queue
typedef VkFlags VkVideoComponentBitDepthFlagsKHR;
VkVideoComponentBitDepthFlagsKHR is a bitmask type for setting a mask
of zero or more VkVideoComponentBitDepthFlagBitsKHR.
Additional information about the video decode use case can be provided by
adding a VkVideoDecodeUsageInfoKHR structure to the pNext chain
of VkVideoProfileInfoKHR.
The VkVideoDecodeUsageInfoKHR structure is defined as:
// Provided by VK_KHR_video_decode_queue
typedef struct VkVideoDecodeUsageInfoKHR {
VkStructureType sType;
const void* pNext;
VkVideoDecodeUsageFlagsKHR videoUsageHints;
} VkVideoDecodeUsageInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
videoUsageHintsis a bitmask of VkVideoDecodeUsageFlagBitsKHR specifying hints about the intended use of the video decode profile.
The following bits can be specified in
VkVideoDecodeUsageInfoKHR::videoUsageHints as a hint about the
video decode use case:
// Provided by VK_KHR_video_decode_queue
typedef enum VkVideoDecodeUsageFlagBitsKHR {
VK_VIDEO_DECODE_USAGE_DEFAULT_KHR = 0,
VK_VIDEO_DECODE_USAGE_TRANSCODING_BIT_KHR = 0x00000001,
VK_VIDEO_DECODE_USAGE_OFFLINE_BIT_KHR = 0x00000002,
VK_VIDEO_DECODE_USAGE_STREAMING_BIT_KHR = 0x00000004,
} VkVideoDecodeUsageFlagBitsKHR;
-
VK_VIDEO_DECODE_USAGE_TRANSCODING_BIT_KHRspecifies that video decoding is intended to be used in conjunction with video encoding to transcode a video bitstream with the same and/or different codecs. -
VK_VIDEO_DECODE_USAGE_OFFLINE_BIT_KHRspecifies that video decoding is intended to be used to consume a local video bitstream. -
VK_VIDEO_DECODE_USAGE_STREAMING_BIT_KHRspecifies that video decoding is intended to be used to consume a video bitstream received as a continuous flow over network.
|
Note
There are no restrictions on the combination of bits that can be specified by the application. However, applications should use reasonable combinations in order for the implementation to be able to select the most appropriate mode of operation for the particular use case. |
// Provided by VK_KHR_video_decode_queue
typedef VkFlags VkVideoDecodeUsageFlagsKHR;
VkVideoDecodeUsageFlagsKHR is a bitmask type for setting a mask of
zero or more VkVideoDecodeUsageFlagBitsKHR.
Additional information about the video encode use case can be provided by
adding a VkVideoEncodeUsageInfoKHR structure to the pNext chain
of VkVideoProfileInfoKHR.
The VkVideoEncodeUsageInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_queue
typedef struct VkVideoEncodeUsageInfoKHR {
VkStructureType sType;
const void* pNext;
VkVideoEncodeUsageFlagsKHR videoUsageHints;
VkVideoEncodeContentFlagsKHR videoContentHints;
VkVideoEncodeTuningModeKHR tuningMode;
} VkVideoEncodeUsageInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
videoUsageHintsis a bitmask of VkVideoEncodeUsageFlagBitsKHR specifying hints about the intended use of the video encode profile. -
videoContentHintsis a bitmask of VkVideoEncodeContentFlagBitsKHR specifying hints about the content to be encoded using the video encode profile. -
tuningModeis a VkVideoEncodeTuningModeKHR value specifying the tuning mode to use when encoding with the video profile.
The following bits can be specified in
VkVideoEncodeUsageInfoKHR::videoUsageHints as a hint about the
video encode use case:
// Provided by VK_KHR_video_encode_queue
typedef enum VkVideoEncodeUsageFlagBitsKHR {
VK_VIDEO_ENCODE_USAGE_DEFAULT_KHR = 0,
VK_VIDEO_ENCODE_USAGE_TRANSCODING_BIT_KHR = 0x00000001,
VK_VIDEO_ENCODE_USAGE_STREAMING_BIT_KHR = 0x00000002,
VK_VIDEO_ENCODE_USAGE_RECORDING_BIT_KHR = 0x00000004,
VK_VIDEO_ENCODE_USAGE_CONFERENCING_BIT_KHR = 0x00000008,
} VkVideoEncodeUsageFlagBitsKHR;
-
VK_VIDEO_ENCODE_USAGE_TRANSCODING_BIT_KHRspecifies that video encoding is intended to be used in conjunction with video decoding to transcode a video bitstream with the same and/or different codecs. -
VK_VIDEO_ENCODE_USAGE_STREAMING_BIT_KHRspecifies that video encoding is intended to be used to produce a video bitstream that is expected to be sent as a continuous flow over network. -
VK_VIDEO_ENCODE_USAGE_RECORDING_BIT_KHRspecifies that video encoding is intended to be used for real-time recording for offline consumption. -
VK_VIDEO_ENCODE_USAGE_CONFERENCING_BIT_KHRspecifies that video encoding is intended to be used in a video conferencing scenario.
|
Note
There are no restrictions on the combination of bits that can be specified by the application. However, applications should use reasonable combinations in order for the implementation to be able to select the most appropriate mode of operation for the particular use case. |
// Provided by VK_KHR_video_encode_queue
typedef VkFlags VkVideoEncodeUsageFlagsKHR;
VkVideoEncodeUsageFlagsKHR is a bitmask type for setting a mask of
zero or more VkVideoEncodeUsageFlagBitsKHR.
The following bits can be specified in
VkVideoEncodeUsageInfoKHR::videoContentHints as a hint about the
encoded video content:
// Provided by VK_KHR_video_encode_queue
typedef enum VkVideoEncodeContentFlagBitsKHR {
VK_VIDEO_ENCODE_CONTENT_DEFAULT_KHR = 0,
VK_VIDEO_ENCODE_CONTENT_CAMERA_BIT_KHR = 0x00000001,
VK_VIDEO_ENCODE_CONTENT_DESKTOP_BIT_KHR = 0x00000002,
VK_VIDEO_ENCODE_CONTENT_RENDERED_BIT_KHR = 0x00000004,
} VkVideoEncodeContentFlagBitsKHR;
-
VK_VIDEO_ENCODE_CONTENT_CAMERA_BIT_KHRspecifies that video encoding is intended to be used to encode camera content. -
VK_VIDEO_ENCODE_CONTENT_DESKTOP_BIT_KHRspecifies that video encoding is intended to be used to encode desktop content. -
VK_VIDEO_ENCODE_CONTENT_RENDERED_BIT_KHRspecified that video encoding is intended to be used to encode rendered (e.g. game) content.
|
Note
There are no restrictions on the combination of bits that can be specified by the application. However, applications should use reasonable combinations in order for the implementation to be able to select the most appropriate mode of operation for the particular content type. |
// Provided by VK_KHR_video_encode_queue
typedef VkFlags VkVideoEncodeContentFlagsKHR;
VkVideoEncodeContentFlagsKHR is a bitmask type for setting a mask of
zero or more VkVideoEncodeContentFlagBitsKHR.
Possible video encode tuning mode values are as follows:
// Provided by VK_KHR_video_encode_queue
typedef enum VkVideoEncodeTuningModeKHR {
VK_VIDEO_ENCODE_TUNING_MODE_DEFAULT_KHR = 0,
VK_VIDEO_ENCODE_TUNING_MODE_HIGH_QUALITY_KHR = 1,
VK_VIDEO_ENCODE_TUNING_MODE_LOW_LATENCY_KHR = 2,
VK_VIDEO_ENCODE_TUNING_MODE_ULTRA_LOW_LATENCY_KHR = 3,
VK_VIDEO_ENCODE_TUNING_MODE_LOSSLESS_KHR = 4,
} VkVideoEncodeTuningModeKHR;
-
VK_VIDEO_ENCODE_TUNING_MODE_DEFAULT_KHRspecifies the default tuning mode. -
VK_VIDEO_ENCODE_TUNING_MODE_HIGH_QUALITY_KHRspecifies that video encoding is tuned for high quality. When using this tuning mode, the implementation may compromise the latency of video encoding operations to improve quality. -
VK_VIDEO_ENCODE_TUNING_MODE_LOW_LATENCY_KHRspecifies that video encoding is tuned for low latency. When using this tuning mode, the implementation may compromise quality to increase the performance and lower the latency of video encode operations. -
VK_VIDEO_ENCODE_TUNING_MODE_ULTRA_LOW_LATENCY_KHRspecifies that video encoding is tuned for ultra-low latency. When using this tuning mode, the implementation may compromise quality to maximize the performance and minimize the latency of video encoding operations. -
VK_VIDEO_ENCODE_TUNING_MODE_LOSSLESS_KHRspecifies that video encoding is tuned for lossless encoding. When using this tuning mode, video encode operations produce lossless output.
The VkVideoProfileListInfoKHR structure is defined as:
// Provided by VK_KHR_video_queue
typedef struct VkVideoProfileListInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t profileCount;
const VkVideoProfileInfoKHR* pProfiles;
} VkVideoProfileListInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
profileCountis the number of elements in thepProfilesarray. -
pProfilesis a pointer to an array of VkVideoProfileInfoKHR structures.
|
Note
Video transcoding is an example of a use case that necessitates the specification of multiple profiles in various contexts. |
When the application provides a video decode profile and one or more video encode profiles in the profile list, the implementation ensures that any capabilitities returned or resources created are suitable for the video transcoding use cases without the need for manual data transformations.
42.4. Video Capabilities
42.4.1. Video Coding Capabilities
To query video coding capabilities for a specific video profile, call:
// Provided by VK_KHR_video_queue
VkResult vkGetPhysicalDeviceVideoCapabilitiesKHR(
VkPhysicalDevice physicalDevice,
const VkVideoProfileInfoKHR* pVideoProfile,
VkVideoCapabilitiesKHR* pCapabilities);
-
physicalDeviceis the physical device from which to query the video decode or encode capabilities. -
pVideoProfileis a pointer to a VkVideoProfileInfoKHR structure. -
pCapabilitiesis a pointer to a VkVideoCapabilitiesKHR structure in which the capabilities are returned.
If the video profile described by pVideoProfile is
supported by the implementation, then this command returns VK_SUCCESS
and pCapabilities is filled with the capabilities supported with the
specified video profile.
Otherwise, one of the video-profile-specific
error codes are returned.
The VkVideoCapabilitiesKHR structure is defined as:
// Provided by VK_KHR_video_queue
typedef struct VkVideoCapabilitiesKHR {
VkStructureType sType;
void* pNext;
VkVideoCapabilityFlagsKHR flags;
VkDeviceSize minBitstreamBufferOffsetAlignment;
VkDeviceSize minBitstreamBufferSizeAlignment;
VkExtent2D pictureAccessGranularity;
VkExtent2D minCodedExtent;
VkExtent2D maxCodedExtent;
uint32_t maxDpbSlots;
uint32_t maxActiveReferencePictures;
VkExtensionProperties stdHeaderVersion;
} VkVideoCapabilitiesKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
flagsis a bitmask of VkVideoCapabilityFlagBitsKHR specifying capability flags. -
minBitstreamBufferOffsetAlignmentis the minimum alignment for bitstream buffer offsets. -
minBitstreamBufferSizeAlignmentis the minimum alignment for bitstream buffer range sizes. -
pictureAccessGranularityis the granularity at which image access to video picture resources happen. -
minCodedExtentis the minimum width and height of the coded frames. -
maxCodedExtentis the maximum width and height of the coded frames. -
maxDpbSlotsis the maximum number of DPB slots supported by a single video session. -
maxActiveReferencePicturesis the maximum number of active reference pictures a single video coding operation can use. -
stdHeaderVersionis a VkExtensionProperties structure reporting the Video Std header name and version supported for the video profile.
|
Note
It is common for video compression standards to allow using all reference
pictures associated with active DPB slots as active reference pictures,
hence for video decode profiles the values returned in |
Bits which can be set in VkVideoCapabilitiesKHR::flags are:
// Provided by VK_KHR_video_queue
typedef enum VkVideoCapabilityFlagBitsKHR {
VK_VIDEO_CAPABILITY_PROTECTED_CONTENT_BIT_KHR = 0x00000001,
VK_VIDEO_CAPABILITY_SEPARATE_REFERENCE_IMAGES_BIT_KHR = 0x00000002,
} VkVideoCapabilityFlagBitsKHR;
-
VK_VIDEO_CAPABILITY_PROTECTED_CONTENT_BIT_KHRindicates that video sessions support producing and consuming protected content. -
VK_VIDEO_CAPABILITY_SEPARATE_REFERENCE_IMAGES_BIT_KHRindicates that the video picture resources associated with the DPB slots of a video session can be backed by separateVkImageobjects. If this capability flag is not present, then all DPB slots of a video session must be associated with video picture resources backed by the sameVkImageobject (e.g. using different layers of the same image).
// Provided by VK_KHR_video_queue
typedef VkFlags VkVideoCapabilityFlagsKHR;
VkVideoCapabilityFlagsKHR is a bitmask type for setting a mask of zero
or more VkVideoCapabilityFlagBitsKHR.
42.4.2. Video Format Capabilities
To enumerate the supported output, input and DPB image formats and corresponding capabilities for a specific video profile, call:
// Provided by VK_KHR_video_queue
VkResult vkGetPhysicalDeviceVideoFormatPropertiesKHR(
VkPhysicalDevice physicalDevice,
const VkPhysicalDeviceVideoFormatInfoKHR* pVideoFormatInfo,
uint32_t* pVideoFormatPropertyCount,
VkVideoFormatPropertiesKHR* pVideoFormatProperties);
-
physicalDeviceis the physical device from which to query the video format properties. -
pVideoFormatInfois a pointer to a VkPhysicalDeviceVideoFormatInfoKHR structure specifying the usage and video profiles for which supported image formats and capabilities are returned. -
pVideoFormatPropertyCountis a pointer to an integer related to the number of video format properties available or queried, as described below. -
pVideoFormatPropertiesis a pointer to an array of VkVideoFormatPropertiesKHR structures in which supported image formats and capabilities are returned.
If pVideoFormatProperties is NULL, then the number of video format
properties supported for the given physicalDevice is returned in
pVideoFormatPropertyCount.
Otherwise, pVideoFormatPropertyCount must point to a variable set by
the user to the number of elements in the pVideoFormatProperties
array, and on return the variable is overwritten with the number of values
actually written to pVideoFormatProperties.
If the value of pVideoFormatPropertyCount is less than the number of
video format properties supported, at most pVideoFormatPropertyCount
values will be written to pVideoFormatProperties, and
VK_INCOMPLETE will be returned instead of VK_SUCCESS, to
indicate that not all the available values were returned.
Video format properties are always queried with respect to a specific set of
video profiles.
These are specified by chaining the VkVideoProfileListInfoKHR
structure to pVideoFormatInfo.
For most use cases, the images are used by a single video session and a single video profile is provided. For a use case such as video transcoding, where a decode session output image can be used as encode input in one or more encode sessions, multiple video profiles corresponding to the video sessions that will share the image must be provided.
If any of the video profiles specified via
VkVideoProfileListInfoKHR::pProfiles are not supported, then
this command returns one of the video-profile-specific error codes.
Furthermore, if VkPhysicalDeviceVideoFormatInfoKHR::imageUsage
includes any image usage flags not supported by the specified video
profiles, then this command returns
VK_ERROR_IMAGE_USAGE_NOT_SUPPORTED_KHR.
This command also returns VK_ERROR_IMAGE_USAGE_NOT_SUPPORTED_KHR if
VkPhysicalDeviceVideoFormatInfoKHR::imageUsage does not include
the appropriate flags as dictated by the decode capability flags returned in
VkVideoDecodeCapabilitiesKHR::flags for any of the profiles
specified in the VkVideoProfileListInfoKHR structure provided in the
pNext chain of pVideoFormatInfo.
If the decode capability flags include
VK_VIDEO_DECODE_CAPABILITY_DPB_AND_OUTPUT_COINCIDE_BIT_KHR but not
VK_VIDEO_DECODE_CAPABILITY_DPB_AND_OUTPUT_DISTINCT_BIT_KHR, then in
order to query video format properties for decode DPB and output usage,
VkPhysicalDeviceVideoFormatInfoKHR::imageUsage must include
both VK_IMAGE_USAGE_VIDEO_DECODE_DPB_BIT_KHR and
VK_IMAGE_USAGE_VIDEO_DECODE_DST_BIT_KHR.
Otherwise, the call will fail with
VK_ERROR_IMAGE_USAGE_NOT_SUPPORTED_KHR.
If the decode capability flags include
VK_VIDEO_DECODE_CAPABILITY_DPB_AND_OUTPUT_DISTINCT_BIT_KHR but not
VK_VIDEO_DECODE_CAPABILITY_DPB_AND_OUTPUT_COINCIDE_BIT_KHR, then in
order to query video format properties for decode DPB usage,
VkPhysicalDeviceVideoFormatInfoKHR::imageUsage must include
VK_IMAGE_USAGE_VIDEO_DECODE_DPB_BIT_KHR, but not
VK_IMAGE_USAGE_VIDEO_DECODE_DST_BIT_KHR.
Otherwise, the call will fail with
VK_ERROR_IMAGE_USAGE_NOT_SUPPORTED_KHR.
Similarly, to query video format properties for decode output usage,
VkPhysicalDeviceVideoFormatInfoKHR::imageUsage must include
VK_IMAGE_USAGE_VIDEO_DECODE_DST_BIT_KHR, but not
VK_IMAGE_USAGE_VIDEO_DECODE_DPB_BIT_KHR.
Otherwise, the call will fail with
VK_ERROR_IMAGE_USAGE_NOT_SUPPORTED_KHR.
The imageUsage member of the VkPhysicalDeviceVideoFormatInfoKHR
structure specifies the expected video usage flags that the returned video
formats must support.
Correspondingly, the imageUsageFlags member of each
VkVideoFormatPropertiesKHR structure returned will contain at least
the same set of image usage flags.
If the implementation supports using video input, output, or DPB images of a
particular format in operations other than video decode/encode then the
imageUsageFlags member of the corresponding
VkVideoFormatPropertiesKHR structure returned will include additional
image usage flags indicating that.
|
Note
For most use cases, only decode or encode related usage flags are going to be specified. For a use case such as transcode, if the image were to be shared between decode and encode session(s), then both decode and encode related usage flags can be set. |
Multiple VkVideoFormatPropertiesKHR entries may be returned with the
same format member with different componentMapping,
imageType, or imageTiling values, as described later.
In addition, a different set of VkVideoFormatPropertiesKHR entries
may be returned depending on the imageUsage member of the
VkPhysicalDeviceVideoFormatInfoKHR structure, even for the same set of
video profiles, for example, based on whether encode input, encode DPB,
decode output, and/or decode DPB usage is requested.
The application can select the parameters returned in the
VkVideoFormatPropertiesKHR entries and use compatible parameters when
creating the input, output, and DPB images.
The implementation will report all image creation and usage flags that are
valid for images used with the requested video profiles but applications
should create images only with those that are necessary for the particular
use case.
Before creating an image, the application can obtain the complete set of
supported image format features by calling
vkGetPhysicalDeviceImageFormatProperties2 using parameters derived
from the members of one of the reported VkVideoFormatPropertiesKHR
entries and adding the same VkVideoProfileListInfoKHR structure to the
pNext chain of VkPhysicalDeviceImageFormatInfo2.
The following applies to all VkVideoFormatPropertiesKHR entries
returned by vkGetPhysicalDeviceVideoFormatPropertiesKHR:
-
vkGetPhysicalDeviceFormatProperties2 must succeed when called with
VkVideoFormatPropertiesKHR::format -
If
VkVideoFormatPropertiesKHR::imageTilingisVK_IMAGE_TILING_OPTIMAL, then theoptimalTilingFeaturesreturned by vkGetPhysicalDeviceFormatProperties2 must include all format features required by the image usage flags reported inVkVideoFormatPropertiesKHR::imageUsageFlagsfor the format, as indicated in the Format Feature Dependent Usage Flags section. -
If
VkVideoFormatPropertiesKHR::imageTilingisVK_IMAGE_TILING_LINEAR, then thelinearTilingFeaturesreturned by vkGetPhysicalDeviceFormatProperties2 must include all format features required by the image usage flags reported inVkVideoFormatPropertiesKHR::imageUsageFlagsfor the format, as indicated in the Format Feature Dependent Usage Flags section. -
vkGetPhysicalDeviceImageFormatProperties2 must succeed when called with a VkPhysicalDeviceImageFormatInfo2 structure containing the following information:
-
The
pNextchain including the same VkVideoProfileListInfoKHR structure used to callvkGetPhysicalDeviceVideoFormatPropertiesKHR. -
formatset to the value ofVkVideoFormatPropertiesKHR::format. -
typeset to the value ofVkVideoFormatPropertiesKHR::imageType. -
tilingset to the value ofVkVideoFormatPropertiesKHR::imageTiling. -
usageset to the value ofVkVideoFormatPropertiesKHR::imageUsageFlags. -
flagsset to the value ofVkVideoFormatPropertiesKHR::imageCreateFlags.
-
The componentMapping member of VkVideoFormatPropertiesKHR
defines the ordering of the Y′CBCR color channels from the perspective of
the video codec operations specified in VkVideoProfileListInfoKHR.
For example, if the implementation produces video decode output with the
format VK_FORMAT_G8_B8R8_2PLANE_420_UNORM where the blue and red
chrominance channels are swapped then the componentMapping member of
the corresponding VkVideoFormatPropertiesKHR structure will have the
following member values:
components.r = VK_COMPONENT_SWIZZLE_B; // Cb component
components.g = VK_COMPONENT_SWIZZLE_IDENTITY; // Y component
components.b = VK_COMPONENT_SWIZZLE_R; // Cr component
components.a = VK_COMPONENT_SWIZZLE_IDENTITY; // unused, defaults to 1.0
The VkPhysicalDeviceVideoFormatInfoKHR structure is defined as:
// Provided by VK_KHR_video_queue
typedef struct VkPhysicalDeviceVideoFormatInfoKHR {
VkStructureType sType;
const void* pNext;
VkImageUsageFlags imageUsage;
} VkPhysicalDeviceVideoFormatInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
imageUsageis a bitmask of VkImageUsageFlagBits specifying the intended usage of the video images.
The VkVideoFormatPropertiesKHR structure is defined as:
// Provided by VK_KHR_video_queue
typedef struct VkVideoFormatPropertiesKHR {
VkStructureType sType;
void* pNext;
VkFormat format;
VkComponentMapping componentMapping;
VkImageCreateFlags imageCreateFlags;
VkImageType imageType;
VkImageTiling imageTiling;
VkImageUsageFlags imageUsageFlags;
} VkVideoFormatPropertiesKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
formatis a VkFormat that specifies the format that can be used with the specified video profiles and image usages. -
componentMappingdefines the color channel order used for the format.formatalong withcomponentMappingdescribe how the color channels are ordered when producing video decoder output or are expected to be ordered in video encoder input, when applicable. If theformatreported does not require component swizzling then all members ofcomponentMappingwill be set toVK_COMPONENT_SWIZZLE_IDENTITY. -
imageCreateFlagsis a bitmask of VkImageCreateFlagBits specifying the supported image creation flags for the format. -
imageTypeis a VkImageType that specifies the image type the format can be used with. -
imageTilingis a VkImageTiling that specifies the image tiling the format can be used with. -
imageUsageFlagsis a bitmask of VkImageUsageFlagBits specifying the supported image usage flags for the format.
42.5. Video Sessions
Video sessions are objects that represent and maintain the state needed to perform video decode or encode operations using a specific video profile.
In case of video encode profiles this includes the current rate control configuration and the currently set video encode quality level.
Video sessions are represented by VkVideoSessionKHR handles:
// Provided by VK_KHR_video_queue
VK_DEFINE_NON_DISPATCHABLE_HANDLE(VkVideoSessionKHR)
42.5.1. Creating a Video Session
To create a video session object, call:
// Provided by VK_KHR_video_queue
VkResult vkCreateVideoSessionKHR(
VkDevice device,
const VkVideoSessionCreateInfoKHR* pCreateInfo,
const VkAllocationCallbacks* pAllocator,
VkVideoSessionKHR* pVideoSession);
-
deviceis the logical device that creates the video session. -
pCreateInfois a pointer to a VkVideoSessionCreateInfoKHR structure containing parameters to be used to create the video session. -
pAllocatorcontrols host memory allocation as described in the Memory Allocation chapter. -
pVideoSessionis a pointer to a VkVideoSessionKHR handle in which the resulting video session object is returned.
The resulting video session object is said to be created with the video
codec operation specified in
pCreateInfo->pVideoProfile->videoCodecOperation.
The name and version of the codec-specific Video Std header to be used with
the video session is specified by the VkExtensionProperties structure
pointed to by pCreateInfo->pStdHeaderVersion.
If a non-existent or unsupported Video Std header version is specified in
pCreateInfo->pStdHeaderVersion->specVersion, then this command returns
VK_ERROR_VIDEO_STD_VERSION_NOT_SUPPORTED_KHR.
Video session objects are created in uninitialized state.
In order to transition the video session into initial state, the
application must issue a vkCmdControlVideoCodingKHR command with
VkVideoCodingControlInfoKHR::flags including
VK_VIDEO_CODING_CONTROL_RESET_BIT_KHR.
Video session objects also maintain the
state of the DPB.
The number of DPB slots usable with the created video session is specified
in pCreateInfo->maxDpbSlots, and each slot is initially in the
inactive state.
Each DPB slot maintained by the created video session can refer to a reference picture representing a video frame.
In addition, if the videoCodecOperation member of the
VkVideoProfileInfoKHR structure pointed to by
pCreateInfo->pVideoProfile is
VK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR and the
pictureLayout member of the VkVideoDecodeH264ProfileInfoKHR
structure provided in the VkVideoProfileInfoKHR::pNext chain is
not VK_VIDEO_DECODE_H264_PICTURE_LAYOUT_PROGRESSIVE_KHR, then the
created video session supports interlaced frames and each DPB
slot maintained by the created video session can instead refer to
separate top field and bottom field reference pictures
that together can represent a full video frame.
In this case, it is up to the application, driven by the video content,
whether it associates any individual DPB slot with separate top and/or
bottom field pictures or a single picture representing a full frame.
The created video session can be used to perform video coding operations
using video frames up to the maximum size specified in
pCreateInfo->maxCodedExtent.
The minimum frame size allowed is implicitly derived from
VkVideoCapabilitiesKHR::minCodedExtent, as returned by
vkGetPhysicalDeviceVideoCapabilitiesKHR for the video profile
specified by pCreateInfo->pVideoProfile.
Accordingly, the created video session is said to be created with a
minCodedExtent equal to that.
In case of video session objects created with a video encode operation,
implementations may return the
VK_ERROR_INVALID_VIDEO_STD_PARAMETERS_KHR error if any of the
specified Video Std parameters do not adhere to the syntactic or semantic
requirements of the used video compression standard, or if values derived
from parameters according to the rules defined by the used video compression
standard do not adhere to the capabilities of the video compression standard
or the implementation.
|
Note
Applications should not rely on the
|
The VkVideoSessionCreateInfoKHR structure is defined as:
// Provided by VK_KHR_video_queue
typedef struct VkVideoSessionCreateInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t queueFamilyIndex;
VkVideoSessionCreateFlagsKHR flags;
const VkVideoProfileInfoKHR* pVideoProfile;
VkFormat pictureFormat;
VkExtent2D maxCodedExtent;
VkFormat referencePictureFormat;
uint32_t maxDpbSlots;
uint32_t maxActiveReferencePictures;
const VkExtensionProperties* pStdHeaderVersion;
} VkVideoSessionCreateInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
queueFamilyIndexis the index of the queue family the created video session will be used with. -
flagsis a bitmask of VkVideoSessionCreateFlagBitsKHR specifying creation flags. -
pVideoProfileis a pointer to a VkVideoProfileInfoKHR structure specifying the video profile the created video session will be used with. -
pictureFormatis the image format the created video session will be used with. IfpVideoProfile->videoCodecOperationspecifies a decode operation, thenpictureFormatis the image format of decode output pictures usable with the created video session. IfpVideoProfile->videoCodecOperationspecifies an encode operation, thenpictureFormatis the image format of encode input pictures usable with the created video session. -
maxCodedExtentis the maximum width and height of the coded frames the created video session will be used with. -
referencePictureFormatis the image format of reference pictures stored in the DPB the created video session will be used with. -
maxDpbSlotsis the maximum number of DPB Slots that can be used with the created video session. -
maxActiveReferencePicturesis the maximum number of active reference pictures that can be used in a single video coding operation using the created video session. -
pStdHeaderVersionis a pointer to a VkExtensionProperties structure requesting the Video Std header version to use for thevideoCodecOperationspecified inpVideoProfile.
Bits which can be set in VkVideoSessionCreateInfoKHR::flags
are:
// Provided by VK_KHR_video_queue
typedef enum VkVideoSessionCreateFlagBitsKHR {
VK_VIDEO_SESSION_CREATE_PROTECTED_CONTENT_BIT_KHR = 0x00000001,
// Provided by VK_KHR_video_encode_queue
VK_VIDEO_SESSION_CREATE_ALLOW_ENCODE_PARAMETER_OPTIMIZATIONS_BIT_KHR = 0x00000002,
// Provided by VK_KHR_video_maintenance1
VK_VIDEO_SESSION_CREATE_INLINE_QUERIES_BIT_KHR = 0x00000004,
} VkVideoSessionCreateFlagBitsKHR;
-
VK_VIDEO_SESSION_CREATE_PROTECTED_CONTENT_BIT_KHRspecifies that the video session uses protected video content. -
VK_VIDEO_SESSION_CREATE_ALLOW_ENCODE_PARAMETER_OPTIMIZATIONS_BIT_KHRspecifies that the implementation is allowed to override video session parameters and other codec-specific encoding parameters to optimize video encode operations based on the use case information specified in the video profile and the used video encode quality level.NoteNot specifying
VK_VIDEO_SESSION_CREATE_ALLOW_ENCODE_PARAMETER_OPTIMIZATIONS_BIT_KHRdoes not guarantee that the implementation will not do any codec-specific parameter overrides, as certain overrides are necessary for the correct operation of the video encoder implementation due to limitations to the available encoding tools on that implementation. This flag, however, enables the implementation to apply further optimizing overrides. -
VK_VIDEO_SESSION_CREATE_INLINE_QUERIES_BIT_KHRspecifies that queries within video coding scopes using the created video session are executed inline with video coding operations.
// Provided by VK_KHR_video_queue
typedef VkFlags VkVideoSessionCreateFlagsKHR;
VkVideoSessionCreateFlagsKHR is a bitmask type for setting a mask of
zero or more VkVideoSessionCreateFlagBitsKHR.
42.5.2. Destroying a Video Session
To destroy a video session, call:
// Provided by VK_KHR_video_queue
void vkDestroyVideoSessionKHR(
VkDevice device,
VkVideoSessionKHR videoSession,
const VkAllocationCallbacks* pAllocator);
-
deviceis the logical device that destroys the video session. -
videoSessionis the video session to destroy. -
pAllocatorcontrols host memory allocation as described in the Memory Allocation chapter.
42.5.3. Video Session Memory Association
After creating a video session object, and before the object can be used to record video coding operations into command buffers using it, the application must allocate and bind device memory to the video session. Device memory is allocated separately (see Device Memory) and then associated with the video session.
Video sessions may have multiple memory bindings identified by unique unsigned integer values. Appropriate device memory must be bound to each such memory binding before using the video session to record command buffer commands with it.
To determine the memory requirements for a video session object, call:
// Provided by VK_KHR_video_queue
VkResult vkGetVideoSessionMemoryRequirementsKHR(
VkDevice device,
VkVideoSessionKHR videoSession,
uint32_t* pMemoryRequirementsCount,
VkVideoSessionMemoryRequirementsKHR* pMemoryRequirements);
-
deviceis the logical device that owns the video session. -
videoSessionis the video session to query. -
pMemoryRequirementsCountis a pointer to an integer related to the number of memory binding requirements available or queried, as described below. -
pMemoryRequirementsisNULLor a pointer to an array of VkVideoSessionMemoryRequirementsKHR structures in which the memory binding requirements of the video session are returned.
If pMemoryRequirements is NULL, then the number of memory bindings
required for the video session is returned in
pMemoryRequirementsCount.
Otherwise, pMemoryRequirementsCount must point to a variable set by
the user with the number of elements in the pMemoryRequirements array,
and on return the variable is overwritten with the number of memory binding
requirements actually written to pMemoryRequirements.
If pMemoryRequirementsCount is less than the number of memory bindings
required for the video session, then at most pMemoryRequirementsCount
elements will be written to pMemoryRequirements, and
VK_INCOMPLETE will be returned, instead of VK_SUCCESS, to
indicate that not all required memory binding requirements were returned.
The VkVideoSessionMemoryRequirementsKHR structure is defined as:
// Provided by VK_KHR_video_queue
typedef struct VkVideoSessionMemoryRequirementsKHR {
VkStructureType sType;
void* pNext;
uint32_t memoryBindIndex;
VkMemoryRequirements memoryRequirements;
} VkVideoSessionMemoryRequirementsKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
memoryBindIndexis the index of the memory binding. -
memoryRequirementsis a VkMemoryRequirements structure in which the requested memory binding requirements for the binding index specified bymemoryBindIndexare returned.
To attach memory to a video session object, call:
// Provided by VK_KHR_video_queue
VkResult vkBindVideoSessionMemoryKHR(
VkDevice device,
VkVideoSessionKHR videoSession,
uint32_t bindSessionMemoryInfoCount,
const VkBindVideoSessionMemoryInfoKHR* pBindSessionMemoryInfos);
-
deviceis the logical device that owns the video session. -
videoSessionis the video session to be bound with device memory. -
bindSessionMemoryInfoCountis the number of elements inpBindSessionMemoryInfos. -
pBindSessionMemoryInfosis a pointer to an array ofbindSessionMemoryInfoCountVkBindVideoSessionMemoryInfoKHR structures specifying memory regions to be bound to specific memory bindings of the video session.
The valid usage statements below refer to the VkMemoryRequirements
structure corresponding to a specific element of
pBindSessionMemoryInfos, which is defined as follows:
-
If the
memoryBindIndexmember of the element ofpBindSessionMemoryInfosin question matches thememoryBindIndexmember of one of the elements returned inpMemoryRequirementswhen vkGetVideoSessionMemoryRequirementsKHR is called with the samevideoSessionand withpMemoryRequirementsCountequal tobindSessionMemoryInfoCount, then thememoryRequirementsmember of that element ofpMemoryRequirementsis the VkMemoryRequirements structure corresponding to the element ofpBindSessionMemoryInfosin question. -
Otherwise the element of
pBindSessionMemoryInfosin question is said to not have a corresponding VkMemoryRequirements structure.
The VkBindVideoSessionMemoryInfoKHR structure is defined as:
// Provided by VK_KHR_video_queue
typedef struct VkBindVideoSessionMemoryInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t memoryBindIndex;
VkDeviceMemory memory;
VkDeviceSize memoryOffset;
VkDeviceSize memorySize;
} VkBindVideoSessionMemoryInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
memoryBindIndexis the memory binding index to bind memory to. -
memoryis the allocated device memory to be bound to the video session’s memory binding with indexmemoryBindIndex. -
memoryOffsetis the start offset of the region ofmemorywhich is to be bound. -
memorySizeis the size in bytes of the region ofmemory, starting frommemoryOffsetbytes, to be bound.
42.6. Video Profile Compatibility
Resources and query pools used with a particular video session must be compatible with the video profile the video session was created with.
A VkBuffer is compatible with a video profile if it was created with
the VkBufferCreateInfo::pNext chain including a
VkVideoProfileListInfoKHR structure with its pProfiles member
containing an element matching the VkVideoProfileInfoKHR structure
chain describing the video profile, and
VkBufferCreateInfo::usage including at least one bit specific to
video coding usage.
-
VK_BUFFER_USAGE_VIDEO_DECODE_SRC_BIT_KHR -
VK_BUFFER_USAGE_VIDEO_DECODE_DST_BIT_KHR -
VK_BUFFER_USAGE_VIDEO_ENCODE_SRC_BIT_KHR -
VK_BUFFER_USAGE_VIDEO_ENCODE_DST_BIT_KHR
A VkBuffer is also compatible with a video profile if it was created
with VkBufferCreateInfo::flags including
VK_BUFFER_CREATE_VIDEO_PROFILE_INDEPENDENT_BIT_KHR.
A VkImage is compatible with a video profile if it was created with
the VkImageCreateInfo::pNext chain including a
VkVideoProfileListInfoKHR structure with its pProfiles member
containing an element matching the VkVideoProfileInfoKHR structure
chain describing the video profile, and VkImageCreateInfo::usage
including at least one bit specific to video coding usage.
-
VK_IMAGE_USAGE_VIDEO_DECODE_SRC_BIT_KHR -
VK_IMAGE_USAGE_VIDEO_DECODE_DST_BIT_KHR -
VK_IMAGE_USAGE_VIDEO_DECODE_DPB_BIT_KHR -
VK_IMAGE_USAGE_VIDEO_ENCODE_SRC_BIT_KHR -
VK_IMAGE_USAGE_VIDEO_ENCODE_DST_BIT_KHR -
VK_IMAGE_USAGE_VIDEO_ENCODE_DPB_BIT_KHR
A VkImage is also compatible with a video profile if all of the following conditions are true for the VkImageCreateInfo structure the image was created with:
-
VkImageCreateInfo::
flagsincludedVK_IMAGE_CREATE_VIDEO_PROFILE_INDEPENDENT_BIT_KHR. -
The list of VkVideoFormatPropertiesKHR structures, obtained by calling vkGetPhysicalDeviceVideoFormatPropertiesKHR with VkPhysicalDeviceVideoFormatInfoKHR::
imageUsageequal to the VkImageCreateInfo::usagethe image was created with and the VkPhysicalDeviceVideoFormatInfoKHR::pNextchain including a VkVideoProfileListInfoKHR structure with itspProfilesmember containing a single array element specifying the VkVideoProfileInfoKHR structure chain describing the video profile in question, contains an element for which all of the following conditions are true with respect to the VkImageCreateInfo structure the image was created with:-
VkImageCreateInfo::
formatequals VkVideoFormatPropertiesKHR::format. -
VkImageCreateInfo::
flagsonly contains bits also set in VkVideoFormatPropertiesKHR::imageCreateFlags. -
VkImageCreateInfo::
imageTypeequals VkVideoFormatPropertiesKHR::imageType. -
VkImageCreateInfo::
tilingequals VkVideoFormatPropertiesKHR::imageTiling. -
VkImageCreateInfo::
usageonly contains bits also set in VkVideoFormatPropertiesKHR::imageUsageFlags.
-
|
Note
While some of these rules allow creating buffer or image resources that may
be compatible with any video profile, applications should still prefer to
include the specific video profiles the buffer or image resource is expected
to be used with (through a VkVideoProfileListInfoKHR structure
included in the |
A VkImageView is compatible with a video profile if the VkImage it was created from is also compatible with that video profile.
A VkQueryPool is compatible with a video profile if it was created
with the VkQueryPoolCreateInfo::pNext chain including a
VkVideoProfileInfoKHR structure chain describing the same video
profile, and VkQueryPoolCreateInfo::queryType having one of the
following values:
-
VK_QUERY_TYPE_RESULT_STATUS_ONLY_KHR -
VK_QUERY_TYPE_VIDEO_ENCODE_FEEDBACK_KHR
42.7. Video Session Parameters
Video session parameters objects can store preprocessed codec-specific parameters used with a compatible video session, and enable reducing the number of parameters needed to be provided and processed by the implementation while recording video coding operations into command buffers.
Parameters stored in such objects are immutable to facilitate the concurrent use of the stored parameters in multiple threads. At the same time, new parameters can be added to existing objects using the vkUpdateVideoSessionParametersKHR command.
In order to support concurrent use of the stored immutable parameters while
also allowing the video session parameters object to be extended with new
parameters, each video session parameters object maintains an update
sequence counter that is set to 0 at object creation time and must be
incremented by each subsequent update operation.
Certain video sequences that adhere to particular video compression standards permit updating previously supplied parameters. If a parameter update is necessary, the application has the following options:
-
Cache the set of parameters on the application side and create a new video session parameters object adding all the parameters with appropriate changes, as necessary; or
-
Create a new video session parameters object providing only the updated parameters and the previously used object as the template, which ensures that parameters not specified at creation time will be copied unmodified from the template object.
The actual types of parameters that can be stored and the capacity for individual parameter types, and the methods of initializing, updating, and referring to individual parameters are specific to the video codec operation the video session parameters object was created with.
-
For
VK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHRthese are defined in the H.264 Decode Parameter Sets section. -
For
VK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHRthese are defined in the H.265 Decode Parameter Sets section. -
For
VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHRthese are defined in the H.264 Encode Parameter Sets section. -
For
VK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHRthese are defined in the H.265 Encode Parameter Sets section.
Video session parameters objects created with an encode operation are further specialized based on the video encode quality level the video session parameters are used with, as implementations may apply different sets of parameter overrides depending on the used quality level. This enables implementations to store the potentially optimized set of parameters in these objects, further limiting the necessary processing required while recording video encode operations into command buffers.
Video session parameters are represented by
VkVideoSessionParametersKHR handles:
// Provided by VK_KHR_video_queue
VK_DEFINE_NON_DISPATCHABLE_HANDLE(VkVideoSessionParametersKHR)
42.7.1. Creating Video Session Parameters
To create a video session parameters object, call:
// Provided by VK_KHR_video_queue
VkResult vkCreateVideoSessionParametersKHR(
VkDevice device,
const VkVideoSessionParametersCreateInfoKHR* pCreateInfo,
const VkAllocationCallbacks* pAllocator,
VkVideoSessionParametersKHR* pVideoSessionParameters);
-
deviceis the logical device that creates the video session parameters object. -
pCreateInfois a pointer to VkVideoSessionParametersCreateInfoKHR structure containing parameters to be used to create the video session parameters object. -
pAllocatorcontrols host memory allocation as described in the Memory Allocation chapter. -
pVideoSessionParametersis a pointer to a VkVideoSessionParametersKHR handle in which the resulting video session parameters object is returned.
The resulting video session parameters object is said to be created with the
video codec operation pCreateInfo->videoSession was created with.
Video session parameters objects created with an encode operation are always
created with respect to a video encode quality
level.
By default, the created video session parameters objects are created with
quality level zero, unless otherwise specified by including a
VkVideoEncodeQualityLevelInfoKHR structure in the
pCreateInfo->pNext chain, in which case the video session parameters
object is created with the quality level specified in
VkVideoEncodeQualityLevelInfoKHR::qualityLevel.
If pCreateInfo->videoSessionParametersTemplate is not
VK_NULL_HANDLE, then it will be used as a template for constructing
the new video session parameters object.
This happens by first adding any parameters according to the additional
creation parameters provided in the pCreateInfo->pNext chain, followed
by adding any parameters from the template object that have a key that does
not match the key of any of the already added parameters.
For video session parameters objects created with an encode operation, the
template object specified in
pCreateInfo->videoSessionParametersTemplate must have been created
with the same video encode quality level as the
newly created object.
|
Note
This means that codec-specific parameters stored in video session parameters objects can only be reused across different video encode quality levels by re-specifying them, as previously created video session parameters against other quality levels cannot be used as template because the original codec-specific parameters (before the implementation may have applied parameter overrides) may no longer be available in them for the purposes of constructing the derived object. |
If pCreateInfo->videoSession was created with the video codec
operation VK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR, then the
created video session parameters object will initially contain the following
sets of parameter entries:
-
StdVideoH264SequenceParameterSetstructures representing H.264 SPS entries, as follows:-
If the
pParametersAddInfomember of the VkVideoDecodeH264SessionParametersCreateInfoKHR structure provided in thepCreateInfo->pNextchain is notNULL, then the set ofStdVideoH264SequenceParameterSetentries specified inpParametersAddInfo->pStdSPSsare added first; -
If
pCreateInfo->videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH264SequenceParameterSetentry stored in it is copied to the created video session parameters object if the created object does not already contain such an entry with the sameseq_parameter_set_id.
-
-
StdVideoH264PictureParameterSetstructures representing H.264 PPS entries, as follows:-
If the
pParametersAddInfomember of the VkVideoDecodeH264SessionParametersCreateInfoKHR structure provided in thepCreateInfo->pNextchain is notNULL, then the set ofStdVideoH264PictureParameterSetentries specified inpParametersAddInfo->pStdPPSsare added first; -
If
pCreateInfo->videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH264PictureParameterSetentry stored in it is copied to the created video session parameters object if the created object does not already contain such an entry with the sameseq_parameter_set_idandpic_parameter_set_id.
-
If pCreateInfo->videoSession was created with the video codec
operation VK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR, then the
created video session parameters object will initially contain the following
sets of parameter entries:
-
StdVideoH265VideoParameterSetstructures representing H.265 VPS entries, as follows:-
If the
pParametersAddInfomember of the VkVideoDecodeH265SessionParametersCreateInfoKHR structure provided in thepCreateInfo->pNextchain is notNULL, then the set ofStdVideoH265VideoParameterSetentries specified inpParametersAddInfo->pStdVPSsare added first; -
If
pCreateInfo->videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH265VideoParameterSetentry stored in it is copied to the created video session parameters object if the created object does not already contain such an entry with the samevps_video_parameter_set_id.
-
-
StdVideoH265SequenceParameterSetstructures representing H.265 SPS entries, as follows:-
If the
pParametersAddInfomember of the VkVideoDecodeH265SessionParametersCreateInfoKHR structure provided in thepCreateInfo->pNextchain is notNULL, then the set ofStdVideoH265SequenceParameterSetentries specified inpParametersAddInfo->pStdSPSsare added first; -
If
pCreateInfo->videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH265SequenceParameterSetentry stored in it is copied to the created video session parameters object if the created object does not already contain such an entry with the samesps_video_parameter_set_idandsps_seq_parameter_set_id.
-
-
StdVideoH265PictureParameterSetstructures representing H.265 PPS entries, as follows:-
If the
pParametersAddInfomember of the VkVideoDecodeH265SessionParametersCreateInfoKHR structure provided in thepCreateInfo->pNextchain is notNULL, then the set ofStdVideoH265PictureParameterSetentries specified inpParametersAddInfo->pStdPPSsare added first; -
If
pCreateInfo->videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH265PictureParameterSetentry stored in it is copied to the created video session parameters object if the created object does not already contain such an entry with the samesps_video_parameter_set_id,pps_seq_parameter_set_id, andpps_pic_parameter_set_id.
-
If pCreateInfo->videoSession was created with the video codec
operation VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR, then the
created video session parameters object will initially contain the following
sets of parameter entries:
-
StdVideoH264SequenceParameterSetstructures representing H.264 SPS entries, as follows:-
If the
pParametersAddInfomember of the VkVideoEncodeH264SessionParametersCreateInfoKHR structure provided in thepCreateInfo->pNextchain is notNULL, then the set ofStdVideoH264SequenceParameterSetentries specified inpParametersAddInfo->pStdSPSsare added first; -
If
pCreateInfo->videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH264SequenceParameterSetentry stored in it is copied to the created video session parameters object if the created object does not already contain such an entry with the sameseq_parameter_set_id.
-
-
StdVideoH264PictureParameterSetstructures representing H.264 PPS entries, as follows:-
If the
pParametersAddInfomember of the VkVideoEncodeH264SessionParametersCreateInfoKHR structure provided in thepCreateInfo->pNextchain is notNULL, then the set ofStdVideoH264PictureParameterSetentries specified inpParametersAddInfo->pStdPPSsare added first; -
If
pCreateInfo->videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH264PictureParameterSetentry stored in it is copied to the created video session parameters object if the created object does not already contain such an entry with the sameseq_parameter_set_idandpic_parameter_set_id.
-
If pCreateInfo->videoSession was created with the video codec
operation VK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR, then the
created video session parameters object will initially contain the following
sets of parameter entries:
-
StdVideoH265VideoParameterSetstructures representing H.265 VPS entries, as follows:-
If the
pParametersAddInfomember of the VkVideoEncodeH265SessionParametersCreateInfoKHR structure provided in thepCreateInfo->pNextchain is notNULL, then the set ofStdVideoH265VideoParameterSetentries specified inpParametersAddInfo->pStdVPSsare added first; -
If
pCreateInfo->videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH265VideoParameterSetentry stored in it is copied to the created video session parameters object if the created object does not already contain such an entry with the samevps_video_parameter_set_id.
-
-
StdVideoH265SequenceParameterSetstructures representing H.265 SPS entries, as follows:-
If the
pParametersAddInfomember of the VkVideoEncodeH265SessionParametersCreateInfoKHR structure provided in thepCreateInfo->pNextchain is notNULL, then the set ofStdVideoH265SequenceParameterSetentries specified inpParametersAddInfo->pStdSPSsare added first; -
If
pCreateInfo->videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH265SequenceParameterSetentry stored in it is copied to the created video session parameters object if the created object does not already contain such an entry with the samesps_video_parameter_set_idandsps_seq_parameter_set_id.
-
-
StdVideoH265PictureParameterSetstructures representing H.265 PPS entries, as follows:-
If the
pParametersAddInfomember of the VkVideoEncodeH265SessionParametersCreateInfoKHR structure provided in thepCreateInfo->pNextchain is notNULL, then the set ofStdVideoH265PictureParameterSetentries specified inpParametersAddInfo->pStdPPSsare added first; -
If
pCreateInfo->videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH265PictureParameterSetentry stored in it is copied to the created video session parameters object if the created object does not already contain such an entry with the samesps_video_parameter_set_id,pps_seq_parameter_set_id, andpps_pic_parameter_set_id.
-
In case of video session parameters objects created with a video encode
operation, implementations may return the
VK_ERROR_INVALID_VIDEO_STD_PARAMETERS_KHR error if any of the
specified Video Std parameters do not adhere to the syntactic or semantic
requirements of the used video compression standard, or if values derived
from parameters according to the rules defined by the used video compression
standard do not adhere to the capabilities of the video compression standard
or the implementation.
|
Note
Applications should not rely on the
|
The VkVideoSessionParametersCreateInfoKHR structure is defined as:
// Provided by VK_KHR_video_queue
typedef struct VkVideoSessionParametersCreateInfoKHR {
VkStructureType sType;
const void* pNext;
VkVideoSessionParametersCreateFlagsKHR flags;
VkVideoSessionParametersKHR videoSessionParametersTemplate;
VkVideoSessionKHR videoSession;
} VkVideoSessionParametersCreateInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
flagsis reserved for future use. -
videoSessionParametersTemplateisVK_NULL_HANDLEor a valid handle to a VkVideoSessionParametersKHR object used as a template for constructing the new video session parameters object. -
videoSessionis the video session object against which the video session parameters object is going to be created.
Limiting values are defined below that are referenced by the relevant valid usage statements of this structure.
-
If
videoSessionwas created with the codec operationVK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR, then letStdVideoH264SequenceParameterSet spsAddList[]be the list of H.264 SPS entries to add to the created video session parameters object, defined as follows:-
If the
pParametersAddInfomember of the VkVideoDecodeH264SessionParametersCreateInfoKHR structure provided in thepNextchain is notNULL, then the set ofStdVideoH264SequenceParameterSetentries specified inpParametersAddInfo->pStdSPSsare added tospsAddList; -
If
videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH264SequenceParameterSetentry stored in it withseq_parameter_set_idnot matching any of the entries already inspsAddListis added tospsAddList.
-
-
If
videoSessionwas created with the codec operationVK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR, then letStdVideoH264PictureParameterSet ppsAddList[]be the list of H.264 PPS entries to add to the created video session parameters object, defined as follows:-
If the
pParametersAddInfomember of the VkVideoDecodeH264SessionParametersCreateInfoKHR structure provided in thepNextchain is notNULL, then the set ofStdVideoH264PictureParameterSetentries specified inpParametersAddInfo->pStdPPSsare added toppsAddList; -
If
videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH264PictureParameterSetentry stored in it withseq_parameter_set_idorpic_parameter_set_idnot matching any of the entries already inppsAddListis added toppsAddList.
-
-
If
videoSessionwas created with the codec operationVK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR, then letStdVideoH265VideoParameterSet vpsAddList[]be the list of H.265 VPS entries to add to the created video session parameters object, defined as follows:-
If the
pParametersAddInfomember of the VkVideoDecodeH265SessionParametersCreateInfoKHR structure provided in thepNextchain is notNULL, then the set ofStdVideoH265VideoParameterSetentries specified inpParametersAddInfo->pStdVPSsare added tovpsAddList; -
If
videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH265VideoParameterSetentry stored in it withvps_video_parameter_set_idnot matching any of the entries already invpsAddListis added tovpsAddList.
-
-
If
videoSessionwas created with the codec operationVK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR, then letStdVideoH265SequenceParameterSet spsAddList[]be the list of H.265 SPS entries to add to the created video session parameters object, defined as follows:-
If the
pParametersAddInfomember of the VkVideoDecodeH265SessionParametersCreateInfoKHR structure provided in thepNextchain is notNULL, then the set ofStdVideoH265SequenceParameterSetentries specified inpParametersAddInfo->pStdSPSsare added tospsAddList; -
If
videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH265SequenceParameterSetentry stored in it withsps_video_parameter_set_idorsps_seq_parameter_set_idnot matching any of the entries already inspsAddListis added tospsAddList.
-
-
If
videoSessionwas created with the codec operationVK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR, then letStdVideoH265PictureParameterSet ppsAddList[]be the list of H.265 PPS entries to add to the created video session parameters object, defined as follows:-
If the
pParametersAddInfomember of the VkVideoDecodeH265SessionParametersCreateInfoKHR structure provided in thepNextchain is notNULL, then the set ofStdVideoH265PictureParameterSetentries specified inpParametersAddInfo->pStdPPSsare added toppsAddList; -
If
videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH265PictureParameterSetentry stored in it withsps_video_parameter_set_id,pps_seq_parameter_set_id, orpps_pic_parameter_set_idnot matching any of the entries already inppsAddListis added toppsAddList.
-
-
If
videoSessionwas created with an encode operation, then letuint32_t qualityLevelbe the video encode quality level of the created video session parameters object, defined as follows:-
If the
pNextchain of this structure includes a VkVideoEncodeQualityLevelInfoKHR structure, thenqualityLevelis equal to VkVideoEncodeQualityLevelInfoKHR::qualityLevel. -
Otherwise
qualityLevelis0
-
-
If
videoSessionwas created with the codec operationVK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR, then letStdVideoH264SequenceParameterSet spsAddList[]be the list of H.264 SPS entries to add to the created video session parameters object, defined as follows:-
If the
pParametersAddInfomember of the VkVideoEncodeH264SessionParametersCreateInfoKHR structure provided in thepNextchain is notNULL, then the set ofStdVideoH264SequenceParameterSetentries specified inpParametersAddInfo->pStdSPSsare added tospsAddList; -
If
videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH264SequenceParameterSetentry stored in it withseq_parameter_set_idnot matching any of the entries already inspsAddListis added tospsAddList.
-
-
If
videoSessionwas created with the codec operationVK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR, then letStdVideoH264PictureParameterSet ppsAddList[]be the list of H.264 PPS entries to add to the created video session parameters object, defined as follows:-
If the
pParametersAddInfomember of the VkVideoEncodeH264SessionParametersCreateInfoKHR structure provided in thepNextchain is notNULL, then the set ofStdVideoH264PictureParameterSetentries specified inpParametersAddInfo->pStdPPSsare added toppsAddList; -
If
videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH264PictureParameterSetentry stored in it withseq_parameter_set_idorpic_parameter_set_idnot matching any of the entries already inppsAddListis added toppsAddList.
-
-
If
videoSessionwas created with the codec operationVK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR, then letStdVideoH265VideoParameterSet vpsAddList[]be the list of H.265 VPS entries to add to the created video session parameters object, defined as follows:-
If the
pParametersAddInfomember of the VkVideoEncodeH265SessionParametersCreateInfoKHR structure provided in thepNextchain is notNULL, then the set ofStdVideoH265VideoParameterSetentries specified inpParametersAddInfo->pStdVPSsare added tovpsAddList; -
If
videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH265VideoParameterSetentry stored in it withvps_video_parameter_set_idnot matching any of the entries already invpsAddListis added tovpsAddList.
-
-
If
videoSessionwas created with the codec operationVK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR, then letStdVideoH265SequenceParameterSet spsAddList[]be the list of H.265 SPS entries to add to the created video session parameters object, defined as follows:-
If the
pParametersAddInfomember of the VkVideoEncodeH265SessionParametersCreateInfoKHR structure provided in thepNextchain is notNULL, then the set ofStdVideoH265SequenceParameterSetentries specified inpParametersAddInfo->pStdSPSsare added tospsAddList; -
If
videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH265SequenceParameterSetentry stored in it withsps_video_parameter_set_idorsps_seq_parameter_set_idnot matching any of the entries already inspsAddListis added tospsAddList.
-
-
If
videoSessionwas created with the codec operationVK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR, then letStdVideoH265PictureParameterSet ppsAddList[]be the list of H.265 PPS entries to add to the created video session parameters object, defined as follows:-
If the
pParametersAddInfomember of the VkVideoEncodeH265SessionParametersCreateInfoKHR structure provided in thepNextchain is notNULL, then the set ofStdVideoH265PictureParameterSetentries specified inpParametersAddInfo->pStdPPSsare added toppsAddList; -
If
videoSessionParametersTemplateis notVK_NULL_HANDLE, then eachStdVideoH265PictureParameterSetentry stored in it withsps_video_parameter_set_id,pps_seq_parameter_set_id, orpps_pic_parameter_set_idnot matching any of the entries already inppsAddListis added toppsAddList.
-
// Provided by VK_KHR_video_queue
typedef VkFlags VkVideoSessionParametersCreateFlagsKHR;
VkVideoSessionParametersCreateFlagsKHR is a bitmask type for setting a
mask, but is currently reserved for future use.
42.7.2. Destroying Video Session Parameters
To destroy a video session parameters object, call:
// Provided by VK_KHR_video_queue
void vkDestroyVideoSessionParametersKHR(
VkDevice device,
VkVideoSessionParametersKHR videoSessionParameters,
const VkAllocationCallbacks* pAllocator);
-
deviceis the logical device that destroys the video session parameters object. -
videoSessionParametersis the video session parameters object to destroy. -
pAllocatorcontrols host memory allocation as described in the Memory Allocation chapter.
42.7.3. Updating Video Session Parameters
To update video session parameters object with new parameters, call:
// Provided by VK_KHR_video_queue
VkResult vkUpdateVideoSessionParametersKHR(
VkDevice device,
VkVideoSessionParametersKHR videoSessionParameters,
const VkVideoSessionParametersUpdateInfoKHR* pUpdateInfo);
-
deviceis the logical device that updates the video session parameters. -
videoSessionParametersis the video session parameters object to update. -
pUpdateInfois a pointer to a VkVideoSessionParametersUpdateInfoKHR structure specifying the parameter update information.
After a successful call to this command, the
update sequence counter of
videoSessionParameters is changed to the value specified in
pUpdateInfo->updateSequenceCount.
|
Note
As each update issued to a video session parameters object needs to specify the next available update sequence count value, concurrent updates of the same video session parameters object are inherently disallowed. However, recording video coding operations to command buffers referring to parameters previously added to the video session parameters object is allowed, even if there is a concurrent update in progress adding some new entries to the object. |
If videoSessionParameters was created with the video codec operation
VK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR and the
pUpdateInfo->pNext chain includes a
VkVideoDecodeH264SessionParametersAddInfoKHR structure, then this
command adds the following parameter entries to
videoSessionParameters:
-
The H.264 SPS entries specified in VkVideoDecodeH264SessionParametersAddInfoKHR::
pStdSPSs. -
The H.264 PPS entries specified in VkVideoDecodeH264SessionParametersAddInfoKHR::
pStdPPSs.
If videoSessionParameters was created with the video codec operation
VK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR and the
pUpdateInfo->pNext chain includes a
VkVideoDecodeH265SessionParametersAddInfoKHR structure, then this
command adds the following parameter entries to
videoSessionParameters:
-
The H.265 VPS entries specified in VkVideoDecodeH265SessionParametersAddInfoKHR::
pStdVPSs. -
The H.265 SPS entries specified in VkVideoDecodeH265SessionParametersAddInfoKHR::
pStdSPSs. -
The H.265 PPS entries specified in VkVideoDecodeH265SessionParametersAddInfoKHR::
pStdPPSs.
If videoSessionParameters was created with the video codec operation
VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR and the
pUpdateInfo->pNext chain includes a
VkVideoEncodeH264SessionParametersAddInfoKHR structure, then this
command adds the following parameter entries to
videoSessionParameters:
-
The H.264 SPS entries specified in VkVideoEncodeH264SessionParametersAddInfoKHR::
pStdSPSs. -
The H.264 PPS entries specified in VkVideoEncodeH264SessionParametersAddInfoKHR::
pStdPPSs.
If videoSessionParameters was created with the video codec operation
VK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR and the
pUpdateInfo->pNext chain includes a
VkVideoEncodeH265SessionParametersAddInfoKHR structure, then this
command adds the following parameter entries to
videoSessionParameters:
-
The H.265 VPS entries specified in VkVideoEncodeH265SessionParametersAddInfoKHR::
pStdVPSs. -
The H.265 SPS entries specified in VkVideoEncodeH265SessionParametersAddInfoKHR::
pStdSPSs. -
The H.265 PPS entries specified in VkVideoEncodeH265SessionParametersAddInfoKHR::
pStdPPSs.
In case of video session parameters objects created with a video encode
operation, implementations may return the
VK_ERROR_INVALID_VIDEO_STD_PARAMETERS_KHR error if any of the
specified Video Std parameters do not adhere to the syntactic or semantic
requirements of the used video compression standard, or if values derived
from parameters according to the rules defined by the used video compression
standard do not adhere to the capabilities of the video compression standard
or the implementation.
|
Note
Applications should not rely on the
|
The VkVideoSessionParametersUpdateInfoKHR structure is defined as:
// Provided by VK_KHR_video_queue
typedef struct VkVideoSessionParametersUpdateInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t updateSequenceCount;
} VkVideoSessionParametersUpdateInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
updateSequenceCountis the new update sequence count to set for the video session parameters object.
42.8. Video Coding Scope
Applications can record video coding commands for a video session only within a video coding scope.
To begin a video coding scope, call:
// Provided by VK_KHR_video_queue
void vkCmdBeginVideoCodingKHR(
VkCommandBuffer commandBuffer,
const VkVideoBeginCodingInfoKHR* pBeginInfo);
-
commandBufferis the command buffer in which to record the command. -
pBeginInfois a pointer to a VkVideoBeginCodingInfoKHR structure specifying the parameters of the video coding scope, including the video session and video session parameters object to use.
After beginning a video coding scope, the video session object specified in
pBeginInfo->videoSession is bound to the command buffer, and the
command buffer is ready to record video coding operations.
Similarly, if pBeginInfo->videoSessionParameters is not
VK_NULL_HANDLE, it is also bound to the command buffer, and video
coding operations can refer to the codec-specific parameters stored in it.
This command also establishes the set of bound reference picture resources that can be used as reconstructed pictures or reference pictures within the video coding scope. Each element of this set consists of a video picture resource and the DPB slot index associated with it, if there is one.
The set of bound reference picture resources is immutable within a video coding scope, however, the DPB slot index associated with any of the bound reference picture resources can change during the video coding scope in response to video coding operations.
The VkVideoReferenceSlotInfoKHR structures provided as the elements of
pBeginInfo->pReferenceSlots are interpreted by this command as
follows:
-
If
slotIndexis non-negative andpPictureResourceis notNULL, then the video picture resource defined by the VkVideoPictureResourceInfoKHR structure pointed to bypPictureResourceis added to the set of bound reference picture resources and is associated with the DPB slot index specified inslotIndex. -
If
slotIndexis non-negative andpPictureResourceisNULL, then the DPB slot with indexslotIndexis deactivated by this command. -
If
slotIndexis negative andpPictureResourceis notNULL, then the video picture resource defined by the VkVideoPictureResourceInfoKHR structure pointed to bypPictureResourceis added to the set of bound reference picture resources without an associated DPB slot. Such a picture resource can be subsequently used as a reconstructed picture to associate it with a DPB slot. -
If
slotIndexis negative andpPictureResourceisNULL, then the element is ignored.
|
Note
It is possible for multiple bound reference picture resources to be associated with the same DPB slot index, or for a single bound reference picture to refer to multiple separate reference pictures. For example, in case of an H.264 decode profile with interlaced frame support a single DPB slot can refer to two separate pictures for the top and bottom fields. Depending on the picture layout used by the H.264 decode profile, the following special cases may arise:
|
All non-negative slotIndex values specified in the elements of
pBeginInfo->pReferenceSlots must identify DPB slots of the video
session that are in the active state at the time this
command is executed on the device.
|
Note
The application does not have to specify an entry in
|
In case of a video encode session, the application is also responsible for
providing information about the current rate
control state configured for the video session by including an instance of
the VkVideoEncodeRateControlInfoKHR structure in the pNext chain
of pBeginInfo.
If no VkVideoEncodeRateControlInfoKHR is included, then the presence
of an empty VkVideoEncodeRateControlInfoKHR structure is implied which
indicates that the current rate control mode
is VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DEFAULT_KHR.
The specified state must match the
effective rate control state configured for the video session at the time
the recorded command is executed on the device.
|
Note
Including an instance of the VkVideoEncodeRateControlInfoKHR structure
in the |
The VkVideoBeginCodingInfoKHR structure is defined as:
// Provided by VK_KHR_video_queue
typedef struct VkVideoBeginCodingInfoKHR {
VkStructureType sType;
const void* pNext;
VkVideoBeginCodingFlagsKHR flags;
VkVideoSessionKHR videoSession;
VkVideoSessionParametersKHR videoSessionParameters;
uint32_t referenceSlotCount;
const VkVideoReferenceSlotInfoKHR* pReferenceSlots;
} VkVideoBeginCodingInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
flagsis reserved for future use. -
videoSessionis the video session object to be bound for the processing of the video commands. -
videoSessionParametersisVK_NULL_HANDLEor a handle of a VkVideoSessionParametersKHR object to be used for the processing of the video commands. IfVK_NULL_HANDLE, then no video session parameters object is bound for the duration of the video coding scope. -
referenceSlotCountis the number of elements in thepReferenceSlotsarray. -
pReferenceSlotsis a pointer to an array of VkVideoReferenceSlotInfoKHR structures specifying the information used to determine the set of bound reference picture resources for the video coding scope and their initial association with DPB slot indices.
Limiting values are defined below that are referenced by the relevant valid usage statements of this structure.
-
Let
VkOffset2D codedOffsetGranularitybe the minimum alignment requirement for the coded offset of video picture resources. Unless otherwise defined, the value of thexandymembers ofcodedOffsetGranularityare0.-
If
videoSessionwas created with an H.264 decode profile with a VkVideoDecodeH264ProfileInfoKHR::pictureLayoutofVK_VIDEO_DECODE_H264_PICTURE_LAYOUT_INTERLACED_SEPARATE_PLANES_BIT_KHR, thencodedOffsetGranularityis equal to VkVideoDecodeH264CapabilitiesKHR::fieldOffsetGranularity, as returned by vkGetPhysicalDeviceVideoCapabilitiesKHR for that video profile.
-
// Provided by VK_KHR_video_queue
typedef VkFlags VkVideoBeginCodingFlagsKHR;
VkVideoBeginCodingFlagsKHR is a bitmask type for setting a mask, but
is currently reserved for future use.
The VkVideoReferenceSlotInfoKHR structure is defined as:
// Provided by VK_KHR_video_queue
typedef struct VkVideoReferenceSlotInfoKHR {
VkStructureType sType;
const void* pNext;
int32_t slotIndex;
const VkVideoPictureResourceInfoKHR* pPictureResource;
} VkVideoReferenceSlotInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
slotIndexis the index of the DPB slot or a negative integer value. -
pPictureResourceisNULLor a pointer to a VkVideoPictureResourceInfoKHR structure describing the video picture resource associated with the DPB slot index specified byslotIndex.
To end a video coding scope, call:
// Provided by VK_KHR_video_queue
void vkCmdEndVideoCodingKHR(
VkCommandBuffer commandBuffer,
const VkVideoEndCodingInfoKHR* pEndCodingInfo);
-
commandBufferis the command buffer in which to record the command. -
pEndCodingInfois a pointer to a VkVideoEndCodingInfoKHR structure specifying the parameters for ending the video coding scope.
After ending a video coding scope, the video session object, the optional video session parameters object, and all reference picture resources previously bound by the corresponding vkCmdBeginVideoCodingKHR command are unbound.
The VkVideoEndCodingInfoKHR structure is defined as:
// Provided by VK_KHR_video_queue
typedef struct VkVideoEndCodingInfoKHR {
VkStructureType sType;
const void* pNext;
VkVideoEndCodingFlagsKHR flags;
} VkVideoEndCodingInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
flagsis reserved for future use.
// Provided by VK_KHR_video_queue
typedef VkFlags VkVideoEndCodingFlagsKHR;
VkVideoEndCodingFlagsKHR is a bitmask type for setting a mask, but is
currently reserved for future use.
42.9. Video Coding Control
To apply dynamic controls to the currently bound video session object, call:
// Provided by VK_KHR_video_queue
void vkCmdControlVideoCodingKHR(
VkCommandBuffer commandBuffer,
const VkVideoCodingControlInfoKHR* pCodingControlInfo);
-
commandBufferis the command buffer in which to record the command. -
pCodingControlInfois a pointer to a VkVideoCodingControlInfoKHR structure specifying the control parameters.
The control parameters provided in this call are applied to the video session at the time the command executes on the device and are in effect until a subsequent call to this command with the same video session bound changes the corresponding control parameters.
A newly created video session must be reset before performing video coding
operations using it by including VK_VIDEO_CODING_CONTROL_RESET_BIT_KHR
in pCodingControlInfo->flags.
The reset operation also returns all DPB slots of the video session to the
inactive state.
Correspondingly, any DPB slot index associated with the
bound reference picture resources is
removed.
For encode sessions, the reset operation returns rate control configuration to implementation default settings and sets the video encode quality level to zero.
After video coding operations are performed using a video session, the reset operation can be used to return the video session to the same initial state as after the reset of a newly created video session. This can be used, for example, when different video sequences are needed to be processed with the same video session object.
If pCodingControlInfo->flags includes
VK_VIDEO_CODING_CONTROL_ENCODE_RATE_CONTROL_BIT_KHR, then the command
replaces the rate control configuration maintained
by the video session with the configuration specified in the
VkVideoEncodeRateControlInfoKHR structure included in the
pCodingControlInfo->pNext chain.
If pCodingControlInfo->flags includes
VK_VIDEO_CODING_CONTROL_ENCODE_QUALITY_LEVEL_BIT_KHR, then the command
changes the current video encode quality level to
the value specified in the qualityLevel member of the
VkVideoEncodeQualityLevelInfoKHR structure included in the
pCodingControlInfo->pNext chain.
The VkVideoCodingControlInfoKHR structure is defined as:
// Provided by VK_KHR_video_queue
typedef struct VkVideoCodingControlInfoKHR {
VkStructureType sType;
const void* pNext;
VkVideoCodingControlFlagsKHR flags;
} VkVideoCodingControlInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
flagsis a bitmask of VkVideoCodingControlFlagsKHR specifying control flags.
Bits which can be set in VkVideoCodingControlInfoKHR::flags,
specifying the video coding control parameters to be modified, are:
// Provided by VK_KHR_video_queue
typedef enum VkVideoCodingControlFlagBitsKHR {
VK_VIDEO_CODING_CONTROL_RESET_BIT_KHR = 0x00000001,
// Provided by VK_KHR_video_encode_queue
VK_VIDEO_CODING_CONTROL_ENCODE_RATE_CONTROL_BIT_KHR = 0x00000002,
// Provided by VK_KHR_video_encode_queue
VK_VIDEO_CODING_CONTROL_ENCODE_QUALITY_LEVEL_BIT_KHR = 0x00000004,
} VkVideoCodingControlFlagBitsKHR;
-
VK_VIDEO_CODING_CONTROL_RESET_BIT_KHRindicates a request for the bound video session to be reset before other coding control parameters are applied. -
VK_VIDEO_CODING_CONTROL_ENCODE_RATE_CONTROL_BIT_KHRindicates that the coding control parameters include video encode rate control parameters (see VkVideoEncodeRateControlInfoKHR). -
VK_VIDEO_CODING_CONTROL_ENCODE_QUALITY_LEVEL_BIT_KHRindicates that the coding control parameters include video encode quality level parameters (see VkVideoEncodeQualityLevelInfoKHR).
// Provided by VK_KHR_video_queue
typedef VkFlags VkVideoCodingControlFlagsKHR;
VkVideoCodingControlFlagsKHR is a bitmask type for setting a mask of
zero or more VkVideoCodingControlFlagBitsKHR.
42.10. Inline Queries
If a video session was created with
VK_VIDEO_SESSION_CREATE_INLINE_QUERIES_BIT_KHR, beginning queries
using commands such as vkCmdBeginQuery within a video coding scope is
not allowed.
Instead, queries are executed inline by including an instance of the
VkVideoInlineQueryInfoKHR structure in the pNext chain of the
parameters of one of the video coding commands, with its queryPool
member set to a valid VkQueryPool handle.
The VkVideoInlineQueryInfoKHR structure is defined as:
// Provided by VK_KHR_video_maintenance1
typedef struct VkVideoInlineQueryInfoKHR {
VkStructureType sType;
const void* pNext;
VkQueryPool queryPool;
uint32_t firstQuery;
uint32_t queryCount;
} VkVideoInlineQueryInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
queryPoolisVK_NULL_HANDLEor a valid handle to a VkQueryPool object that will manage the results of the queries. -
firstQueryis the query index within the query pool that will contain the query results for the first video coding operation. The query results of subsequent video coding operations will be contained by subsequent query indices. -
queryCountis the number of queries to execute.NoteIn practice, if
queryPoolis notVK_NULL_HANDLE, thenqueryCountwill always have to match the number of video coding operations issued by the video coding command this structure is specified to, meaning that using inline queries in a video coding command will always execute a query for each issued video coding operation.
This structure can be included in the pNext chain of the input
parameter structure of video coding commands.
-
In the
pNextchain of thepDecodeInfoparameter of the vkCmdDecodeVideoKHR command to execute a query for each video decode operation issued by the command. -
In the
pNextchain of thepEncodeInfoparameter of the vkCmdEncodeVideoKHR command to execute a query for each video encode operation issued by the command.
42.11. Video Decode Operations
Video decode operations consume compressed video data from a video bitstream buffer and zero or more reference pictures, and produce a decode output picture and an optional reconstructed picture.
|
Note
Such decode output pictures can be shared with the Decoded Picture Buffer, and can also be used as the input of video encode operations, with graphics or compute operations, or with Window System Integration APIs, depending on the capabilities of the implementation. |
Video decode operations may access the following resources in the
VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR stage:
-
The source video bitstream buffer range and the image subregions corresponding to the list of active reference pictures with access
VK_ACCESS_2_VIDEO_DECODE_READ_BIT_KHR. -
The image subregions corresponding to the target decode output picture and reconstructed picture with access
VK_ACCESS_2_VIDEO_DECODE_WRITE_BIT_KHR.
The image subresource of each video picture resource accessed by the video decode operation is specified using a corresponding VkVideoPictureResourceInfoKHR structure. Each such image subresource must be in the appropriate image layout as follows:
-
If the image subresource is used in the video decode operation only as decode output picture, then it must be in the
VK_IMAGE_LAYOUT_VIDEO_DECODE_DST_KHRlayout. -
If the image subresource is used in the video decode operation both as decode output picture and reconstructed picture, then it must be in the
VK_IMAGE_LAYOUT_VIDEO_DECODE_DPB_KHRlayout. -
If the image subresource is used in the video decode operation only as reconstructed picture, then it must be in the
VK_IMAGE_LAYOUT_VIDEO_DECODE_DPB_KHRlayout. -
If the image subresource is used in the video decode operation as a reference picture, then it must be in the
VK_IMAGE_LAYOUT_VIDEO_DECODE_DPB_KHRlayout.
A video decode operation may complete unsuccessfully. In this case the decode output picture will have undefined contents. Similarly, if reference picture setup is requested, the reconstructed picture will also have undefined contents, and the activated DPB slot will have an invalid picture reference.
42.11.1. Codec-Specific Semantics
The following aspects of video decode operations are codec-specific:
-
The interpretation of the contents of the source video bitstream buffer range.
-
The construction and interpretation of the list of active reference pictures and the interpretation of the picture data referred to by the corresponding image subregions.
-
The construction and interpretation of information related to the decode output picture and the generation of picture data to the corresponding image subregion.
-
The decision on reference picture setup.
-
The construction and interpretation of information related to the optional reconstructed picture and the generation of picture data to the corresponding image subregion.
These codec-specific behaviors are defined for each video codec operation separately.
-
If the used video codec operation is
VK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR, then the codec-specific aspects of the video decoding process are performed as defined in the H.264 Decode Operations section. -
If the used video codec operation is
VK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR, then the codec-specific aspects of the video decoding process are performed as defined in the H.265 Decode Operations section.
42.11.2. Video Decode Operation Steps
Each video decode operation performs the following steps in the
VK_PIPELINE_STAGE_2_VIDEO_DECODE_BIT_KHR stage:
-
Reads the encoded video data from the source video bitstream buffer range.
-
Performs picture reconstruction of the encoded video data according to the codec-specific semantics, applying any prediction data read from the active reference pictures in the process;
-
Writes the decoded picture data to the decode output picture, and optionally to the reconstructed picture, if one is specified and is different from the decode output picture, according to the codec-specific semantics;
-
If reference picture setup is requested, the DPB slot index specified in the reconstructed picture information is activated with the reconstructed picture.
When reconstructed picture information is provided, the specified DPB slot index is associated with the corresponding bound reference picture resource, indifferent of whether reference picture setup is requested.
42.11.3. Capabilities
When calling vkGetPhysicalDeviceVideoCapabilitiesKHR with
pVideoProfile->videoCodecOperation specifying a decode operation, the
VkVideoDecodeCapabilitiesKHR structure must be included in the
pNext chain of the VkVideoCapabilitiesKHR structure to retrieve
capabilities specific to video decoding.
The VkVideoDecodeCapabilitiesKHR structure is defined as:
// Provided by VK_KHR_video_decode_queue
typedef struct VkVideoDecodeCapabilitiesKHR {
VkStructureType sType;
void* pNext;
VkVideoDecodeCapabilityFlagsKHR flags;
} VkVideoDecodeCapabilitiesKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
flagsis a bitmask of VkVideoDecodeCapabilityFlagBitsKHR describing the supported video decoding capabilities.
Bits which may be set in VkVideoDecodeCapabilitiesKHR::flags,
indicating the decoding capabilities supported, are:
// Provided by VK_KHR_video_decode_queue
typedef enum VkVideoDecodeCapabilityFlagBitsKHR {
VK_VIDEO_DECODE_CAPABILITY_DPB_AND_OUTPUT_COINCIDE_BIT_KHR = 0x00000001,
VK_VIDEO_DECODE_CAPABILITY_DPB_AND_OUTPUT_DISTINCT_BIT_KHR = 0x00000002,
} VkVideoDecodeCapabilityFlagBitsKHR;
-
VK_VIDEO_DECODE_CAPABILITY_DPB_AND_OUTPUT_COINCIDE_BIT_KHRindicates support for using the same video picture resource as the reconstructed picture and decode output picture in a video decode operation. -
VK_VIDEO_DECODE_CAPABILITY_DPB_AND_OUTPUT_DISTINCT_BIT_KHRindicates support for using distinct video picture resources as the reconstructed picture and decode output picture in a video decode operation.
Implementations are only required to support one of
VK_VIDEO_DECODE_CAPABILITY_DPB_AND_OUTPUT_COINCIDE_BIT_KHR and
VK_VIDEO_DECODE_CAPABILITY_DPB_AND_OUTPUT_DISTINCT_BIT_KHR.
Accordingly, applications should handle both cases to maximize portability.
|
Note
If both |
// Provided by VK_KHR_video_decode_queue
typedef VkFlags VkVideoDecodeCapabilityFlagsKHR;
VkVideoDecodeCapabilityFlagsKHR is a bitmask type for setting a mask
of zero or more VkVideoDecodeCapabilityFlagBitsKHR.
42.11.4. Video Decode Commands
To launch video decode operations, call:
// Provided by VK_KHR_video_decode_queue
void vkCmdDecodeVideoKHR(
VkCommandBuffer commandBuffer,
const VkVideoDecodeInfoKHR* pDecodeInfo);
-
commandBufferis the command buffer in which to record the command. -
pDecodeInfois a pointer to a VkVideoDecodeInfoKHR structure specifying the parameters of the video decode operations.
Each call issues one or more video decode operations.
The implicit parameter opCount corresponds to the number of video
decode operations issued by the command.
After calling this command, the
active query index of each
active query is incremented by opCount.
Currently each call to this command results in the issue of a single video decode operation.
If the bound video session was created with
VK_VIDEO_SESSION_CREATE_INLINE_QUERIES_BIT_KHR and the pNext
chain of pDecodeInfo includes a VkVideoInlineQueryInfoKHR
structure with its queryPool member specifying a valid
VkQueryPool handle, then this command will execute a query for each
video decode operation issued by it.
- Active Reference Picture Information
-
The list of active reference pictures used by a video decode operation is a list of image subregions used as the source of reference picture data and related parameters, and is derived from the VkVideoReferenceSlotInfoKHR structures provided as the elements of the
pDecodeInfo->pReferenceSlotsarray. For each element ofpDecodeInfo->pReferenceSlots, one or more elements are added to the active reference picture list, as defined by the codec-specific semantics. Each element of this list contains the following information:-
The image subregion within the image subresource referred to by the video picture resource used as the reference picture.
-
The DPB slot index the reference picture is associated with.
-
The codec-specific reference information related to the reference picture.
-
- Reconstructed Picture Information
-
Information related to the optional reconstructed picture used by a video decode operation is derived from the VkVideoReferenceSlotInfoKHR structure pointed to by
pDecodeInfo->pSetupReferenceSlot, if notNULL, as defined by the codec-specific semantics, and consists of the following:-
The image subregion within the image subresource referred to by the video picture resource used as the reconstructed picture.
-
The DPB slot index to use for picture reconstruction.
-
The codec-specific reference information related to the reconstructed picture.
-
Specifying a valid VkVideoReferenceSlotInfoKHR structure in
pDecodeInfo->pSetupReferenceSlot is always required, unless the video
session was created with VkVideoSessionCreateInfoKHR::maxDpbSlot
equal to zero.
However, the DPB slot identified by
pDecodeInfo->pSetupReferenceSlot->slotIndex is only
activated with the reconstructed
picture specified in
pDecodeInfo->pSetupReferenceSlot->pPictureResource if reference
picture setup is requested according to the
codec-specific semantics.
If reconstructed picture information is specified, and
pDecodeInfo->pSetupReferenceSlot->pPictureResource refers to a
video picture resource different than that of
the decode output picture, but reference picture
setup is not requested, the contents of the video
picture resource corresponding to the reconstructed picture will be
undefined after the video decode operation.
|
Note
Some implementations may always output the reconstructed picture or use it as temporary storage during the video decode operation even when the reconstructed picture is not marked for future reference. |
- Decode Output Picture Information
-
Information related to the decode output picture used by a video decode operation is derived from
pDecodeInfo->dstPictureResourceand any codec-specific parameters provided in thepDecodeInfo->pNextchain, as defined by the codec-specific semantics, and consists of the following:-
The image subregion within the image subresource referred to by the video picture resource used as the decode output picture.
-
The codec-specific picture information related to the decode output picture.
-
Several limiting values are defined below that are referenced by the relevant valid usage statements of this command.
-
Let
uint32_t activeReferencePictureCountbe the size of the list of active reference pictures used by the video decode operation. Unless otherwise defined,activeReferencePictureCountis set to the value ofpDecodeInfo->referenceSlotCount.-
If the bound video session was created with an H.264 decode profile, then let
activeReferencePictureCountbe the value ofpDecodeInfo->referenceSlotCountplus the number of elements of thepDecodeInfo->pReferenceSlotsarray that have a VkVideoDecodeH264DpbSlotInfoKHR structure included in theirpNextchain with bothpStdReferenceInfo->flags.top_field_flagandpStdReferenceInfo->flags.bottom_field_flagset.NoteThis means that the elements of
pDecodeInfo->pReferenceSlotsthat include both a top and bottom field reference are counted as two separate active reference pictures, as described in the active reference picture list construction rules for H.264 decode operations.
-
-
Let
VkOffset2D codedOffsetGranularitybe the minimum alignment requirement for the coded offset of video picture resources. Unless otherwise defined, the value of thexandymembers ofcodedOffsetGranularityare0.-
If the bound video session was created with an H.264 decode profile with a VkVideoDecodeH264ProfileInfoKHR::
pictureLayoutofVK_VIDEO_DECODE_H264_PICTURE_LAYOUT_INTERLACED_SEPARATE_PLANES_BIT_KHR, thencodedOffsetGranularityis equal to VkVideoDecodeH264CapabilitiesKHR::fieldOffsetGranularity, as returned by vkGetPhysicalDeviceVideoCapabilitiesKHR for that video profile.
-
-
Let
uint32_t dpbFrameUseCount[]be an array of sizemaxDpbSlots, wheremaxDpbSlotsis the VkVideoSessionCreateInfoKHR::maxDpbSlotsthe bound video session was created with, with each element indicating the number of times a frame associated with the corresponding DPB slot index is referred to by the video coding operation. Let the initial value of each element of the array be0.-
If
pDecodeInfo->pSetupReferenceSlotis notNULL, thendpbFrameUseCount[i]is incremented by one, whereiequalspDecodeInfo->pSetupReferenceSlot->slotIndex. If the bound video session object was created with an H.264 decode profile, thendpbFrameUseCount[i]is decremented by one if eitherpStdReferenceInfo->flags.top_field_flagorpStdReferenceInfo->flags.bottom_field_flagis set in the VkVideoDecodeH264DpbSlotInfoKHR structure in thepDecodeInfo->pSetupReferenceSlot->pNextchain. -
For each element of
pDecodeInfo->pReferenceSlots,dpbFrameUseCount[i]is incremented by one, whereiequals theslotIndexmember of the corresponding element. If the bound video session object was created with an H.264 decode profile, thendpbFrameUseCount[i]is decremented by one if eitherpStdReferenceInfo->flags.top_field_flagorpStdReferenceInfo->flags.bottom_field_flagis set in the VkVideoDecodeH264DpbSlotInfoKHR structure in thepNextchain of the corresponding element ofpDecodeInfo->pReferenceSlots.
-
-
Let
uint32_t dpbTopFieldUseCount[]anduint32_t dpbBottomFieldUseCount[]be arrays of sizemaxDpbSlots, wheremaxDpbSlotsis the VkVideoSessionCreateInfoKHR::maxDpbSlotsthe bound video session was created with, with each element indicating the number of times the top field or the bottom field, respectively, associated with the corresponding DPB slot index is referred to by the video coding operation. Let the initial value of each element of the arrays be0.-
If the bound video session object was created with an H.264 decode profile and
pDecodeInfo->pSetupReferenceSlotis notNULL, then perform the following:-
If
pStdReferenceInfo->flags.top_field_flagis set in the VkVideoDecodeH264DpbSlotInfoKHR structure in thepDecodeInfo->pSetupReferenceSlot->pNextchain, thendpbTopFieldUseCount[i]is incremented by one, whereiequalspDecodeInfo->pSetupReferenceSlot->slotIndex. -
If
pStdReferenceInfo->flags.bottom_field_flagis set in the VkVideoDecodeH264DpbSlotInfoKHR structure in thepDecodeInfo->pSetupReferenceSlot->pNextchain, thendpbBottomFieldUseCount[i]is incremented by one, whereiequalspDecodeInfo->pSetupReferenceSlot->slotIndex.
-
-
If the bound video session object was created with an H.264 decode profile, then perform the following for each element of
pDecodeInfo->pReferenceSlots:-
If
pStdReferenceInfo->flags.top_field_flagis set in the VkVideoDecodeH264DpbSlotInfoKHR structure in thepNextchain of the element, thendpbTopFieldUseCount[i]is incremented by one, whereiequals theslotIndexmember of the element. -
If
pStdReferenceInfo->flags.bottom_field_flagis set in the VkVideoDecodeH264DpbSlotInfoKHR structure in thepNextchain of the element, thendpbBottomFieldUseCount[i]is incremented by one, whereiequals theslotIndexmember of the element.
-
-
The VkVideoDecodeInfoKHR structure is defined as:
// Provided by VK_KHR_video_decode_queue
typedef struct VkVideoDecodeInfoKHR {
VkStructureType sType;
const void* pNext;
VkVideoDecodeFlagsKHR flags;
VkBuffer srcBuffer;
VkDeviceSize srcBufferOffset;
VkDeviceSize srcBufferRange;
VkVideoPictureResourceInfoKHR dstPictureResource;
const VkVideoReferenceSlotInfoKHR* pSetupReferenceSlot;
uint32_t referenceSlotCount;
const VkVideoReferenceSlotInfoKHR* pReferenceSlots;
} VkVideoDecodeInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
flagsis reserved for future use. -
srcBufferis the source video bitstream buffer to read the encoded bitstream from. -
srcBufferOffsetis the starting offset in bytes from the start ofsrcBufferto read the encoded bitstream from. -
srcBufferRangeis the size in bytes of the encoded bitstream to decode fromsrcBuffer, starting fromsrcBufferOffset. -
dstPictureResourceis the video picture resource to use as the decode output picture. -
pSetupReferenceSlotisNULLor a pointer to a VkVideoReferenceSlotInfoKHR structure specifying the reconstructed picture information. -
referenceSlotCountis the number of elements in thepReferenceSlotsarray. -
pReferenceSlotsisNULLor a pointer to an array of VkVideoReferenceSlotInfoKHR structures describing the DPB slots and corresponding reference picture resources to use in this video decode operation (the set of active reference pictures).
// Provided by VK_KHR_video_decode_queue
typedef VkFlags VkVideoDecodeFlagsKHR;
VkVideoDecodeFlagsKHR is a bitmask type for setting a mask, but is
currently reserved for future use.
42.12. H.264 Decode Operations
Video decode operations using an H.264 decode profile can be used to decode elementary video stream sequences compliant to the ITU-T H.264 Specification.
|
Note
Refer to the Preamble for information on how the Khronos Intellectual Property Rights Policy relates to normative references to external materials not created by Khronos. |
This process is performed according to the video decode operation steps with the codec-specific semantics defined in section 8 of the ITU-T H.264 Specification as follows:
-
Syntax elements, derived values, and other parameters are applied from the following structures:
-
The
StdVideoH264SequenceParameterSetstructure corresponding to the active SPS specifying the H.264 sequence parameter set. -
The
StdVideoH264PictureParameterSetstructure corresponding to the active PPS specifying the H.264 picture parameter set. -
The
StdVideoDecodeH264PictureInfostructure specifying the H.264 picture information. -
The
StdVideoDecodeH264ReferenceInfostructures specifying the H.264 reference information corresponding to the optional reconstructed picture and any active reference pictures.
-
-
The contents of the provided video bitstream buffer range are interpreted as defined in the H.264 Decode Bitstream Data Access section.
-
Picture data in the video picture resources corresponding to the used active reference pictures, decode output picture, and optional reconstructed picture is accessed as defined in the H.264 Decode Picture Data Access section.
-
The decision on reference picture setup is made according to the parameters specified in the H.264 picture information.
If the parameters and the bitstream adhere to the syntactic and semantic requirements defined in the corresponding sections of the ITU-T H.264 Specification, as described above, and the DPB slots associated with the active reference pictures all refer to valid picture references, then the video decode operation will complete successfully. Otherwise, the video decode operation may complete unsuccessfully.
42.12.1. H.264 Decode Bitstream Data Access
If the target decode output picture is a frame, then the video bitstream buffer range should contain a VCL NAL unit comprised of the slice headers and data of a picture representing an entire frame, as defined in sections 7.3.3 and 7.3.4, and this data is interpreted as defined in sections 7.4.3 and 7.4.4 of the ITU-T H.264 Specification, respectively.
If the target decode output picture is a field, then the video bitstream buffer range should contain a VCL NAL unit comprised of the slice headers and data of a picture representing a field, as defined in sections 7.3.3 and 7.3.4, and this data is interpreted as defined in sections 7.4.3 and 7.4.4 of the ITU-T H.264 Specification, respectively.
The offsets provided in
VkVideoDecodeH264PictureInfoKHR::pSliceOffsets should specify
the starting offsets corresponding to each slice header within the video
bitstream buffer range.
42.12.2. H.264 Decode Picture Data Access
The effective imageOffset and imageExtent corresponding to a
decode output picture,
reference picture, or
reconstructed picture used in video decode
operations with an H.264 decode profile are defined
as follows:
-
imageOffsetis (codedOffset.x,codedOffset.y) andimageExtentis (codedExtent.width,codedExtent.height), if the picture represents a frame. -
imageOffsetis (codedOffset.x,codedOffset.y) andimageExtentis (codedExtent.width,codedExtent.height), if the picture represents a field and the picture layout of the used H.264 decode profile isVK_VIDEO_DECODE_H264_PICTURE_LAYOUT_INTERLACED_INTERLEAVED_LINES_BIT_KHR. -
imageOffsetis (codedOffset.x,codedOffset.y) andimageExtentis (codedExtent.width,codedExtent.height/ 2), if the picture represents a field and the picture layout of the used H.264 decode profile isVK_VIDEO_DECODE_H264_PICTURE_LAYOUT_INTERLACED_SEPARATE_PLANES_BIT_KHR.
Where codedOffset and codedExtent are the members of the
VkVideoPictureResourceInfoKHR structure corresponding to the picture.
However, accesses to image data within a video picture resource happen at
the granularity indicated by
VkVideoCapabilitiesKHR::pictureAccessGranularity, as returned by
vkGetPhysicalDeviceVideoCapabilitiesKHR for the used video profile.
This means that the complete image subregion accessed by video coding
operations using an H.264 decode profile for the
video picture resource is defined as the set of texels within the coordinate
range:
-
([
startX,endX),[startY,endY))
Where:
-
startXequalsimageOffset.xrounded down to the nearest integer multiple ofpictureAccessGranularity.width; -
endXequalsimageOffset.x+imageExtent.widthrounded up to the nearest integer multiple ofpictureAccessGranularity.widthand clamped to the width of the image subresource referred to by the corresponding VkVideoPictureResourceInfoKHR structure; -
startY equals
imageOffset.yrounded down to the nearest integer multiple ofpictureAccessGranularity.height; -
endY equals
imageOffset.y+imageExtent.heightrounded up to the nearest integer multiple ofpictureAccessGranularity.heightand clamped to the height of the image subresource referred to by the corresponding VkVideoPictureResourceInfoKHR structure.
In case of video decode operations using an H.264
decode profile, any access to a picture at the coordinates
(x,y), as defined by the ITU-T H.264
Specification, is an access to the image subresource
referred to by the corresponding
VkVideoPictureResourceInfoKHR structure at the texel coordinates
specified below:
-
(
x,y), if the accessed picture represents a frame. -
(
x,y× 2), if the accessed picture represents a top field and the picture layout of the used H.264 decode profile isVK_VIDEO_DECODE_H264_PICTURE_LAYOUT_INTERLACED_INTERLEAVED_LINES_BIT_KHR. -
(
x,y× 2 + 1), if the accessed picture represents a bottom field and the picture layout of the used H.264 decode profile isVK_VIDEO_DECODE_H264_PICTURE_LAYOUT_INTERLACED_INTERLEAVED_LINES_BIT_KHR. -
(
x,y), if the accessed picture represents a top field and the picture layout of the used H.264 decode profile isVK_VIDEO_DECODE_H264_PICTURE_LAYOUT_INTERLACED_SEPARATE_PLANES_BIT_KHR. -
(
codedOffset.x+x,codedOffset.y+y), if the accessed picture represents a bottom field and the picture layout of the used H.264 decode profile isVK_VIDEO_DECODE_H264_PICTURE_LAYOUT_INTERLACED_SEPARATE_PLANES_BIT_KHR.
Where codedOffset is the member of the corresponding
VkVideoPictureResourceInfoKHR structure.
42.12.3. H.264 Decode Profile
A video profile supporting H.264 video decode operations is specified by
setting VkVideoProfileInfoKHR::videoCodecOperation to
VK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR and adding a
VkVideoDecodeH264ProfileInfoKHR structure to the
VkVideoProfileInfoKHR::pNext chain.
The VkVideoDecodeH264ProfileInfoKHR structure is defined as:
// Provided by VK_KHR_video_decode_h264
typedef struct VkVideoDecodeH264ProfileInfoKHR {
VkStructureType sType;
const void* pNext;
StdVideoH264ProfileIdc stdProfileIdc;
VkVideoDecodeH264PictureLayoutFlagBitsKHR pictureLayout;
} VkVideoDecodeH264ProfileInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
stdProfileIdcis aStdVideoH264ProfileIdcvalue specifying the H.264 codec profile IDC, as defined in section A.2 of the ITU-T H.264 Specification. -
pictureLayoutis a VkVideoDecodeH264PictureLayoutFlagBitsKHR value specifying the picture layout used by the H.264 video sequence to be decoded.
The H.264 video decode picture layout flags are defined as follows:
// Provided by VK_KHR_video_decode_h264
typedef enum VkVideoDecodeH264PictureLayoutFlagBitsKHR {
VK_VIDEO_DECODE_H264_PICTURE_LAYOUT_PROGRESSIVE_KHR = 0,
VK_VIDEO_DECODE_H264_PICTURE_LAYOUT_INTERLACED_INTERLEAVED_LINES_BIT_KHR = 0x00000001,
VK_VIDEO_DECODE_H264_PICTURE_LAYOUT_INTERLACED_SEPARATE_PLANES_BIT_KHR = 0x00000002,
} VkVideoDecodeH264PictureLayoutFlagBitsKHR;
-
VK_VIDEO_DECODE_H264_PICTURE_LAYOUT_PROGRESSIVE_KHRspecifies support for progressive content. This flag has the value0. -
VK_VIDEO_DECODE_H264_PICTURE_LAYOUT_INTERLACED_INTERLEAVED_LINES_BIT_KHRspecifies support for or use of a picture layout for interlaced content where all lines belonging to the top field are decoded to the even-numbered lines within the picture resource, and all lines belonging to the bottom field are decoded to the odd-numbered lines within the picture resource. -
VK_VIDEO_DECODE_H264_PICTURE_LAYOUT_INTERLACED_SEPARATE_PLANES_BIT_KHRspecifies support for or use of a picture layout for interlaced content where all lines belonging to a field are grouped together in a single image subregion, and the two fields comprising the frame can be stored in separate image subregions of the same image subresource or in separate image subresources.
// Provided by VK_KHR_video_decode_h264
typedef VkFlags VkVideoDecodeH264PictureLayoutFlagsKHR;
VkVideoDecodeH264PictureLayoutFlagsKHR is a bitmask type for setting a
mask of zero or more VkVideoDecodeH264PictureLayoutFlagBitsKHR.
42.12.4. H.264 Decode Capabilities
When calling vkGetPhysicalDeviceVideoCapabilitiesKHR to query the
capabilities for an H.264 decode profile, the
VkVideoCapabilitiesKHR::pNext chain must include a
VkVideoDecodeH264CapabilitiesKHR structure that will be filled with
the profile-specific capabilities.
The VkVideoDecodeH264CapabilitiesKHR structure is defined as:
// Provided by VK_KHR_video_decode_h264
typedef struct VkVideoDecodeH264CapabilitiesKHR {
VkStructureType sType;
void* pNext;
StdVideoH264LevelIdc maxLevelIdc;
VkOffset2D fieldOffsetGranularity;
} VkVideoDecodeH264CapabilitiesKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
maxLevelIdcis aStdVideoH264LevelIdcvalue indicating the maximum H.264 level supported by the profile, where enum constantSTD_VIDEO_H264_LEVEL_IDC_<major>_<minor>identifies H.264 level<major>.<minor>as defined in section A.3 of the ITU-T H.264 Specification. -
fieldOffsetGranularityis the minimum alignment for VkVideoPictureResourceInfoKHR::codedOffsetspecified for a video picture resource when using the picture layoutVK_VIDEO_DECODE_H264_PICTURE_LAYOUT_INTERLACED_SEPARATE_PLANES_BIT_KHR.
42.12.5. H.264 Decode Parameter Sets
Video session parameters objects created with
the video codec operation VK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR
can contain the following types of parameters:
- H.264 Sequence Parameter Sets (SPS)
-
Represented by
StdVideoH264SequenceParameterSetstructures and interpreted as follows:-
reserved1andreserved2are used only for padding purposes and are otherwise ignored; -
seq_parameter_set_idis used as the key of the SPS entry; -
level_idcis one of the enum constantsSTD_VIDEO_H264_LEVEL_IDC_<major>_<minor>identifying the H.264 level<major>.<minor>as defined in section A.3 of the ITU-T H.264 Specification; -
if
flags.seq_scaling_matrix_present_flagis set, then theStdVideoH264ScalingListsstructure pointed to bypScalingListsis interpreted as follows:-
scaling_list_present_maskis a bitmask where bit index i corresponds toseq_scaling_list_present_flag[i]as defined in section 7.4.2.1 of the ITU-T H.264 Specification; -
use_default_scaling_matrix_maskis a bitmask where bit index i corresponds toUseDefaultScalingMatrix4x4Flag[i], when i < 6, or corresponds toUseDefaultScalingMatrix8x8Flag[i-6], otherwise, as defined in section 7.3.2.1 of the ITU-T H.264 Specification; -
ScalingList4x4andScalingList8x8correspond to the identically named syntax elements defined in section 7.3.2.1 of the ITU-T H.264 Specification;
-
-
if
flags.vui_parameters_present_flagis set, thenpSequenceParameterSetVuiis a pointer to aStdVideoH264SequenceParameterSetVuistructure that is interpreted as follows:-
reserved1is used only for padding purposes and is otherwise ignored; -
if
flags.nal_hrd_parameters_present_flagorflags.vcl_hrd_parameters_present_flagis set, then theStdVideoH264HrdParametersstructure pointed to bypHrdParametersis interpreted as follows:-
reserved1is used only for padding purposes and is otherwise ignored; -
all other members of
StdVideoH264HrdParametersare interpreted as defined in section E.2.2 of the ITU-T H.264 Specification;
-
-
all other members of
StdVideoH264SequenceParameterSetVuiare interpreted as defined in section E.2.1 of the ITU-T H.264 Specification;
-
-
all other members of
StdVideoH264SequenceParameterSetare interpreted as defined in section 7.4.2.1 of the ITU-T H.264 Specification.
-
- H.264 Picture Parameter Sets (PPS)
-
Represented by
StdVideoH264PictureParameterSetstructures and interpreted as follows:-
the pair constructed from
seq_parameter_set_idandpic_parameter_set_idis used as the key of the PPS entry; -
if
flags.pic_scaling_matrix_present_flagis set, then theStdVideoH264ScalingListsstructure pointed to bypScalingListsis interpreted as follows:-
scaling_list_present_maskis a bitmask where bit index i corresponds topic_scaling_list_present_flag[i]as defined in section 7.4.2.2 of the ITU-T H.264 Specification; -
use_default_scaling_matrix_maskis a bitmask where bit index i corresponds toUseDefaultScalingMatrix4x4Flag[i], when i < 6, or corresponds toUseDefaultScalingMatrix8x8Flag[i-6], otherwise, as defined in section 7.3.2.2 of the ITU-T H.264 Specification; -
ScalingList4x4andScalingList8x8correspond to the identically named syntax elements defined in section 7.3.2.2 of the ITU-T H.264 Specification;
-
-
all other members of
StdVideoH264PictureParameterSetare interpreted as defined in section 7.4.2.2 of the ITU-T H.264 Specification.
-
When a video session parameters object is
created with the codec operation
VK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR, the
VkVideoSessionParametersCreateInfoKHR::pNext chain must include
a VkVideoDecodeH264SessionParametersCreateInfoKHR structure specifying
the capacity and initial contents of the object.
The VkVideoDecodeH264SessionParametersCreateInfoKHR structure is
defined as:
// Provided by VK_KHR_video_decode_h264
typedef struct VkVideoDecodeH264SessionParametersCreateInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t maxStdSPSCount;
uint32_t maxStdPPSCount;
const VkVideoDecodeH264SessionParametersAddInfoKHR* pParametersAddInfo;
} VkVideoDecodeH264SessionParametersCreateInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
maxStdSPSCountis the maximum number of H.264 SPS entries the createdVkVideoSessionParametersKHRcan contain. -
maxStdPPSCountis the maximum number of H.264 PPS entries the createdVkVideoSessionParametersKHRcan contain. -
pParametersAddInfoisNULLor a pointer to a VkVideoDecodeH264SessionParametersAddInfoKHR structure specifying H.264 parameters to add upon object creation.
The VkVideoDecodeH264SessionParametersAddInfoKHR structure is defined
as:
// Provided by VK_KHR_video_decode_h264
typedef struct VkVideoDecodeH264SessionParametersAddInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t stdSPSCount;
const StdVideoH264SequenceParameterSet* pStdSPSs;
uint32_t stdPPSCount;
const StdVideoH264PictureParameterSet* pStdPPSs;
} VkVideoDecodeH264SessionParametersAddInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
stdSPSCountis the number of elements in thepStdSPSsarray. -
pStdSPSsis a pointer to an array ofStdVideoH264SequenceParameterSetstructures describing the H.264 SPS entries to add. -
stdPPSCountis the number of elements in thepStdPPSsarray. -
pStdPPSsis a pointer to an array ofStdVideoH264PictureParameterSetstructures describing the H.264 PPS entries to add.
This structure can be specified in the following places:
-
In the
pParametersAddInfomember of the VkVideoDecodeH264SessionParametersCreateInfoKHR structure specified in thepNextchain of VkVideoSessionParametersCreateInfoKHR used to create a video session parameters object. In this case, if the video codec operation the video session parameters object is created with isVK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR, then it defines the set of initial parameters to add to the created object (see Creating Video Session Parameters). -
In the
pNextchain of VkVideoSessionParametersUpdateInfoKHR. In this case, if the video codec operation the video session parameters object to be updated was created with isVK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR, then it defines the set of parameters to add to it (see Updating Video Session Parameters).
42.12.6. H.264 Decoding Parameters
The VkVideoDecodeH264PictureInfoKHR structure is defined as:
// Provided by VK_KHR_video_decode_h264
typedef struct VkVideoDecodeH264PictureInfoKHR {
VkStructureType sType;
const void* pNext;
const StdVideoDecodeH264PictureInfo* pStdPictureInfo;
uint32_t sliceCount;
const uint32_t* pSliceOffsets;
} VkVideoDecodeH264PictureInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
pStdPictureInfois a pointer to aStdVideoDecodeH264PictureInfostructure specifying H.264 picture information. -
sliceCountis the number of elements inpSliceOffsets. -
pSliceOffsetsis a pointer to an array ofsliceCountoffsets specifying the start offset of the slices of the picture within the video bitstream buffer range specified in VkVideoDecodeInfoKHR.
This structure is specified in the pNext chain of the
VkVideoDecodeInfoKHR structure passed to vkCmdDecodeVideoKHR to
specify the codec-specific picture information for an H.264
decode operation.
- Decode Output Picture Information
-
When this structure is specified in the
pNextchain of the VkVideoDecodeInfoKHR structure passed to vkCmdDecodeVideoKHR, the information related to the decode output picture is defined as follows:-
If
pStdPictureInfo->flags.field_pic_flagis not set, then the picture represents a frame. -
If
pStdPictureInfo->flags.field_pic_flagis set, then the picture represents a field. Specifically:-
If
pStdPictureInfo->flags.bottom_field_flagis not set, then the picture represents the top field of the frame. -
If
pStdPictureInfo->flags.bottom_field_flagis set, then the picture represents the bottom field of the frame.
-
-
The image subregion used is determined according to the H.264 Decode Picture Data Access section.
-
The decode output picture is associated with the H.264 picture information provided in
pStdPictureInfo.
-
- Std Picture Information
-
The members of the
StdVideoDecodeH264PictureInfostructure pointed to bypStdPictureInfoare interpreted as follows:-
reserved1andreserved2are used only for padding purposes and are otherwise ignored; -
flags.is_intraas defined in section 3.73 of the ITU-T H.264 Specification; -
flags.is_referenceas defined in section 3.136 of the ITU-T H.264 Specification; -
flags.complementary_field_pairas defined in section 3.35 of the ITU-T H.264 Specification; -
seq_parameter_set_idandpic_parameter_set_idare used to identify the active parameter sets, as described below; -
all other members are interpreted as defined in section 7.4.3 of the ITU-T H.264 Specification.
-
Reference picture setup is controlled by the value of
StdVideoDecodeH264PictureInfo::flags.is_reference.
If it is set and a reconstructed
picture is specified, then the latter is used as the target of picture
reconstruction to activate the DPB slot
specified in pDecodeInfo->pSetupReferenceSlot->slotIndex.
If StdVideoDecodeH264PictureInfo::flags.is_reference is not set,
but a reconstructed picture is
specified, then the corresponding picture reference associated with the
DPB slot is invalidated, as described in the
DPB Slot States section.
- Active Parameter Sets
-
The members of the
StdVideoDecodeH264PictureInfostructure pointed to bypStdPictureInfoare used to select the active parameter sets to use from the bound video session parameters object, as follows:-
The active SPS is the SPS identified by the key specified in
StdVideoDecodeH264PictureInfo::seq_parameter_set_id. -
The active PPS is the PPS identified by the key specified by the pair constructed from
StdVideoDecodeH264PictureInfo::seq_parameter_set_idandStdVideoDecodeH264PictureInfo::pic_parameter_set_id.
-
The VkVideoDecodeH264DpbSlotInfoKHR structure is defined as:
// Provided by VK_KHR_video_decode_h264
typedef struct VkVideoDecodeH264DpbSlotInfoKHR {
VkStructureType sType;
const void* pNext;
const StdVideoDecodeH264ReferenceInfo* pStdReferenceInfo;
} VkVideoDecodeH264DpbSlotInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
pStdReferenceInfois a pointer to aStdVideoDecodeH264ReferenceInfostructure specifying H.264 reference information.
This structure is specified in the pNext chain of
VkVideoDecodeInfoKHR::pSetupReferenceSlot, if not NULL, and
the pNext chain of the elements of
VkVideoDecodeInfoKHR::pReferenceSlots to specify the
codec-specific reference picture information for an H.264
decode operation.
- Active Reference Picture Information
-
When this structure is specified in the
pNextchain of the elements of VkVideoDecodeInfoKHR::pReferenceSlots, one or two elements are added to the list of active reference pictures used by the video decode operation for each element of VkVideoDecodeInfoKHR::pReferenceSlotsas follows:-
If neither
pStdReferenceInfo->flags.top_field_flagnorpStdReferenceInfo->flags.bottom_field_flagis set, then the picture is added as a frame reference to the list of active reference pictures. -
If
pStdReferenceInfo->flags.top_field_flagis set, then the picture is added as a top field reference to the list of active reference pictures. -
If
pStdReferenceInfo->flags.bottom_field_flagis set, then the picture is added as a bottom field reference to the list of active reference pictures. -
For each added reference picture, the corresponding image subregion used is determined according to the H.264 Decode Picture Data Access section.
-
Each added reference picture is associated with the DPB slot index specified in the
slotIndexmember of the corresponding element of VkVideoDecodeInfoKHR::pReferenceSlots. -
Each added reference picture is associated with the H.264 reference information provided in
pStdReferenceInfo.
-
|
Note
When both the top and bottom field of an interlaced frame currently
associated with a DPB slot is intended to be used as an active reference
picture and both fields are stored in the same image subregion (which is the
case when using
|
- Reconstructed Picture Information
-
When this structure is specified in the
pNextchain of VkVideoDecodeInfoKHR::pSetupReferenceSlot, the information related to the reconstructed picture is defined as follows:-
If neither
pStdReferenceInfo->flags.top_field_flagnorpStdReferenceInfo->flags.bottom_field_flagis set, then the picture represents a frame. -
If
pStdReferenceInfo->flags.top_field_flagis set, then the picture represents a field, specifically, the top field of the frame. -
If
pStdReferenceInfo->flags.bottom_field_flagis set, then the picture represents a field, specifically, the bottom field of the frame. -
The image subregion used is determined according to the H.264 Decode Picture Data Access section.
-
If reference picture setup is requested, then the reconstructed picture is used to activate the DPB slot with the index specified in VkVideoDecodeInfoKHR::
pSetupReferenceSlot->slotIndex. -
The reconstructed picture is associated with the H.264 reference information provided in
pStdReferenceInfo.
-
- Std Reference Information
-
The members of the
StdVideoDecodeH264ReferenceInfostructure pointed to bypStdReferenceInfoare interpreted as follows:-
flags.top_field_flagis used to indicate whether the reference is used as top field reference; -
flags.bottom_field_flagis used to indicate whether the reference is used as bottom field reference; -
flags.used_for_long_term_referenceis used to indicate whether the picture is marked as “used for long-term reference” as defined in section 8.2.5.1 of the ITU-T H.264 Specification; -
flags.is_non_existingis used to indicate whether the picture is marked as “non-existing” as defined in section 8.2.5.2 of the ITU-T H.264 Specification; -
all other members are interpreted as defined in section 8.2 of the ITU-T H.264 Specification.
-
42.12.7. H.264 Decode Requirements
This section describes the required H.264 decoding capabilities for
physical devices that have at least one queue family that supports the video
codec operation VK_VIDEO_CODEC_OPERATION_DECODE_H264_BIT_KHR, as
returned by vkGetPhysicalDeviceQueueFamilyProperties2 in
VkQueueFamilyVideoPropertiesKHR::videoCodecOperations.
| Video Std Header Name | Version |
|---|---|
|
1.0.0 |
| Video Capability | Requirement | Requirement Type1 |
|---|---|---|
|
- |
min |
|
4096 |
max |
|
4096 |
max |
|
(64,64) |
max |
|
- |
max |
|
- |
min |
|
0 |
min |
|
0 |
min |
|
|
min |
|
|
min |
|
(0,0) except for profiles using |
implementation-dependent |
- 1
-
The Requirement Type column specifies the requirement is either the minimum value all implementations must support, the maximum value all implementations must support, or the exact value all implementations must support. For bitmasks a minimum value is the least bits all implementations must set, but they may have additional bits set beyond this minimum.
42.13. H.265 Decode Operations
Video decode operations using an H.265 decode profile can be used to decode elementary video stream sequences compliant to the ITU-T H.265 Specification.
|
Note
Refer to the Preamble for information on how the Khronos Intellectual Property Rights Policy relates to normative references to external materials not created by Khronos. |
This process is performed according to the video decode operation steps with the codec-specific semantics defined in section 8 of ITU-T H.265 Specification:
-
Syntax elements, derived values, and other parameters are applied from the following structures:
-
The
StdVideoH265VideoParameterSetstructure corresponding to the active VPS specifying the H.265 video parameter set. -
The
StdVideoH265SequenceParameterSetstructure corresponding to the active SPS specifying the H.265 sequence parameter set. -
The
StdVideoH265PictureParameterSetstructure corresponding to the active PPS specifying the H.265 picture parameter set. -
The
StdVideoDecodeH265PictureInfostructure specifying the H.265 picture information. -
The
StdVideoDecodeH265ReferenceInfostructures specifying the H.265 reference information corresponding to the optional reconstructed picture and any active reference pictures.
-
-
The contents of the provided video bitstream buffer range are interpreted as defined in the H.265 Decode Bitstream Data Access section.
-
Picture data in the video picture resources corresponding to the used active reference pictures, decode output picture, and optional reconstructed picture is accessed as defined in the H.265 Decode Picture Data Access section.
-
The decision on reference picture setup is made according to the parameters specified in the H.265 picture information.
If the parameters and the bitstream adhere to the syntactic and semantic requirements defined in the corresponding sections of the ITU-T H.265 Specification, as described above, and the DPB slots associated with the active reference pictures all refer to valid picture references, then the video decode operation will complete successfully. Otherwise, the video decode operation may complete unsuccessfully.
42.13.1. H.265 Decode Bitstream Data Access
The video bitstream buffer range should contain a VCL NAL unit comprised of the slice segment headers and data of a picture representing a frame, as defined in sections 7.3.6 and 7.3.8, and this data is interpreted as defined in sections 7.4.7 and 7.4.9 of the ITU-T H.265 Specification, respectively.
The offsets provided in
VkVideoDecodeH265PictureInfoKHR::pSliceSegmentOffsets should
specify the starting offsets corresponding to each slice segment header
within the video bitstream buffer range.
42.13.2. H.265 Decode Picture Data Access
Accesses to image data within a video picture resource happen at the
granularity indicated by
VkVideoCapabilitiesKHR::pictureAccessGranularity, as returned by
vkGetPhysicalDeviceVideoCapabilitiesKHR for the used video profile.
Accordingly, the complete image subregion of a
decode output picture,
reference picture, or
reconstructed picture accessed by video coding
operations using an H.265 decode profile is defined
as the set of texels within the coordinate range:
-
([0,
endX),[0,endY))
Where:
-
endXequalscodedExtent.widthrounded up to the nearest integer multiple ofpictureAccessGranularity.widthand clamped to the width of the image subresource referred to by the corresponding VkVideoPictureResourceInfoKHR structure; -
endY equals
codedExtent.heightrounded up to the nearest integer multiple ofpictureAccessGranularity.heightand clamped to the height of the image subresource referred to by the corresponding VkVideoPictureResourceInfoKHR structure;
Where codedExtent is the member of the
VkVideoPictureResourceInfoKHR structure corresponding to the picture.
In case of video decode operations using an H.265
decode profile, any access to a picture at the coordinates
(x,y), as defined by the ITU-T H.265
Specification, is an access to the image subresource
referred to by the corresponding
VkVideoPictureResourceInfoKHR structure at the texel coordinates
(x,y).
42.13.3. H.265 Decode Profile
A video profile supporting H.265 video decode operations is specified by
setting VkVideoProfileInfoKHR::videoCodecOperation to
VK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR and adding a
VkVideoDecodeH265ProfileInfoKHR structure to the
VkVideoProfileInfoKHR::pNext chain.
The VkVideoDecodeH265ProfileInfoKHR structure is defined as:
// Provided by VK_KHR_video_decode_h265
typedef struct VkVideoDecodeH265ProfileInfoKHR {
VkStructureType sType;
const void* pNext;
StdVideoH265ProfileIdc stdProfileIdc;
} VkVideoDecodeH265ProfileInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
stdProfileIdcis aStdVideoH265ProfileIdcvalue specifying the H.265 codec profile IDC, as defined in section A.3 of the ITU-T H.265 Specification.
42.13.4. H.265 Decode Capabilities
When calling vkGetPhysicalDeviceVideoCapabilitiesKHR to query the
capabilities for an H.265 decode profile, the
VkVideoCapabilitiesKHR::pNext chain must include a
VkVideoDecodeH265CapabilitiesKHR structure that will be filled with
the profile-specific capabilities.
The VkVideoDecodeH265CapabilitiesKHR structure is defined as:
// Provided by VK_KHR_video_decode_h265
typedef struct VkVideoDecodeH265CapabilitiesKHR {
VkStructureType sType;
void* pNext;
StdVideoH265LevelIdc maxLevelIdc;
} VkVideoDecodeH265CapabilitiesKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
maxLevelIdcis aStdVideoH265LevelIdcvalue indicating the maximum H.265 level supported by the profile, where enum constantSTD_VIDEO_H265_LEVEL_IDC_<major>_<minor>identifies H.265 level<major>.<minor>as defined in section A.4 of the ITU-T H.265 Specification.
42.13.5. H.265 Decode Parameter Sets
Video session parameters objects created with
the video codec operation VK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR
can contain the following types of parameters:
- H.265 Video Parameter Sets (VPS)
-
Represented by
StdVideoH265VideoParameterSetstructures and interpreted as follows:-
reserved1,reserved2, andreserved3are used only for padding purposes and are otherwise ignored; -
vps_video_parameter_set_idis used as the key of the VPS entry; -
the
max_latency_increase_plus1,max_dec_pic_buffering_minus1, andmax_num_reorder_picsmembers of theStdVideoH265DecPicBufMgrstructure pointed to bypDecPicBufMgrcorrespond tovps_max_latency_increase_plus1,vps_max_dec_pic_buffering_minus1, andvps_max_num_reorder_pics, respectively, as defined in section 7.4.3.1 of the ITU-T H.265 Specification; -
the
StdVideoH265HrdParametersstructure pointed to bypHrdParametersis interpreted as follows:-
reservedis used only for padding purposes and is otherwise ignored; -
flags.fixed_pic_rate_general_flagis a bitmask where bit index i corresponds tofixed_pic_rate_general_flag[i]as defined in section E.3.2 of the ITU-T H.265 Specification; -
flags.fixed_pic_rate_within_cvs_flagis a bitmask where bit index i corresponds tofixed_pic_rate_within_cvs_flag[i]as defined in section E.3.2 of the ITU-T H.265 Specification; -
flags.low_delay_hrd_flagis a bitmask where bit index i corresponds tolow_delay_hrd_flag[i]as defined in section E.3.2 of the ITU-T H.265 Specification; -
if
flags.nal_hrd_parameters_present_flagis set, thenpSubLayerHrdParametersNalis a pointer to an array ofvps_max_sub_layers_minus1+ 1 number ofStdVideoH265SubLayerHrdParametersstructures wherevps_max_sub_layers_minus1is the corresponding member of the encompassingStdVideoH265VideoParameterSetstructure and each element is interpreted as follows:-
cbr_flagis a bitmask where bit index i corresponds tocbr_flag[i]as defined in section E.3.3 of the ITU-T H.265 Specification; -
all other members of the
StdVideoH265SubLayerHrdParametersstructure are interpreted as defined in section E.3.3 of the ITU-T H.265 Specification;
-
-
if
flags.vcl_hrd_parameters_present_flagis set, thenpSubLayerHrdParametersVclis a pointer to an array ofvps_max_sub_layers_minus1+ 1 number ofStdVideoH265SubLayerHrdParametersstructures wherevps_max_sub_layers_minus1is the corresponding member of the encompassingStdVideoH265VideoParameterSetstructure and each element is interpreted as follows:-
cbr_flagis a bitmask where bit index i corresponds tocbr_flag[i]as defined in section E.3.3 of the ITU-T H.265 Specification; -
all other members of the
StdVideoH265SubLayerHrdParametersstructure are interpreted as defined in section E.3.3 of the ITU-T H.265 Specification;
-
-
all other members of
StdVideoH265HrdParametersare interpreted as defined in section E.3.2 of the ITU-T H.265 Specification;
-
-
the
StdVideoH265ProfileTierLevelstructure pointed to bypProfileTierLevelare interpreted as follows:-
general_level_idcis one of the enum constantsSTD_VIDEO_H265_LEVEL_IDC_<major>_<minor>identifying the H.265 level<major>.<minor>as defined in section A.4 of the ITU-T H.265 Specification; -
all other members of
StdVideoH265ProfileTierLevelare interpreted as defined in section 7.4.4 of the ITU-T H.265 Specification;
-
-
all other members of
StdVideoH265VideoParameterSetare interpreted as defined in section 7.4.3.1 of the ITU-T H.265 Specification.
-
- H.265 Sequence Parameter Sets (SPS)
-
Represented by
StdVideoH265SequenceParameterSetstructures and interpreted as follows:-
reserved1andreserved2are used only for padding purposes and are otherwise ignored; -
the pair constructed from
sps_video_parameter_set_idandsps_seq_parameter_set_idis used as the key of the SPS entry; -
the
StdVideoH265ProfileTierLevelstructure pointed to bypProfileTierLevelare interpreted as follows:-
general_level_idcis one of the enum constantsSTD_VIDEO_H265_LEVEL_IDC_<major>_<minor>identifying the H.265 level<major>.<minor>as defined in section A.4 of the ITU-T H.265 Specification; -
all other members of
StdVideoH265ProfileTierLevelare interpreted as defined in section 7.4.4 of the ITU-T H.265 Specification;
-
-
the
max_latency_increase_plus1,max_dec_pic_buffering_minus1, andmax_num_reorder_picsmembers of theStdVideoH265DecPicBufMgrstructure pointed to bypDecPicBufMgrcorrespond tosps_max_latency_increase_plus1,sps_max_dec_pic_buffering_minus1, andsps_max_num_reorder_pics, respectively, as defined in section 7.4.3.2 of the ITU-T H.265 Specification; -
if
flags.sps_scaling_list_data_present_flagis set, then theStdVideoH265ScalingListsstructure pointed to bypScalingListsis interpreted as follows:-
ScalingList4x4,ScalingList8x8,ScalingList16x16, andScalingList32x32correspond toScalingList[0],ScalingList[1],ScalingList[2], andScalingList[3], respectively, as defined in section 7.3.4 of the ITU-T H.265 Specification; -
ScalingListDCCoef16x16andScalingListDCCoef32x32correspond toscaling_list_dc_coef_minus8[0]andscaling_list_dc_coef_minus8[1], respectively, as defined in section 7.3.4 of the ITU-T H.265 Specification;
-
-
pShortTermRefPicSetis a pointer to an array ofnum_short_term_ref_pic_setsnumber ofStdVideoH265ShortTermRefPicSetstructures where each element is interpreted as follows:-
reserved1,reserved2, andreserved3are used only for padding purposes and are otherwise ignored; -
used_by_curr_pic_flagis a bitmask where bit index i corresponds toused_by_curr_pic_flag[i]as defined in section 7.4.8 of the ITU-T H.265 Specification; -
use_delta_flagis a bitmask where bit index i corresponds touse_delta_flag[i]as defined in section 7.4.8 of the ITU-T H.265 Specification; -
used_by_curr_pic_s0_flagis a bitmask where bit index i corresponds toused_by_curr_pic_s0_flag[i]as defined in section 7.4.8 of the ITU-T H.265 Specification; -
used_by_curr_pic_s1_flagis a bitmask where bit index i corresponds toused_by_curr_pic_s1_flag[i]as defined in section 7.4.8 of the ITU-T H.265 Specification; -
all other members of
StdVideoH265ShortTermRefPicSetare interpreted as defined in section 7.4.8 of the ITU-T H.265 Specification;
-
-
if
flags.long_term_ref_pics_present_flagis set then theStdVideoH265LongTermRefPicsSpsstructure pointed to bypLongTermRefPicsSpsis interpreted as follows:-
used_by_curr_pic_lt_sps_flagis a bitmask where bit index i corresponds toused_by_curr_pic_lt_sps_flag[i]as defined in section 7.4.3.2 of the ITU-T H.265 Specification; -
all other members of
StdVideoH265LongTermRefPicsSpsare interpreted as defined in section 7.4.3.2 of the ITU-T H.265 Specification;
-
-
if
flags.vui_parameters_present_flagis set, then theStdVideoH265SequenceParameterSetVuistructure pointed to bypSequenceParameterSetVuiis interpreted as follows:-
reserved1,reserved2, andreserved3are used only for padding purposes and are otherwise ignored; -
the
StdVideoH265HrdParametersstructure pointed to bypHrdParametersis interpreted as follows:-
flags.fixed_pic_rate_general_flagis a bitmask where bit index i corresponds tofixed_pic_rate_general_flag[i]as defined in section E.3.2 of the ITU-T H.265 Specification; -
flags.fixed_pic_rate_within_cvs_flagis a bitmask where bit index i corresponds tofixed_pic_rate_within_cvs_flag[i]as defined in section E.3.2 of the ITU-T H.265 Specification; -
flags.low_delay_hrd_flagis a bitmask where bit index i corresponds tolow_delay_hrd_flag[i]as defined in section E.3.2 of the ITU-T H.265 Specification; -
if
flags.nal_hrd_parameters_present_flagis set, thenpSubLayerHrdParametersNalis a pointer to an array ofsps_max_sub_layers_minus1+ 1 number ofStdVideoH265SubLayerHrdParametersstructures wheresps_max_sub_layers_minus1is the corresponding member of the encompassingStdVideoH265SequenceParameterSetstructure and each element is interpreted as follows:-
cbr_flagis a bitmask where bit index i corresponds tocbr_flag[i]as defined in section E.3.3 of the ITU-T H.265 Specification; -
all other members of the
StdVideoH265SubLayerHrdParametersstructure are interpreted as defined in section E.3.3 of the ITU-T H.265 Specification;
-
-
if
flags.vcl_hrd_parameters_present_flagis set, thenpSubLayerHrdParametersVclis a pointer to an array ofsps_max_sub_layers_minus1+ 1 number ofStdVideoH265SubLayerHrdParametersstructures wheresps_max_sub_layers_minus1is the corresponding member of the encompassingStdVideoH265SequenceParameterSetstructure and each element is interpreted as follows:-
cbr_flagis a bitmask where bit index i corresponds tocbr_flag[i]as defined in section E.3.3 of the ITU-T H.265 Specification; -
all other members of the
StdVideoH265SubLayerHrdParametersstructure are interpreted as defined in section E.3.3 of the ITU-T H.265 Specification;
-
-
all other members of
StdVideoH265HrdParametersare interpreted as defined in section E.3.2 of the ITU-T H.265 Specification;
-
-
all other members of
pSequenceParameterSetVuiare interpreted as defined in section E.3.1 of the ITU-T H.265 Specification;
-
-
if
flags.sps_palette_predictor_initializer_present_flagis set, then thePredictorPaletteEntriesmember of theStdVideoH265PredictorPaletteEntriesstructure pointed to bypPredictorPaletteEntriesis interpreted as defined in section 7.4.9.13 of the ITU-T H.265 Specification; -
all other members of
StdVideoH265SequenceParameterSetare interpreted as defined in section 7.4.3.1 of the ITU-T H.265 Specification.
-
- H.265 Picture Parameter Sets (PPS)
-
Represented by
StdVideoH265PictureParameterSetstructures and interpreted as follows:-
reserved1,reserved2, andreserved3are used only for padding purposes and are otherwise ignored; -
the triplet constructed from
sps_video_parameter_set_id,pps_seq_parameter_set_id, andpps_pic_parameter_set_idis used as the key of the PPS entry; -
if
flags.pps_scaling_list_data_present_flagis set, then theStdVideoH265ScalingListsstructure pointed to bypScalingListsis interpreted as follows:-
ScalingList4x4,ScalingList8x8,ScalingList16x16, andScalingList32x32correspond toScalingList[0],ScalingList[1],ScalingList[2], andScalingList[3], respectively, as defined in section 7.3.4 of the ITU-T H.265 Specification; -
ScalingListDCCoef16x16andScalingListDCCoef32x32correspond toscaling_list_dc_coef_minus8[0]andscaling_list_dc_coef_minus8[1], respectively, as defined in section 7.3.4 of the ITU-T H.265 Specification;
-
-
if
flags.pps_palette_predictor_initializer_present_flagis set, then thePredictorPaletteEntriesmember of theStdVideoH265PredictorPaletteEntriesstructure pointed to bypPredictorPaletteEntriesis interpreted as defined in section 7.4.9.13 of the ITU-T H.265 Specification; -
all other members of
StdVideoH265PictureParameterSetare interpreted as defined in section 7.4.3.3 of the ITU-T H.265 Specification.
-
When a video session parameters object is
created with the codec operation
VK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR, the
VkVideoSessionParametersCreateInfoKHR::pNext chain must include
a VkVideoDecodeH265SessionParametersCreateInfoKHR structure specifying
the capacity and initial contents of the object.
The VkVideoDecodeH265SessionParametersCreateInfoKHR structure is
defined as:
// Provided by VK_KHR_video_decode_h265
typedef struct VkVideoDecodeH265SessionParametersCreateInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t maxStdVPSCount;
uint32_t maxStdSPSCount;
uint32_t maxStdPPSCount;
const VkVideoDecodeH265SessionParametersAddInfoKHR* pParametersAddInfo;
} VkVideoDecodeH265SessionParametersCreateInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
maxStdVPSCountis the maximum number of H.265 VPS entries the createdVkVideoSessionParametersKHRcan contain. -
maxStdSPSCountis the maximum number of H.265 SPS entries the createdVkVideoSessionParametersKHRcan contain. -
maxStdPPSCountis the maximum number of H.265 PPS entries the createdVkVideoSessionParametersKHRcan contain. -
pParametersAddInfoisNULLor a pointer to a VkVideoDecodeH265SessionParametersAddInfoKHR structure specifying H.265 parameters to add upon object creation.
The VkVideoDecodeH265SessionParametersAddInfoKHR structure is defined
as:
// Provided by VK_KHR_video_decode_h265
typedef struct VkVideoDecodeH265SessionParametersAddInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t stdVPSCount;
const StdVideoH265VideoParameterSet* pStdVPSs;
uint32_t stdSPSCount;
const StdVideoH265SequenceParameterSet* pStdSPSs;
uint32_t stdPPSCount;
const StdVideoH265PictureParameterSet* pStdPPSs;
} VkVideoDecodeH265SessionParametersAddInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
stdVPSCountis the number of elements in thepStdVPSsarray. -
pStdVPSsis a pointer to an array ofStdVideoH265VideoParameterSetstructures describing the H.265 VPS entries to add. -
stdSPSCountis the number of elements in thepStdSPSsarray. -
pStdSPSsis a pointer to an array ofStdVideoH265SequenceParameterSetstructures describing the H.265 SPS entries to add. -
stdPPSCountis the number of elements in thepStdPPSsarray. -
pStdPPSsis a pointer to an array ofStdVideoH265PictureParameterSetstructures describing the H.265 PPS entries to add.
This structure can be specified in the following places:
-
In the
pParametersAddInfomember of the VkVideoDecodeH265SessionParametersCreateInfoKHR structure specified in thepNextchain of VkVideoSessionParametersCreateInfoKHR used to create a video session parameters object. In this case, if the video codec operation the video session parameters object is created with isVK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR, then it defines the set of initial parameters to add to the created object (see Creating Video Session Parameters). -
In the
pNextchain of VkVideoSessionParametersUpdateInfoKHR. In this case, if the video codec operation the video session parameters object to be updated was created with isVK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR, then it defines the set of parameters to add to it (see Updating Video Session Parameters).
42.13.6. H.265 Decoding Parameters
The VkVideoDecodeH265PictureInfoKHR structure is defined as:
// Provided by VK_KHR_video_decode_h265
typedef struct VkVideoDecodeH265PictureInfoKHR {
VkStructureType sType;
const void* pNext;
const StdVideoDecodeH265PictureInfo* pStdPictureInfo;
uint32_t sliceSegmentCount;
const uint32_t* pSliceSegmentOffsets;
} VkVideoDecodeH265PictureInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
pStdPictureInfois a pointer to aStdVideoDecodeH265PictureInfostructure specifying H.265 picture information. -
sliceSegmentCountis the number of elements inpSliceSegmentOffsets. -
pSliceSegmentOffsetsis a pointer to an array ofsliceSegmentCountoffsets specifying the start offset of the slice segments of the picture within the video bitstream buffer range specified in VkVideoDecodeInfoKHR.
This structure is specified in the pNext chain of the
VkVideoDecodeInfoKHR structure passed to vkCmdDecodeVideoKHR to
specify the codec-specific picture information for an H.265
decode operation.
- Decode Output Picture Information
-
When this structure is specified in the
pNextchain of the VkVideoDecodeInfoKHR structure passed to vkCmdDecodeVideoKHR, the information related to the decode output picture is defined as follows:-
The image subregion used is determined according to the H.265 Decode Picture Data Access section.
-
The decode output picture is associated with the H.265 picture information provided in
pStdPictureInfo.
-
- Std Picture Information
-
The members of the
StdVideoDecodeH265PictureInfostructure pointed to bypStdPictureInfoare interpreted as follows:-
reservedis used only for padding purposes and is otherwise ignored; -
flags.IrapPicFlagas defined in section 3.73 of the ITU-T H.265 Specification; -
flags.IdrPicFlagas defined in section 3.67 of the ITU-T H.265 Specification; -
flags.IsReferenceas defined in section 3.132 of the ITU-T H.265 Specification; -
sps_video_parameter_set_id,pps_seq_parameter_set_id, andpps_pic_parameter_set_idare used to identify the active parameter sets, as described below; -
PicOrderCntValas defined in section 8.3.1 of the ITU-T H.265 Specification; -
NumBitsForSTRefPicSetInSliceis the number of bits used inst_ref_pic_setwhenshort_term_ref_pic_set_sps_flagis0, or0otherwise, as defined in sections 7.4.7 and 7.4.8 of the ITU-T H.265 Specification; -
NumDeltaPocsOfRefRpsIdxis the value ofNumDeltaPocs[RefRpsIdx]whenshort_term_ref_pic_set_sps_flagis1, or0otherwise, as defined in sections 7.4.7 and 7.4.8 of the ITU-T H.265 Specification; -
RefPicSetStCurrBefore,RefPicSetStCurrAfter, andRefPicSetLtCurrare interpreted as defined in section 8.3.2 of the ITU-T H.265 Specification where each element of these arrays either identifies an active reference picture using its DPB slot index or contains the valueSTD_VIDEO_H265_NO_REFERENCE_PICTUREto indicate “no reference picture”; -
all other members are interpreted as defined in section 8.3.2 of the ITU-T H.265 Specification.
-
Reference picture setup is controlled by the value of
StdVideoDecodeH265PictureInfo::flags.IsReference.
If it is set and a reconstructed
picture is specified, then the latter is used as the target of picture
reconstruction to activate the corresponding
DPB slot.
If StdVideoDecodeH265PictureInfo::flags.IsReference is not set,
but a reconstructed picture is
specified, then the corresponding picture reference associated with the
DPB slot is invalidated, as described in the
DPB Slot States section.
- Active Parameter Sets
-
The members of the
StdVideoDecodeH265PictureInfostructure pointed to bypStdPictureInfoare used to select the active parameter sets to use from the bound video session parameters object, as follows:-
The active VPS is the VPS identified by the key specified in
StdVideoDecodeH265PictureInfo::sps_video_parameter_set_id. -
The active SPS is the SPS identified by the key specified by the pair constructed from
StdVideoDecodeH265PictureInfo::sps_video_parameter_set_idandStdVideoDecodeH265PictureInfo::pps_seq_parameter_set_id. -
The active PPS is the PPS identified by the key specified by the triplet constructed from
StdVideoDecodeH265PictureInfo::sps_video_parameter_set_id,StdVideoDecodeH265PictureInfo::pps_seq_parameter_set_id, andStdVideoDecodeH265PictureInfo::pps_pic_parameter_set_id.
-
The VkVideoDecodeH265DpbSlotInfoKHR structure is defined as:
// Provided by VK_KHR_video_decode_h265
typedef struct VkVideoDecodeH265DpbSlotInfoKHR {
VkStructureType sType;
const void* pNext;
const StdVideoDecodeH265ReferenceInfo* pStdReferenceInfo;
} VkVideoDecodeH265DpbSlotInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
pStdReferenceInfois a pointer to aStdVideoDecodeH265ReferenceInfostructure specifying reference picture information described in section 8.3 of the ITU-T H.265 Specification.
This structure is specified in the pNext chain of
VkVideoDecodeInfoKHR::pSetupReferenceSlot, if not NULL, and
the pNext chain of the elements of
VkVideoDecodeInfoKHR::pReferenceSlots to specify the
codec-specific reference picture information for an H.265
decode operation.
- Active Reference Picture Information
-
When this structure is specified in the
pNextchain of the elements of VkVideoDecodeInfoKHR::pReferenceSlots, one element is added to the list of active reference pictures used by the video decode operation for each element of VkVideoDecodeInfoKHR::pReferenceSlotsas follows:-
The image subregion used is determined according to the H.265 Decode Picture Data Access section.
-
The reference picture is associated with the DPB slot index specified in the
slotIndexmember of the corresponding element of VkVideoDecodeInfoKHR::pReferenceSlots. -
The reference picture is associated with the H.265 reference information provided in
pStdReferenceInfo.
-
- Reconstructed Picture Information
-
When this structure is specified in the
pNextchain of VkVideoDecodeInfoKHR::pSetupReferenceSlot, the information related to the reconstructed picture is defined as follows:-
The image subregion used is determined according to the H.265 Decode Picture Data Access section.
-
If reference picture setup is requested, then the reconstructed picture is used to activate the DPB slot with the index specified in VkVideoDecodeInfoKHR::
pSetupReferenceSlot->slotIndex. -
The reconstructed picture is associated with the H.265 reference information provided in
pStdReferenceInfo.
-
- Std Reference Information
-
The members of the
StdVideoDecodeH265ReferenceInfostructure pointed to bypStdReferenceInfoare interpreted as follows:-
flags.used_for_long_term_referenceis used to indicate whether the picture is marked as “used for long-term reference” as defined in section 8.3.2 of the ITU-T H.265 Specification; -
flags.unused_for_referenceis used to indicate whether the picture is marked as “unused for reference” as defined in section 8.3.2 of the ITU-T H.265 Specification; -
all other members are interpreted as defined in section 8.3 of the ITU-T H.265 Specification.
-
42.13.7. H.265 Decode Requirements
This section describes the required H.265 decoding capabilities for
physical devices that have at least one queue family that supports the video
codec operation VK_VIDEO_CODEC_OPERATION_DECODE_H265_BIT_KHR, as
returned by vkGetPhysicalDeviceQueueFamilyProperties2 in
VkQueueFamilyVideoPropertiesKHR::videoCodecOperations.
| Video Std Header Name | Version |
|---|---|
|
1.0.0 |
| Video Capability | Requirement | Requirement Type1 |
|---|---|---|
|
- |
min |
|
4096 |
max |
|
4096 |
max |
|
(64,64) |
max |
|
- |
max |
|
- |
min |
|
0 |
min |
|
0 |
min |
|
|
min |
|
|
min |
- 1
-
The Requirement Type column specifies the requirement is either the minimum value all implementations must support, the maximum value all implementations must support, or the exact value all implementations must support. For bitmasks a minimum value is the least bits all implementations must set, but they may have additional bits set beyond this minimum.
42.14. Video Encode Operations
Video encode operations consume an encode input picture and zero or more reference pictures, and produce compressed video data to a video bitstream buffer and an optional reconstructed picture.
|
Note
Such encode input pictures can be used as the output of video decode operations, with graphics or compute operations, or with Window System Integration APIs, depending on the capabilities of the implementation. |
Video encode operations may access the following resources in the
VK_PIPELINE_STAGE_2_VIDEO_ENCODE_BIT_KHR stage:
-
The image subregions corresponding to the source encode input picture and active reference pictures with access
VK_ACCESS_2_VIDEO_ENCODE_READ_BIT_KHR. -
The destination video bitstream buffer range and the optional reconstructed picture with access
VK_ACCESS_2_VIDEO_ENCODE_WRITE_BIT_KHR.
The image subresource of each video picture resource accessed by the video encode operation is specified using a corresponding VkVideoPictureResourceInfoKHR structure. Each such image subresource must be in the appropriate image layout as follows:
-
If the image subresource is used in the video encode operation as an encode input picture, then it must be in the
VK_IMAGE_LAYOUT_VIDEO_ENCODE_SRC_KHRlayout. -
If the image subresource is used in the video encode operation as a reconstructed picture or reference picture, then it must be in the
VK_IMAGE_LAYOUT_VIDEO_ENCODE_DPB_KHRlayout.
A video encode operation may complete unsuccessfully. In this case the target video bitstream buffer will have undefined contents. Similarly, if reference picture setup is requested, the reconstructed-picture will also have undefined contents, and the activated DPB slot will have an invalid picture reference.
If a video encode operation completes successfully and the codec-specific parameters provided by the application adhere to the syntactic and semantic requirements defined in the corresponding video compression standard, then the target video bitstream buffer will contain compressed video data after the execution of the video encode operation according to the respective codec-specific semantics.
42.14.1. Codec-Specific Semantics
The following aspects of video encode operations are codec-specific:
-
The compressed video data written to the target video bitstream buffer range.
-
The construction and interpretation of the list of active reference pictures and the interpretation of the picture data referred to by the corresponding image subregions.
-
The construction and interpretation of information related to the encode input picture and the interpretation of the picture data referred to by the corresponding image subregion.
-
The decision on reference picture setup.
-
The construction and interpretation of information related to the optional reconstructed picture and the generation of picture data to the corresponding image subregion.
-
Certain aspects of rate control.
These codec-specific behaviors are defined for each video codec operation separately.
-
If the used video codec operation is
VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR, then the codec-specific aspects of the video encoding process are performed as defined in the H.264 Encode Operations section. -
If the used video codec operation is
VK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR, then the codec-specific aspects of the video encoding process are performed as defined in the H.265 Encode Operations section.
Video Encode Parameter Overrides
Implementations supporting video encode operations for any particular video codec operation often support only a subset of the available encoding tools defined by the corresponding video compression standards. Accordingly, certain implementation-dependent limitations may apply to codec-specific parameters provided through the structures defined in the Video Std headers corresponding to the used video codec operation.
Exposing all of these restrictions on particular codec-specific parameter values or combinations thereof in the form of application-queryable capabilities is impractical, hence this specification allows implementations to override the value of any of the codec-specific parameters, unless otherwise specified, as long as all of the following conditions are met:
-
If the application-provided codec-specific parameters adhere to the syntactic and semantic requirements and rules defined by the used video compression standard, and thus would be usable to produce a video bitstream compliant with that standard, then the codec-specific parameters resulting from the process of implementation overrides must also adhere to the same requirements and rules, and any video bitstream produced using the overridden parameters must also be compliant.
-
The overridden codec-specific parameter values must not have an impact on the codec-independent behaviors defined for video encode operations.
-
The implementation must not override any codec-specific parameters specified to a command that may cause application-provided codec-specific parameters specified to subsequent commands to no longer adhere to the semantic requirements and rules defined by the used video compression standard, unless the implementation also overrides those parameters to adhere to any such requirements and rules.
-
The overridden codec-specific parameter values must not have an impact on the codec-specific picture data access semantics.
-
The overridden codec-specific parameter values may change the contents of the codec-specific bitstream elements produced by video encode operations or otherwise retrieved by the application (e.g. using the vkGetEncodedVideoSessionParametersKHR command) but must still adhere to the codec-specific semantics defined for that video codec operation, including, but not limited to, the number, type, and order of the encoded codec-specific bitstream elements.
Besides codec-specific parameter overrides performed for implementation-dependent reasons, applications can enable the implementation to apply additional optimizing overrides that may improve the efficiency or performance of video encoding operations. However, implementations must meet the conditions listed above even in case of such optimizing overrides.
|
Note
Unless the application opts in for optimizing overrides, implementations are not expected to override any of the codec-specific parameters, except when such overrides are necessary for the correct operation of video encoder implementation due to limitations to the available encoding tools on that implementation. |
42.14.2. Video Encode Operation Steps
Each video encode operation performs the following steps in the
VK_PIPELINE_STAGE_2_VIDEO_ENCODE_BIT_KHR stage:
-
Reads the input picture data from the encode input picture;
-
Determine derived encoding quality parameters according to the codec-specific semantics and the current rate control state;
-
Compresses the input picture data according to the codec-specific semantics, applying any prediction data read from the active reference pictures and rate control restrictions in the process;
-
Writes the encoded bitstream data to the destination video bitstream buffer range;
-
Performs picture reconstruction of the encoded video data according to the codec-specific semantics, applying any prediction data read from the active reference pictures in the process, if a reconstructed picture is specified and reference picture setup is requested;
-
If reference picture setup is requested, the DPB slot index specified in the reconstructed picture information is activated with the reconstructed picture;
-
Writes the reconstructed picture data to the reconstructed picture, if one is specified, according to the codec-specific semantics.
When reconstructed picture information is provided, the specified DPB slot index is associated with the corresponding bound reference picture resource, indifferent of whether reference picture setup is requested.
42.14.3. Capabilities
When calling vkGetPhysicalDeviceVideoCapabilitiesKHR with
pVideoProfile->videoCodecOperation specifying an encode operation, the
VkVideoEncodeCapabilitiesKHR structure must be included in the
pNext chain of the VkVideoCapabilitiesKHR structure to retrieve
capabilities specific to video encoding.
The VkVideoEncodeCapabilitiesKHR structure is defined as:
// Provided by VK_KHR_video_encode_queue
typedef struct VkVideoEncodeCapabilitiesKHR {
VkStructureType sType;
void* pNext;
VkVideoEncodeCapabilityFlagsKHR flags;
VkVideoEncodeRateControlModeFlagsKHR rateControlModes;
uint32_t maxRateControlLayers;
uint64_t maxBitrate;
uint32_t maxQualityLevels;
VkExtent2D encodeInputPictureGranularity;
VkVideoEncodeFeedbackFlagsKHR supportedEncodeFeedbackFlags;
} VkVideoEncodeCapabilitiesKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
flagsis a bitmask of VkVideoEncodeCapabilityFlagBitsKHR describing supported encoding features. -
rateControlModesis a bitmask of VkVideoEncodeRateControlModeFlagBitsKHR indicating supported rate control modes. -
maxRateControlLayersindicates the maximum number of rate control layers supported. -
maxBitrateindicates the maximum supported bitrate. -
maxQualityLevelsindicates the number of discrete video encode quality levels supported. Implementations must support at least one quality level. -
encodeInputPictureGranularityindicates the granularity at which encode input picture data is encoded and may indicate a texel granularity up to the size of the codec-specific coding block size. This capability does not impose any valid usage constraints on the application, however, depending on the contents of the encode input picture, it may have effects on the encoded bitstream, as described in more detail below. -
supportedEncodeFeedbackFlagsis a bitmask of VkVideoEncodeFeedbackFlagBitsKHR values specifying the supported flags for video encode feedback queries.
Implementations must include support for at least
VK_VIDEO_ENCODE_FEEDBACK_BITSTREAM_BUFFER_OFFSET_BIT_KHR and
VK_VIDEO_ENCODE_FEEDBACK_BITSTREAM_BYTES_WRITTEN_BIT_KHR in
supportedEncodeFeedbackFlags.
encodeInputPictureGranularity provides information about the way
encode input picture data is used as input to video
encode operations.
In particular, some implementations may not be able to limit the set of
texels used to encode the output video bitstream to the image subregion
specified in the VkVideoPictureResourceInfoKHR structure corresponding
to the encode input picture (i.e. to the resolution of the image data to
encode specified in its codedExtent member).
|
Note
For example, the application requests the coded extent to be 1920x1080, but
the implementation is only able to source the encode input picture data at
the granularity of the codec-specific coding block size which is 16x16
pixels (or as otherwise indicated in |
If codedExtent rounded up to the next integer multiple of
encodeInputPictureGranularity is greater than the extent of the image
subresource specified for the encode input picture,
then the texel values corresponding to texel coordinates outside of the
bounds of the image subresource may be undefined.
However, implementations should use well-defined default values for such
texels in order to maximize the encoding efficiency for the last coding
block row/column, and/or to ensure consistent encoding results across
repeated encoding of the same input content.
Nonetheless, the values used for such texels must not have an effect on
whether the video encode operation produces a compliant bitstream, and must
not have any other effects on the encoded picture data beyond what may
otherwise result from using these texel values as input to any compression
algorithm, as defined in the used video compression standard.
|
Note
While not required, it is generally a good practice for applications to make
sure that the image subresource used for the encode input picture has an
extent that is an integer multiple of the codec-specific coding block size
(or at least |
Bits which may be set in VkVideoEncodeCapabilitiesKHR::flags,
indicating the encoding tools supported, are:
// Provided by VK_KHR_video_encode_queue
typedef enum VkVideoEncodeCapabilityFlagBitsKHR {
VK_VIDEO_ENCODE_CAPABILITY_PRECEDING_EXTERNALLY_ENCODED_BYTES_BIT_KHR = 0x00000001,
VK_VIDEO_ENCODE_CAPABILITY_INSUFFICIENT_BITSTREAM_BUFFER_RANGE_DETECTION_BIT_KHR = 0x00000002,
} VkVideoEncodeCapabilityFlagBitsKHR;
-
VK_VIDEO_ENCODE_CAPABILITY_PRECEDING_EXTERNALLY_ENCODED_BYTES_BIT_KHRindicates that the implementation supports the use of VkVideoEncodeInfoKHR::precedingExternallyEncodedBytes. -
VK_VIDEO_ENCODE_CAPABILITY_INSUFFICIENT_BITSTREAM_BUFFER_RANGE_DETECTION_BIT_KHRindicates that the implementation is able to detect and report when the destination video bitstream buffer range provided by the application is not sufficiently large to fit the encoded bitstream data produced by a video encode operation by reporting theVK_QUERY_RESULT_STATUS_INSUFFICIENT_BITSTREAM_BUFFER_RANGE_KHRquery result status code.NoteSome implementations may not be able to reliably detect insufficient bitstream buffer range conditions in all situations. Such implementations will not report support for the
VK_VIDEO_ENCODE_CAPABILITY_INSUFFICIENT_BITSTREAM_BUFFER_RANGE_DETECTION_BIT_KHRencode capability flag for the video profile, but may still report theVK_QUERY_RESULT_STATUS_INSUFFICIENT_BITSTREAM_BUFFER_RANGE_KHRquery result status code in certain cases. Applications should always check for the specific query result status codeVK_QUERY_RESULT_STATUS_INSUFFICIENT_BITSTREAM_BUFFER_RANGE_KHReven when this encode capability flag is not supported by the implementation for the video profile in question. However, applications must not assume that a different negative query result status code indicating an unsuccessful completion of a video encode operation is not the result of an insufficient bitstream buffer condition unless this encode capability flag is supported.
// Provided by VK_KHR_video_encode_queue
typedef VkFlags VkVideoEncodeCapabilityFlagsKHR;
VkVideoEncodeCapabilityFlagsKHR is a bitmask type for setting a mask
of zero or more VkVideoEncodeCapabilityFlagBitsKHR.
42.14.4. Video Encode Quality Levels
Implementations can support more than one video encode quality levels for a video encode profile, which control the number and type of implementation-specific encoding tools and algorithms utilized in the encoding process.
|
Note
Generally, using higher video encode quality levels may produce higher quality video streams at the cost of additional processing time. However, as the final quality of an encoded picture depends on the contents of the encode input picture, the contents of the active reference pictures, the codec-specific encode parameters, and the particular implementation-specific tools used corresponding to the individual video encode quality levels, there are no guarantees that using a higher video encode quality level will always produce a higher quality encoded picture for any given set of inputs. |
To query properties for a specific video encode quality level supported by a video encode profile, call:
// Provided by VK_KHR_video_encode_queue
VkResult vkGetPhysicalDeviceVideoEncodeQualityLevelPropertiesKHR(
VkPhysicalDevice physicalDevice,
const VkPhysicalDeviceVideoEncodeQualityLevelInfoKHR* pQualityLevelInfo,
VkVideoEncodeQualityLevelPropertiesKHR* pQualityLevelProperties);
-
physicalDeviceis the physical device to query the video encode quality level properties for. -
pQualityLevelInfois a pointer to a VkPhysicalDeviceVideoEncodeQualityLevelInfoKHR structure specifying the video encode profile and quality level to query properties for. -
pQualityLevelPropertiesis a pointer to a VkVideoEncodeQualityLevelPropertiesKHR structure in which the properties are returned.
The VkPhysicalDeviceVideoEncodeQualityLevelInfoKHR structure is
defined as:
// Provided by VK_KHR_video_encode_queue
typedef struct VkPhysicalDeviceVideoEncodeQualityLevelInfoKHR {
VkStructureType sType;
const void* pNext;
const VkVideoProfileInfoKHR* pVideoProfile;
uint32_t qualityLevel;
} VkPhysicalDeviceVideoEncodeQualityLevelInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
pVideoProfileis a pointer to a VkVideoProfileInfoKHR structure specifying the video profile to query the video encode quality level properties for. -
qualityLevelis the video encode quality level to query properties for.
The VkVideoEncodeQualityLevelPropertiesKHR structure is defined as:
// Provided by VK_KHR_video_encode_queue
typedef struct VkVideoEncodeQualityLevelPropertiesKHR {
VkStructureType sType;
void* pNext;
VkVideoEncodeRateControlModeFlagBitsKHR preferredRateControlMode;
uint32_t preferredRateControlLayerCount;
} VkVideoEncodeQualityLevelPropertiesKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
preferredRateControlModeis a VkVideoEncodeRateControlModeFlagBitsKHR value indicating the preferred rate control mode to use with the video encode quality level. -
preferredRateControlLayerCountindicates the preferred number of rate control layers to use with the video encode quality level.
The VkVideoEncodeQualityLevelInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_queue
typedef struct VkVideoEncodeQualityLevelInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t qualityLevel;
} VkVideoEncodeQualityLevelInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
qualityLevelis the used video encode quality level.
This structure can be specified in the following places:
-
In the
pNextchain of VkVideoSessionParametersCreateInfoKHR to specify the video encode quality level to use for a video session parameters object created for a video encode session. If no instance of this structure is included in thepNextchain of VkVideoSessionParametersCreateInfoKHR, then the video session parameters object is created with a video encode quality level of zero. -
In the
pNextchain of VkVideoCodingControlInfoKHR to change the video encode quality level state of the bound video session.
42.14.5. Retrieving Encoded Session Parameters
Any codec-specific parameters stored in video session parameters objects may need to be separately encoded and included in the final video bitstream data, depending on the used video compression standard. In such cases the application must call the vkGetEncodedVideoSessionParametersKHR command to retrieve the encoded parameter data from the used video session parameters object in order to be able to produce a compliant video bitstream.
|
Note
This is needed because implementations may have changed some of the codec-specific parameters stored in the video session parameters object, as defined in the Video Encode Parameter Overrides section. In addition, the vkGetEncodedVideoSessionParametersKHR command enables the application to retrieve the encoded parameter data without having to encode these codec-specific parameters manually. |
Encoded parameter data can be retrieved from a video session parameters object created with a video encode operation using the command:
// Provided by VK_KHR_video_encode_queue
VkResult vkGetEncodedVideoSessionParametersKHR(
VkDevice device,
const VkVideoEncodeSessionParametersGetInfoKHR* pVideoSessionParametersInfo,
VkVideoEncodeSessionParametersFeedbackInfoKHR* pFeedbackInfo,
size_t* pDataSize,
void* pData);
-
deviceis the logical device that owns the video session parameters object. -
pVideoSessionParametersInfois a pointer to a VkVideoEncodeSessionParametersGetInfoKHR structure specifying the parameters of the encoded parameter data to retrieve. -
pFeedbackInfois eitherNULLor a pointer to a VkVideoEncodeSessionParametersFeedbackInfoKHR structure in which feedback about the requested parameter data is returned. -
pDataSizeis a pointer to asize_tvalue related to the amount of encode parameter data returned, as described below. -
pDatais eitherNULLor a pointer to a buffer to write the encoded parameter data to.
If pData is NULL, then the size of the encoded parameter data, in
bytes, that can be retrieved is returned in pDataSize.
Otherwise, pDataSize must point to a variable set by the application
to the size of the buffer, in bytes, pointed to by pData, and on
return the variable is overwritten with the number of bytes actually written
to pData.
If pDataSize is less than the size of the encoded parameter data that
can be retrieved, then no data will be written to pData, zero will be
written to pDataSize, and VK_INCOMPLETE will be returned instead
of VK_SUCCESS, to indicate that no encoded parameter data was
returned.
If pFeedbackInfo is not NULL then the members of the
VkVideoEncodeSessionParametersFeedbackInfoKHR structure and any
additional structures included in its pNext chain that are applicable
to the video session parameters object specified in
pVideoSessionParametersInfo->videoSessionParameters will be filled
with feedback about the requested parameter data on all successful calls to
this command.
|
Note
This includes the cases when |
The VkVideoEncodeSessionParametersGetInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_queue
typedef struct VkVideoEncodeSessionParametersGetInfoKHR {
VkStructureType sType;
const void* pNext;
VkVideoSessionParametersKHR videoSessionParameters;
} VkVideoEncodeSessionParametersGetInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
videoSessionParametersis the VkVideoSessionParametersKHR object to retrieve encoded parameter data from.
Depending on the used video encode operation, additional codec-specific
structures may need to be included in the pNext chain of this
structure to identify the specific video session parameters to retrieve
encoded parameter data for, as described in the corresponding sections.
The VkVideoEncodeSessionParametersFeedbackInfoKHR structure is defined
as:
// Provided by VK_KHR_video_encode_queue
typedef struct VkVideoEncodeSessionParametersFeedbackInfoKHR {
VkStructureType sType;
void* pNext;
VkBool32 hasOverrides;
} VkVideoEncodeSessionParametersFeedbackInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
hasOverridesindicates whether any of the requested parameter data were overridden by the implementation.
Depending on the used video encode operation, additional codec-specific
structures can be be included in the pNext chain of this structure to
capture codec-specific feedback information about the requested parameter
data, as described in the corresponding sections.
42.14.6. Video Encode Commands
To launch video encode operations, call:
// Provided by VK_KHR_video_encode_queue
void vkCmdEncodeVideoKHR(
VkCommandBuffer commandBuffer,
const VkVideoEncodeInfoKHR* pEncodeInfo);
-
commandBufferis the command buffer in which to record the command. -
pEncodeInfois a pointer to a VkVideoEncodeInfoKHR structure specifying the parameters of the video encode operations.
Each call issues one or more video encode operations.
The implicit parameter opCount corresponds to the number of video
encode operations issued by the command.
After calling this command, the
active query index of each
active query is incremented by opCount.
Currently each call to this command results in the issue of a single video encode operation.
If the bound video session was created with
VK_VIDEO_SESSION_CREATE_INLINE_QUERIES_BIT_KHR and the pNext
chain of pEncodeInfo includes a VkVideoInlineQueryInfoKHR
structure with its queryPool member specifying a valid
VkQueryPool handle, then this command will execute a query for each
video encode operation issued by it.
- Active Reference Picture Information
-
The list of active reference pictures used by a video encode operation is a list of image subregions used as the source of reference picture data and related parameters, and is derived from the VkVideoReferenceSlotInfoKHR structures provided as the elements of the
pEncodeInfo->pReferenceSlotsarray. For each element ofpEncodeInfo->pReferenceSlots, one or more elements are added to the active reference picture list, as defined by the codec-specific semantics. Each element of this list contains the following information:-
The image subregion within the image subresource referred to by the video picture resource used as the reference picture.
-
The DPB slot index the reference picture is associated with.
-
The codec-specific reference information related to the reference picture.
-
- Reconstructed Picture Information
-
Information related to the optional reconstructed picture used by a video encode operation is derived from the VkVideoReferenceSlotInfoKHR structure pointed to by
pEncodeInfo->pSetupReferenceSlot, if notNULL, as defined by the codec-specific semantics, and consists of the following:-
The image subregion within the image subresource referred to by the video picture resource used as the reconstructed picture.
-
The DPB slot index to use for picture reconstruction.
-
The codec-specific reference information related to the reconstructed picture.
-
Specifying a valid VkVideoReferenceSlotInfoKHR structure in
pEncodeInfo->pSetupReferenceSlot is always required, unless the video
session was created with VkVideoSessionCreateInfoKHR::maxDpbSlot
equal to zero.
However, the DPB slot identified by
pEncodeInfo->pSetupReferenceSlot->slotIndex is only
activated with the reconstructed
picture specified in
pEncodeInfo->pSetupReferenceSlot->pPictureResource if reference
picture setup is requested according to the
codec-specific semantics.
If reconstructed picture information is specified, but reference picture setup is not requested, according to the codec-specific semantics, the contents of the video picture resource corresponding to the reconstructed picture will be undefined after the video encode operation.
|
Note
Some implementations may always output the reconstructed picture or use it as temporary storage during the video encode operation even when the reconstructed picture is not marked for future reference. |
- Encode Input Picture Information
-
Information related to the encode input picture used by a video encode operation is derived from
pEncodeInfo->srcPictureResourceand any codec-specific parameters provided in thepEncodeInfo->pNextchain, as defined by the codec-specific semantics, and consists of the following:-
The image subregion within the image subresource referred to by the video picture resource used as the encode input picture.
-
The codec-specific picture information related to the encoded picture.
-
Several limiting values are defined below that are referenced by the relevant valid usage statements of this command.
-
Let
uint32_t activeReferencePictureCountbe the size of the list of active reference pictures used by the video encode operation. Unless otherwise defined,activeReferencePictureCountis set to the value ofpEncodeInfo->referenceSlotCount. -
Let
VkOffset2D codedOffsetGranularitybe the minimum alignment requirement for the coded offset of video picture resources. Unless otherwise defined, the value of thexandymembers ofcodedOffsetGranularityare0. -
Let
uint32_t dpbFrameUseCount[]be an array of sizemaxDpbSlots, wheremaxDpbSlotsis the VkVideoSessionCreateInfoKHR::maxDpbSlotsthe bound video session was created with, with each element indicating the number of times a frame associated with the corresponding DPB slot index is referred to by the video coding operation. Let the initial value of each element of the array be0.-
If
pEncodeInfo->pSetupReferenceSlotis notNULL, thendpbFrameUseCount[i]is incremented by one, whereiequalspEncodeInfo->pSetupReferenceSlot->slotIndex. -
For each element of
pEncodeInfo->pReferenceSlots,dpbFrameUseCount[i]is incremented by one, whereiequals theslotIndexmember of the corresponding element.
-
-
Let
VkExtent2D maxCodingBlockSizebe the maximum codec-specific coding block size that may be used by the video encode operation.-
If the bound video session object was created with an H.264 encode profile, then let
maxCodingBlockSizebe equal to the size of an H.264 macroblock, i.e.{16,16}. -
If the bound video session object was created with an H.265 encode profile, then let
maxCodingBlockSizebe equal to the maximum H.265 coding block size that may be used by the video encode operation derived as the maximum of the CTB sizes corresponding to the VkVideoEncodeH265CtbSizeFlagBitsKHR bits set in VkVideoEncodeH265CapabilitiesKHR::ctbSizes, as returned by vkGetPhysicalDeviceVideoCapabilitiesKHR for the video profile the bound video session was created with. -
Otherwise,
maxCodingBlockSizeis undefined.
-
-
If
maxCodingBlockSizeis defined, then letVkExtent2D minCodingBlockExtentbe the coded extent of the encode input picture expressed in terms of codec-specific coding blocks, assuming the maximum size of such coding blocks, as defined bymaxCodingBlockSize, calculated from the value of thecodedExtentmember ofpEncodeInfo->srcPictureResourceas follows:-
minCodingBlockExtent.width= (codedExtent.width
maxCodingBlockSize.width- 1) /maxCodingBlockSize.width -
minCodingBlockExtent.height= (codedExtent.height
maxCodingBlockSize.height- 1) /maxCodingBlockSize.height
-
-
If the bound video session object was created with an H.264 encode profile, then:
-
Let
StdVideoH264PictureType h264PictureTypebe the picture type of the encoded picture set to the value ofpStdPictureInfo->primary_pic_typespecified in the VkVideoEncodeH264PictureInfoKHR structure included in thepEncodeInfo->pNextchain. -
Let
StdVideoH264PictureType h264L0PictureTypes[]andStdVideoH264PictureType h264L1PictureTypes[]be the picture types of the reference pictures in the L0 and L1 reference lists, respectively. IfpStdPictureInfo->pRefListsspecified in the VkVideoEncodeH264PictureInfoKHR structure included in thepEncodeInfo->pNextchain is notNULL, then for each reference index specified in the elements of thepStdPictureInfo->pRefLists->RefPicList0andpStdPictureInfo->pRefLists->RefPicList1arrays, if the reference index is notSTD_VIDEO_H264_NO_REFERENCE_PICTURE,pStdReferenceInfo->primary_pic_typeis added toh264L0PictureTypesorh264L1PictureTypes, respectively, wherepStdReferenceInfois the member of the VkVideoEncodeH264DpbSlotInfoKHR structure included in thepNextchain of the element ofpEncodeInfo->pReferenceSlotsfor whichslotIndexequals the reference index in question.
-
-
If the bound video session object was created with an H.265 encode profile, then:
-
Let
StdVideoH265PictureType h265PictureTypebe the picture type of the encoded picture set to the value ofpStdPictureInfo->pic_typespecified in the VkVideoEncodeH265PictureInfoKHR structure included in thepEncodeInfo->pNextchain. -
Let
StdVideoH265PictureType h265L0PictureTypes[]andStdVideoH265PictureType h265L1PictureTypes[]be the picture types of the reference pictures in the L0 and L1 reference lists, respectively. IfpStdPictureInfo->pRefListsspecified in the VkVideoEncodeH265PictureInfoKHR structure included in thepEncodeInfo->pNextchain is notNULL, then for each reference index specified in the elements of thepStdPictureInfo->pRefLists->RefPicList0andpStdPictureInfo->pRefLists->RefPicList1arrays, if the reference index is notSTD_VIDEO_H265_NO_REFERENCE_PICTURE,pStdReferenceInfo->pic_typeis added toh265L0PictureTypesorh265L1PictureTypes, respectively, wherepStdReferenceInfois the member of the VkVideoEncodeH265DpbSlotInfoKHR structure included in thepNextchain of the element ofpEncodeInfo->pReferenceSlotsfor whichslotIndexequals the reference index in question.
-
The VkVideoEncodeInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_queue
typedef struct VkVideoEncodeInfoKHR {
VkStructureType sType;
const void* pNext;
VkVideoEncodeFlagsKHR flags;
VkBuffer dstBuffer;
VkDeviceSize dstBufferOffset;
VkDeviceSize dstBufferRange;
VkVideoPictureResourceInfoKHR srcPictureResource;
const VkVideoReferenceSlotInfoKHR* pSetupReferenceSlot;
uint32_t referenceSlotCount;
const VkVideoReferenceSlotInfoKHR* pReferenceSlots;
uint32_t precedingExternallyEncodedBytes;
} VkVideoEncodeInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextis a pointer to a structure extending this structure. -
flagsis reserved for future use. -
dstBufferis the destination video bitstream buffer to write the encoded bitstream to. -
dstBufferOffsetis the starting offset in bytes from the start ofdstBufferto write the encoded bitstream to. -
dstBufferRangeis the maximum bitstream size in bytes that can be written todstBuffer, starting fromdstBufferOffset. -
srcPictureResourceis the video picture resource to use as the encode input picture. -
pSetupReferenceSlotisNULLor a pointer to a VkVideoReferenceSlotInfoKHR structure specifying the reconstructed picture information. -
referenceSlotCountis the number of elements in thepReferenceSlotsarray. -
pReferenceSlotsisNULLor a pointer to an array of VkVideoReferenceSlotInfoKHR structures describing the DPB slots and corresponding reference picture resources to use in this video encode operation (the set of active reference pictures). -
precedingExternallyEncodedBytesis the number of bytes externally encoded by the application to the video bitstream and is used to update the internal state of the implementation’s rate control algorithm to account for the bitrate budget consumed by these externally encoded bytes.
// Provided by VK_KHR_video_encode_queue
typedef VkFlags VkVideoEncodeFlagsKHR;
VkVideoEncodeFlagsKHR is a bitmask type for setting a mask, but is currently reserved for future use.
42.15. Video Encode Rate Control
The size of the encoded bitstream data produced by video encode operations is a function of the following set of constraints:
-
The capabilities of the compression algorithms defined and employed by the used video compression standard;
-
Restrictions imposed by the selected video profile according to the rules defined by the used video compression standard;
-
Further restrictions imposed by the capabilities supported by the implementation for the selected video profile;
-
The image data in the encode input picture and the set of active reference pictures (as these affect the effectiveness of the compression algorithms employed by the video encode operations);
-
The set of codec-specific and codec-independent encoding parameters provided by the application.
These also inherently define the set of decoder capabilities required for reconstructing and processing the picture data in the encoded bitstream.
Video coding uses bitrate as the quantitative metric associated with encoded bitstream data size which expresses the rate at which video bitstream data can be transferred or processed, measured in number of bits per second. This bitrate is both a function of the encoded bitstream data size of the encoded pictures as well as the frame rate used by the video sequence.
Rate control algorithms are used by video encode operations to enable adjusting encoding parameters to achieve a target bitrate, or otherwise directly or indirectly control the bitrate of the generated video bitstream data. These algorithms are usually not defined by the used video compression standard, although some video compression standards do provide non-normative guidelines for implementations.
Accordingly, this specification does not mandate implementations to produce identical encoded bitstream data outputs in response to video encode operations, however, it does define a set of codec-independent and codec-specific parameters that enable the application to control the behavior of the rate control algorithms supported by the implementation. Some of these parameters guarantee certain implementation behavior while others provide guidance for implementations to apply various rate control heuristics.
|
Note
Applications need to make sure that they configure rate control parameters appropriately and that they follow the promises made to the implementation through parameters providing guidance for the implementation’s rate control algorithms and heuristics in order to be able to get the desired rate control behavior and to be able to hit the set bitrate targets. In addition, the behavior of rate control may also differ across implementations even if the capabilities of the used video profile match between those implementations. This may happen due to implementations applying different rate control algorithms or heuristics internally, and thus even the same set of guidance parameter values may have different effects on the rate control behavior across implementations. |
42.15.1. Rate Control Modes
After a video session is reset to the initial state, the default behavior and parameters of video encode rate control are entirely implementation-dependent and the application cannot affect the bitrate or quality parameters of the encoded bitstream data produced by video encode operations unless the application changes the rate control configuration of the video session, as described in the Video Coding Control section.
For each supported video profile, the implementation may expose a set of rate control modes that are available for use by the application when encoding bitstreams targeting that video profile. These modes allow using different rate control algorithms that fall into one of the following two categories:
-
Per-operation rate control
-
Stream-level rate control
In case of per-operation rate control, the bitrate of the generated video bitstream data is indirectly controlled by quality, size, or other encoding parameters specified by the application for each individual video encode operation.
In case of stream-level rate control, the application can directly specify target bitrates besides other encoding parameters to control the behavior of the rate control algorithm used by the implementation across multiple video encode operations.
The rate control modes are defined with the following enums:
// Provided by VK_KHR_video_encode_queue
typedef enum VkVideoEncodeRateControlModeFlagBitsKHR {
VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DEFAULT_KHR = 0,
VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR = 0x00000001,
VK_VIDEO_ENCODE_RATE_CONTROL_MODE_CBR_BIT_KHR = 0x00000002,
VK_VIDEO_ENCODE_RATE_CONTROL_MODE_VBR_BIT_KHR = 0x00000004,
} VkVideoEncodeRateControlModeFlagBitsKHR;
-
VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DEFAULT_KHRspecifies the use of implementation-specific rate control. -
VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHRspecifies that rate control is disabled and the application will specify per-operation rate control parameters controlling the encoding quality. In this mode implementations will encode pictures independently of the output bitrate of prior video encode operations.-
When using an H.264 encode profile, implementations will use the QP value specified in VkVideoEncodeH264NaluSliceInfoKHR::
constantQpto control the quality of the encoded picture. -
When using an H.265 encode profile, implementations will use the QP value specified in VkVideoEncodeH265NaluSliceSegmentInfoKHR::
constantQpto control the quality of the encoded picture.
-
-
VK_VIDEO_ENCODE_RATE_CONTROL_MODE_CBR_BIT_KHRspecifies the use of constant bitrate (CBR) rate control mode. In this mode the implementation will attempt to produce the encoded bitstream at a constant bitrate while conforming to the constraints of other rate control parameters. -
VK_VIDEO_ENCODE_RATE_CONTROL_MODE_VBR_BIT_KHRspecifies the use of variable bitrate (VBR) rate control mode. In this mode the implementation will produce the encoded bitstream at a variable bitrate according to the constraints of other rate control parameters.
// Provided by VK_KHR_video_encode_queue
typedef VkFlags VkVideoEncodeRateControlModeFlagsKHR;
VkVideoEncodeRateControlModeFlagsKHR is a bitmask type for setting a
mask of zero or more VkVideoEncodeRateControlModeFlagBitsKHR.
42.15.2. Leaky Bucket Model
Video encoding implementations use the leaky bucket model for stream-level rate control. The leaky bucket is a concept referring to the interface between the video encoder and the consumer (for example, a network connection), where the video encoder produces encoded bitstream data corresponding to the encoded pictures and adds them in the leaky bucket while its content are drained by the consumer.
Analogously, a similar leaky bucket is considered to exist at the input interface of a video decoder, into which encoded bitstream data is continuously added and is subsequently consumed by the video decoder. It is desirable to avoid overflowing or underflowing this leaky bucked because:
-
In case of an underflow, the video decoder will be unable to consume encoded bitstream data in order to decode pictures (and optionally display them).
-
In case of an overflow, the leaky bucket will be unable to accommodate more encoded bitstream data and such data may need to be thrown away, leading to the loss of the corresponding encoded pictures.
These requirements can be satisfied by imposing various constraints on the encoder-side leaky bucket to avoid its overflow or underflow, depending on the used rate control algorithm and codec parameters. However, enumerating these constraints is outside the scope of this specification.
The term virtual buffer is often used as an alternative to refer to the leaky bucket.
This virtual buffer model is defined by the following parameters:
-
The bitrate (
R) at which the encoded bitstream is expected to be processed. -
The size (
B) of the virtual buffer. -
The initial occupancy (
F) of the virtual buffer.
In this model the virtual buffer is used to smooth out fluctuations in the bitrate of the encoded bitstream over time without experiencing buffer overflow or underflow, as long as the bitrate of the encoded stream does not diverge from the target bitrate for extended periods of time.
This buffering may inherently impose a processing delay, as the goal of the model is to enable decoders maintain a consistent processing rate of an encoded bitstream with varying data rate.
The initial or start-up delay (D) is computed as:
-
D=F/R
|
Note
Applications need to configure the virtual buffer with sufficient size to avoid or minimize buffer overflows and underflows while also keeping it small enough to meet their latency goals. |
42.15.3. Rate Control Layers
Some video compression standards and video profiles allow associating encoded pictures with specific video coding layers. The name, identification, and semantics associated with such video coding layers are defined by the corresponding video compression standards.
Analogously, stream-level rate control can be configured to use one or more rate control layers:
-
When a single rate control layer is configured, it is applied to all encoded pictures, regardless of the picture’s video coding layer. In this case the distribution of the available bitrate budget across video coding layers is implementation-dependent.
-
When multiple rate control layers are configured, each rate control layer is applied to the corresponding video coding layer, i.e. only across encoded pictures pertaining to the corresponding video coding layer.
Individual rate control layers are identified using layer indices between
zero and N-1, where N is the number of active rate control layers.
Rate control layers are only applicable when using stream-level rate control modes.
42.15.4. Rate Control State
Rate control state is maintained by the implementation in the
video session objects and its parameters are specified
using an instance of the VkVideoEncodeRateControlInfoKHR structure.
The complete rate control state of a video session is defined by the
following set of parameters:
-
The values of the members of the VkVideoEncodeRateControlInfoKHR structure used to configure the rate control state.
-
The values of the members of any VkVideoEncodeRateControlLayerInfoKHR structures specified in VkVideoEncodeRateControlInfoKHR::
pLayersused to configure the state of individual rate control layers. -
If the video session was created with an H.264 encode profile:
-
The values of the members of the VkVideoEncodeH264RateControlInfoKHR structure, if one is specified in the
pNextchain of the VkVideoEncodeRateControlInfoKHR used to configure the rate control state. -
The values of the members of any VkVideoEncodeH264RateControlLayerInfoKHR structures included in the
pNextchain of a VkVideoEncodeRateControlLayerInfoKHR structure used to configure the state of a rate control layer.
-
-
If the video session was created with an H.265 encode profile:
-
The values of the members of the VkVideoEncodeH265RateControlInfoKHR structure, if one is specified in the
pNextchain of the VkVideoEncodeRateControlInfoKHR used to configure the rate control state. -
The values of the members of any VkVideoEncodeH265RateControlLayerInfoKHR structures included in the
pNextchain of a VkVideoEncodeRateControlLayerInfoKHR structure used to configure the state of a rate control layer.
-
Two rate control states match if all the parameters listed above match between them.
The VkVideoEncodeRateControlInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_queue
typedef struct VkVideoEncodeRateControlInfoKHR {
VkStructureType sType;
const void* pNext;
VkVideoEncodeRateControlFlagsKHR flags;
VkVideoEncodeRateControlModeFlagBitsKHR rateControlMode;
uint32_t layerCount;
const VkVideoEncodeRateControlLayerInfoKHR* pLayers;
uint32_t virtualBufferSizeInMs;
uint32_t initialVirtualBufferSizeInMs;
} VkVideoEncodeRateControlInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
flagsis reserved for future use. -
rateControlModeis a VkVideoEncodeRateControlModeFlagBitsKHR value specifying the rate control mode. -
layerCountspecifies the number of rate control layers to use. -
pLayersis a pointer to an array oflayerCountVkVideoEncodeRateControlLayerInfoKHR structures, each specifying the rate control configuration of the corresponding rate control layer. -
virtualBufferSizeInMsis the size in milliseconds of the virtual buffer used by the implementation’s rate control algorithm for the leaky bucket model, with respect to the average bitrate of the stream calculated by summing the values of theaverageBitratemembers of the elements of thepLayersarray. -
initialVirtualBufferSizeInMsis the initial occupancy in milliseconds of the virtual buffer used by the implementation’s rate control algorithm for the leaky bucket model.
If layerCount is zero then the values of virtualBufferSizeInMs
and initialVirtualBufferSizeInMs are ignored.
This structure can be specified in the following places:
-
In the
pNextchain of VkVideoBeginCodingInfoKHR to specify the current rate control state expected to be configured when beginning a video coding scope. -
In the
pNextchain of VkVideoCodingControlInfoKHR to change the rate control configuration of the bound video session.
Including this structure in the pNext chain of
VkVideoCodingControlInfoKHR and including
VK_VIDEO_CODING_CONTROL_ENCODE_RATE_CONTROL_BIT_KHR in
VkVideoCodingControlInfoKHR::flags enables updating the rate
control configuration of the bound video session.
This replaces the entire rate control configuration of the bound video
session and may reset the state of all enabled rate control layers to an
initial state according to the codec-specific rate control semantics defined
in the corresponding sections listed below.
When layerCount is greater than one, multiple
rate control layers are configured, and each
rate control layer is applied to the corresponding video coding layer
identified by the index of the corresponding element of pLayer.
-
If the video session was created with the video codec operation
VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR, then this index specifies the H.264 temporal layer ID of the video coding layer the rate control layer is applied to. -
If the video session was created with the video codec operation
VK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR, then this index specifies the H.265 temporal ID of the video coding layer the rate control layer is applied to.
Additional structures providing codec-specific rate control parameters can
be included in the pNext chain of VkVideoCodingControlInfoKHR
depending on the video profile the bound video session
was created.
For further details see:
The new rate control configuration takes effect when the corresponding vkCmdControlVideoCodingKHR is executed on the device, and only impacts video encode operations that follow in execution order.
// Provided by VK_KHR_video_encode_queue
typedef VkFlags VkVideoEncodeRateControlFlagsKHR;
VkVideoEncodeRateControlFlagsKHR is a bitmask type for setting a mask,
but currently reserved for future use.
Rate Control Layer State
The configuration of individual rate control layers is specified using an
instance of the VkVideoEncodeRateControlLayerInfoKHR structure.
The VkVideoEncodeRateControlLayerInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_queue
typedef struct VkVideoEncodeRateControlLayerInfoKHR {
VkStructureType sType;
const void* pNext;
uint64_t averageBitrate;
uint64_t maxBitrate;
uint32_t frameRateNumerator;
uint32_t frameRateDenominator;
} VkVideoEncodeRateControlLayerInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextis a pointer to a structure extending this structure. -
averageBitrateis the average bitrate to be targeted by the implementation’s rate control algorithm. -
maxBitrateis the peak bitrate to be targeted by the implementation’s rate control algorithm. -
frameRateNumeratoris the numerator of the frame rate assumed by the implementation’s rate control algorithm. -
frameRateDenominatoris the denominator of the frame rate assumed by the implementation’s rate control algorithm.
|
Note
The ability of the implementation’s rate control algorithm to be able to match the requested average and/or peak bitrates may be limited by the set of other codec-independent and codec-specific rate control parameters specified by the application, the input content, as well as the application conforming to the rate control guidance provided to the implementation, as described earlier. |
Additional structures providing codec-specific rate control parameters can
be included in the pNext chain of
VkVideoEncodeRateControlLayerInfoKHR depending on the
video profile the bound video session was created with.
For further details see:
42.16. H.264 Encode Operations
Video encode operations using an H.264 encode profile can be used to encode elementary video stream sequences compliant to the ITU-T H.264 Specification.
|
Note
Refer to the Preamble for information on how the Khronos Intellectual Property Rights Policy relates to normative references to external materials not created by Khronos. |
This process is performed according to the video encode operation steps with the codec-specific semantics defined in section 8 of the ITU-T H.264 Specification as follows:
-
Syntax elements, derived values, and other parameters are applied from the following structures:
-
The
StdVideoH264SequenceParameterSetstructure corresponding to the active SPS specifying the H.264 sequence parameter set. -
The
StdVideoH264PictureParameterSetstructure corresponding to the active PPS specifying the H.264 picture parameter set. -
The
StdVideoEncodeH264PictureInfostructure specifying the H.264 picture information. -
The
StdVideoEncodeH264SliceHeaderstructures specifying the H.264 slice header parameters for each encoded H.264 slice. -
The
StdVideoEncodeH264ReferenceInfostructures specifying the H.264 reference information corresponding to the optional reconstructed picture and any active reference pictures.
-
-
The encoded bitstream data is written to the destination video bitstream buffer range as defined in the H.264 Encode Bitstream Data Access section.
-
Picture data in the video picture resources corresponding to the used encode input picture, active reference pictures, and optional reconstructed picture is accessed as defined in the H.264 Encode Picture Data Access section.
-
The decision on reference picture setup is made according to the parameters specified in the H.264 picture information.
If the parameters adhere to the syntactic and semantic requirements defined in the corresponding sections of the ITU-T H.264 Specification, as described above, and the DPB slots associated with the active reference pictures all refer to valid picture references, then the video encode operation will complete successfully. Otherwise, the video encode operation may complete unsuccessfully.
42.16.1. H.264 Encode Parameter Overrides
Implementations may override, unless otherwise specified, any of the H.264 encode parameters specified in the following Video Std structures:
-
StdVideoH264SequenceParameterSet -
StdVideoH264PictureParameterSet -
StdVideoEncodeH264PictureInfo -
StdVideoEncodeH264SliceHeader -
StdVideoEncodeH264ReferenceInfo
All such H.264 encode parameter overrides must fulfill the conditions defined in the Video Encode Parameter Overrides section.
In addition, implementations must not override any of the following H.264 encode parameters:
-
StdVideoEncodeH264PictureInfo::primary_pic_type -
StdVideoEncodeH264SliceHeader::slice_type
In case of H.264 encode parameters stored in video session parameters objects, applications need to use the vkGetEncodedVideoSessionParametersKHR command to determine whether any implementation overrides happened. If the query indicates that implementation overrides were applied, then the application needs to retrieve and use the encoded H.264 parameter sets in the bitstream in order to be able to produce a compliant H.264 video bitstream using the H.264 encode parameters stored in the video session parameters object.
In case of any H.264 encode parameters stored in the encoded bitstream
produced by video encode operations, if the implementation supports the
VK_VIDEO_ENCODE_FEEDBACK_BITSTREAM_HAS_OVERRIDES_BIT_KHR
video encode feedback query flag, the
application can use such queries to retrieve feedback about whether any
implementation overrides have been applied to those H.264 encode parameters.
42.16.2. H.264 Encode Bitstream Data Access
Each video encode operation writes one or more VCL NAL units comprising of
slice headers and data of the encoded picture, in the format defined in
sections 7.3.3 and 7.3.4, according to the semantics defined in sections
7.4.3 and 7.4.4 of the ITU-T H.264 Specification,
respectively.
The number of VCL NAL units written is specified by
VkVideoEncodeH264PictureInfoKHR::naluSliceEntryCount.
In addition, if
VkVideoEncodeH264PictureInfoKHR::generatePrefixNalu is set to
VK_TRUE for the video encode operation, then an additional prefix NAL
unit is written before each VCL NAL unit corresponding to individual slices
in the format defined in section 7.3.2.12, according to the semantics
defined in section 7.4.2.12 of the ITU-T H.264 Specification,
respectively.
42.16.3. H.264 Encode Picture Data Access
Accesses to image data within a video picture resource happen at the
granularity indicated by
VkVideoCapabilitiesKHR::pictureAccessGranularity, as returned by
vkGetPhysicalDeviceVideoCapabilitiesKHR for the used video profile.
Accordingly, the complete image subregion of a encode
input picture, reference picture, or
reconstructed picture accessed by video coding
operations using an H.264 encode profile is defined
as the set of texels within the coordinate range:
-
([0,
endX),[0,endY))
Where:
-
endXequalscodedExtent.widthrounded up to the nearest integer multiple ofpictureAccessGranularity.widthand clamped to the width of the image subresource referred to by the corresponding VkVideoPictureResourceInfoKHR structure; -
endY equals
codedExtent.heightrounded up to the nearest integer multiple ofpictureAccessGranularity.heightand clamped to the height of the image subresource referred to by the corresponding VkVideoPictureResourceInfoKHR structure;
Where codedExtent is the member of the
VkVideoPictureResourceInfoKHR structure corresponding to the picture.
In case of video encode operations using an H.264
encode profile, any access to a picture at the coordinates
(x,y), as defined by the ITU-T H.264
Specification, is an access to the image subresource
referred to by the corresponding
VkVideoPictureResourceInfoKHR structure at the texel coordinates
(x,y).
Implementations may choose not to access some or all texels within particular reference pictures available to a video encode operation (e.g. due to video encode parameter overrides restricting the effective set of used reference pictures, or if the encoding algorithm chooses not to use certain subregions of the reference picture data for sample prediction).
42.16.4. H.264 Frame, Picture, and Slice
H.264 pictures are partitioned into slices, as defined in section 6.3 of the ITU-T H.264 Specification.
Video encode operations using an H.264 encode
profile can encode slices of different types, as defined in section 7.4.3
of the ITU-T H.264 Specification, by specifying the
corresponding enumeration constant value in
StdVideoEncodeH264SliceHeader::slice_type in the
H.264 slice header parameters from the
Video Std enumeration type StdVideoH264SliceType:
-
STD_VIDEO_H264_SLICE_TYPE_Pindicates that the slice is a P slice as defined in section 3.109 of the ITU-T H.264 Specification. -
STD_VIDEO_H264_SLICE_TYPE_Bindicates that the slice is a B slice as defined in section 3.9 of the ITU-T H.264 Specification. -
STD_VIDEO_H264_SLICE_TYPE_Iindicates that the slice is an I slice as defined in section 3.66 of the ITU-T H.264 Specification.
Pictures constructed from such slices can be of different types, as defined
in section 7.4.2.4 of the ITU-T H.264 Specification.
Video encode operations using an H.264 encode
profile can encode pictures of a specific type by specifying the
corresponding enumeration constant value in
StdVideoEncodeH264PictureInfo::primary_pic_type in the
H.264 picture information from the Video Std
enumeration type StdVideoH264PictureType:
-
STD_VIDEO_H264_PICTURE_TYPE_Pindicates that the picture is a P picture. A frame consisting of a P picture is also referred to as a P frame. -
STD_VIDEO_H264_PICTURE_TYPE_Bindicates that the picture is a B picture. A frame consisting of a B picture is also referred to as a B frame. -
STD_VIDEO_H264_PICTURE_TYPE_Iindicates that the picture is an I picture. A frame consisting of an I picture is also referred to as an I frame. -
STD_VIDEO_H264_PICTURE_TYPE_IDRindicates that the picture is a special type of I picture called an IDR picture as defined in section 3.69 of the ITU-T H.264 Specification. A frame consisting of an IDR picture is also referred to as an IDR frame.
42.16.5. H.264 Encode Profile
A video profile supporting H.264 video encode operations is specified by
setting VkVideoProfileInfoKHR::videoCodecOperation to
VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR and adding a
VkVideoEncodeH264ProfileInfoKHR structure to the
VkVideoProfileInfoKHR::pNext chain.
The VkVideoEncodeH264ProfileInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264ProfileInfoKHR {
VkStructureType sType;
const void* pNext;
StdVideoH264ProfileIdc stdProfileIdc;
} VkVideoEncodeH264ProfileInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
stdProfileIdcis aStdVideoH264ProfileIdcvalue specifying the H.264 codec profile IDC, as defined in section A.2 of the ITU-T H.264 Specification.
42.16.6. H.264 Encode Capabilities
When calling vkGetPhysicalDeviceVideoCapabilitiesKHR to query the
capabilities for an H.264 encode profile, the
VkVideoCapabilitiesKHR::pNext chain must include a
VkVideoEncodeH264CapabilitiesKHR structure that will be filled with
the profile-specific capabilities.
The VkVideoEncodeH264CapabilitiesKHR structure is defined as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264CapabilitiesKHR {
VkStructureType sType;
void* pNext;
VkVideoEncodeH264CapabilityFlagsKHR flags;
StdVideoH264LevelIdc maxLevelIdc;
uint32_t maxSliceCount;
uint32_t maxPPictureL0ReferenceCount;
uint32_t maxBPictureL0ReferenceCount;
uint32_t maxL1ReferenceCount;
uint32_t maxTemporalLayerCount;
VkBool32 expectDyadicTemporalLayerPattern;
int32_t minQp;
int32_t maxQp;
VkBool32 prefersGopRemainingFrames;
VkBool32 requiresGopRemainingFrames;
VkVideoEncodeH264StdFlagsKHR stdSyntaxFlags;
} VkVideoEncodeH264CapabilitiesKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
flagsis a bitmask of VkVideoEncodeH264CapabilityFlagBitsKHR indicating supported H.264 encoding capabilities. -
maxLevelIdcis aStdVideoH264LevelIdcvalue indicating the maximum H.264 level supported by the profile, where enum constantSTD_VIDEO_H264_LEVEL_IDC_<major>_<minor>identifies H.264 level<major>.<minor>as defined in section A.3 of the ITU-T H.264 Specification. -
maxSliceCountindicates the maximum number of slices that can be encoded for a single picture. Further restrictions may apply to the number of slices that can be encoded for a single picture depending on other capabilities and codec-specific rules. -
maxPPictureL0ReferenceCountindicates the maximum number of reference pictures the implementation supports in the reference list L0 for P pictures.NoteAs implementations may override the reference lists,
maxPPictureL0ReferenceCountdoes not limit the number of elements that the application can specify in the L0 reference list for P pictures. However, ifmaxPPictureL0ReferenceCountis zero, then the use of P pictures is not allowed. -
maxBPictureL0ReferenceCountindicates the maximum number of reference pictures the implementation supports in the reference list L0 for B pictures. -
maxL1ReferenceCountindicates the maximum number of reference pictures the implementation supports in the reference list L1 if encoding of B pictures is supported.NoteAs implementations may override the reference lists,
maxBPictureL0ReferenceCountandmaxL1ReferenceCountdoes not limit the number of elements that the application can specify in the L0 and L1 reference lists for B pictures. However, ifmaxBPictureL0ReferenceCountandmaxL1ReferenceCountare both zero, then the use of B pictures is not allowed. -
maxTemporalLayerCountindicates the maximum number of H.264 temporal layers supported by the implementation. -
expectDyadicTemporalLayerPatternindicates that the implementation’s rate control algorithms expect the application to use a dyadic temporal layer pattern when encoding multiple temporal layers. -
minQpindicates the minimum QP value supported. -
maxQpindicates the maximum QP value supported. -
prefersGopRemainingFramesindicates that the implementation’s rate control algorithm prefers the application to specify the number of frames of each type remaining in the current group of pictures when beginning a video coding scope. -
requiresGopRemainingFramesindicates that the implementation’s rate control algorithm requires the application to specify the number of frames of each type remaining in the current group of pictures when beginning a video coding scope. -
stdSyntaxFlagsis a bitmask of VkVideoEncodeH264StdFlagBitsKHR indicating capabilities related to H.264 syntax elements.
Bits which may be set in
VkVideoEncodeH264CapabilitiesKHR::flags, indicating the H.264
encoding capabilities supported, are:
// Provided by VK_KHR_video_encode_h264
typedef enum VkVideoEncodeH264CapabilityFlagBitsKHR {
VK_VIDEO_ENCODE_H264_CAPABILITY_HRD_COMPLIANCE_BIT_KHR = 0x00000001,
VK_VIDEO_ENCODE_H264_CAPABILITY_PREDICTION_WEIGHT_TABLE_GENERATED_BIT_KHR = 0x00000002,
VK_VIDEO_ENCODE_H264_CAPABILITY_ROW_UNALIGNED_SLICE_BIT_KHR = 0x00000004,
VK_VIDEO_ENCODE_H264_CAPABILITY_DIFFERENT_SLICE_TYPE_BIT_KHR = 0x00000008,
VK_VIDEO_ENCODE_H264_CAPABILITY_B_FRAME_IN_L0_LIST_BIT_KHR = 0x00000010,
VK_VIDEO_ENCODE_H264_CAPABILITY_B_FRAME_IN_L1_LIST_BIT_KHR = 0x00000020,
VK_VIDEO_ENCODE_H264_CAPABILITY_PER_PICTURE_TYPE_MIN_MAX_QP_BIT_KHR = 0x00000040,
VK_VIDEO_ENCODE_H264_CAPABILITY_PER_SLICE_CONSTANT_QP_BIT_KHR = 0x00000080,
VK_VIDEO_ENCODE_H264_CAPABILITY_GENERATE_PREFIX_NALU_BIT_KHR = 0x00000100,
} VkVideoEncodeH264CapabilityFlagBitsKHR;
-
VK_VIDEO_ENCODE_H264_CAPABILITY_HRD_COMPLIANCE_BIT_KHRindicates whether the implementation may be able to generate HRD compliant bitstreams if any of thenal_hrd_parameters_present_flagorvcl_hrd_parameters_present_flagmembers ofStdVideoH264SpsVuiFlagsare set to1in the active SPS. -
VK_VIDEO_ENCODE_H264_CAPABILITY_PREDICTION_WEIGHT_TABLE_GENERATED_BIT_KHRindicates that ifStdVideoH264PpsFlags::weighted_pred_flagis set to1orStdVideoH264PictureParameterSet::weighted_bipred_idcis set toSTD_VIDEO_H264_WEIGHTED_BIPRED_IDC_EXPLICITin the active PPS when encoding a P picture or B picture, respectively, then the implementation is able to internally decide syntax forpred_weight_table, as defined in section 7.4.3.2 of the ITU-T H.264 Specification, and the application is not required to provide a weight table in the H.264 slice header parameters. -
VK_VIDEO_ENCODE_H264_CAPABILITY_ROW_UNALIGNED_SLICE_BIT_KHRindicates that each slice in a frame with multiple slices may begin or finish at any offset in a macroblock row. If not supported, all slices in the frame must begin at the start of a macroblock row (and hence each slice must finish at the end of a macroblock row). -
VK_VIDEO_ENCODE_H264_CAPABILITY_DIFFERENT_SLICE_TYPE_BIT_KHRindicates that when a frame is encoded with multiple slices, the implementation allows encoding each slice with a differentStdVideoEncodeH264SliceHeader::slice_typespecified in the H.264 slice header parameters. If not supported, all slices of the frame must be encoded with the sameslice_typewhich corresponds to the picture type of the frame. -
VK_VIDEO_ENCODE_H264_CAPABILITY_B_FRAME_IN_L0_LIST_BIT_KHRindicates support for using a B frame as L0 reference, as specified inStdVideoEncodeH264ReferenceListsInfo::RefPicList0in the H.264 picture information. -
VK_VIDEO_ENCODE_H264_CAPABILITY_B_FRAME_IN_L1_LIST_BIT_KHRindicates support for using a B frame as L1 reference, as specified inStdVideoEncodeH264ReferenceListsInfo::RefPicList1in the H.264 picture information. -
VK_VIDEO_ENCODE_H264_CAPABILITY_PER_PICTURE_TYPE_MIN_MAX_QP_BIT_KHRindicates support for specifying different QP values in the members of VkVideoEncodeH264QpKHR. -
VK_VIDEO_ENCODE_H264_CAPABILITY_PER_SLICE_CONSTANT_QP_BIT_KHRindicates support for specifying different constant QP values for each slice. -
VK_VIDEO_ENCODE_H264_CAPABILITY_GENERATE_PREFIX_NALU_BIT_KHRindicates support for generating prefix NAL units by setting VkVideoEncodeH264PictureInfoKHR::generatePrefixNalutoVK_TRUE.
// Provided by VK_KHR_video_encode_h264
typedef VkFlags VkVideoEncodeH264CapabilityFlagsKHR;
VkVideoEncodeH264CapabilityFlagsKHR is a bitmask type for setting a
mask of zero or more VkVideoEncodeH264CapabilityFlagBitsKHR.
Bits which may be set in
VkVideoEncodeH264CapabilitiesKHR::stdSyntaxFlags, indicating the
capabilities related to the H.264 syntax elements, are:
// Provided by VK_KHR_video_encode_h264
typedef enum VkVideoEncodeH264StdFlagBitsKHR {
VK_VIDEO_ENCODE_H264_STD_SEPARATE_COLOR_PLANE_FLAG_SET_BIT_KHR = 0x00000001,
VK_VIDEO_ENCODE_H264_STD_QPPRIME_Y_ZERO_TRANSFORM_BYPASS_FLAG_SET_BIT_KHR = 0x00000002,
VK_VIDEO_ENCODE_H264_STD_SCALING_MATRIX_PRESENT_FLAG_SET_BIT_KHR = 0x00000004,
VK_VIDEO_ENCODE_H264_STD_CHROMA_QP_INDEX_OFFSET_BIT_KHR = 0x00000008,
VK_VIDEO_ENCODE_H264_STD_SECOND_CHROMA_QP_INDEX_OFFSET_BIT_KHR = 0x00000010,
VK_VIDEO_ENCODE_H264_STD_PIC_INIT_QP_MINUS26_BIT_KHR = 0x00000020,
VK_VIDEO_ENCODE_H264_STD_WEIGHTED_PRED_FLAG_SET_BIT_KHR = 0x00000040,
VK_VIDEO_ENCODE_H264_STD_WEIGHTED_BIPRED_IDC_EXPLICIT_BIT_KHR = 0x00000080,
VK_VIDEO_ENCODE_H264_STD_WEIGHTED_BIPRED_IDC_IMPLICIT_BIT_KHR = 0x00000100,
VK_VIDEO_ENCODE_H264_STD_TRANSFORM_8X8_MODE_FLAG_SET_BIT_KHR = 0x00000200,
VK_VIDEO_ENCODE_H264_STD_DIRECT_SPATIAL_MV_PRED_FLAG_UNSET_BIT_KHR = 0x00000400,
VK_VIDEO_ENCODE_H264_STD_ENTROPY_CODING_MODE_FLAG_UNSET_BIT_KHR = 0x00000800,
VK_VIDEO_ENCODE_H264_STD_ENTROPY_CODING_MODE_FLAG_SET_BIT_KHR = 0x00001000,
VK_VIDEO_ENCODE_H264_STD_DIRECT_8X8_INFERENCE_FLAG_UNSET_BIT_KHR = 0x00002000,
VK_VIDEO_ENCODE_H264_STD_CONSTRAINED_INTRA_PRED_FLAG_SET_BIT_KHR = 0x00004000,
VK_VIDEO_ENCODE_H264_STD_DEBLOCKING_FILTER_DISABLED_BIT_KHR = 0x00008000,
VK_VIDEO_ENCODE_H264_STD_DEBLOCKING_FILTER_ENABLED_BIT_KHR = 0x00010000,
VK_VIDEO_ENCODE_H264_STD_DEBLOCKING_FILTER_PARTIAL_BIT_KHR = 0x00020000,
VK_VIDEO_ENCODE_H264_STD_SLICE_QP_DELTA_BIT_KHR = 0x00080000,
VK_VIDEO_ENCODE_H264_STD_DIFFERENT_SLICE_QP_DELTA_BIT_KHR = 0x00100000,
} VkVideoEncodeH264StdFlagBitsKHR;
-
VK_VIDEO_ENCODE_H264_STD_SEPARATE_COLOR_PLANE_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH264SpsFlags::separate_colour_plane_flagin the SPS when that value is1. -
VK_VIDEO_ENCODE_H264_STD_QPPRIME_Y_ZERO_TRANSFORM_BYPASS_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH264SpsFlags::qpprime_y_zero_transform_bypass_flagin the SPS when that value is1. -
VK_VIDEO_ENCODE_H264_STD_SCALING_MATRIX_PRESENT_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided values forStdVideoH264SpsFlags::seq_scaling_matrix_present_flagin the SPS andStdVideoH264PpsFlags::pic_scaling_matrix_present_flagin the PPS when any of those values are1. -
VK_VIDEO_ENCODE_H264_STD_CHROMA_QP_INDEX_OFFSET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH264PictureParameterSet::chroma_qp_index_offsetin the PPS when that value is non-zero. -
VK_VIDEO_ENCODE_H264_STD_SECOND_CHROMA_QP_INDEX_OFFSET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH264PictureParameterSet::second_chroma_qp_index_offsetin the PPS when that value is non-zero. -
VK_VIDEO_ENCODE_H264_STD_PIC_INIT_QP_MINUS26_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH264PictureParameterSet::pic_init_qp_minus26in the PPS when that value is non-zero. -
VK_VIDEO_ENCODE_H264_STD_WEIGHTED_PRED_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH264PpsFlags::weighted_pred_flagin the PPS when that value is1. -
VK_VIDEO_ENCODE_H264_STD_WEIGHTED_BIPRED_IDC_EXPLICIT_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH264PictureParameterSet::weighted_bipred_idcin the PPS when that value isSTD_VIDEO_H264_WEIGHTED_BIPRED_IDC_EXPLICIT. -
VK_VIDEO_ENCODE_H264_STD_WEIGHTED_BIPRED_IDC_IMPLICIT_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH264PictureParameterSet::weighted_bipred_idcin the PPS when that value isSTD_VIDEO_H264_WEIGHTED_BIPRED_IDC_IMPLICIT. -
VK_VIDEO_ENCODE_H264_STD_TRANSFORM_8X8_MODE_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH264PpsFlags::transform_8x8_mode_flagin the PPS when that value is1. -
VK_VIDEO_ENCODE_H264_STD_DIRECT_SPATIAL_MV_PRED_FLAG_UNSET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoEncodeH264SliceHeaderFlags::direct_spatial_mv_pred_flagin the H.264 slice header parameters when that value is0. -
VK_VIDEO_ENCODE_H264_STD_ENTROPY_CODING_MODE_FLAG_UNSET_BIT_KHRindicates whether the implementation supports CAVLC entropy coding, as defined in section 9.2 of the ITU-T H.264 Specification, and thus supports using the application-provided value forStdVideoH264PpsFlags::entropy_coding_mode_flagin the PPS when that value is0. -
VK_VIDEO_ENCODE_H264_STD_ENTROPY_CODING_MODE_FLAG_SET_BIT_KHRindicates whether the implementation supports CABAC entropy coding, as defined in section 9.3 of the ITU-T H.264 Specification, and thus supports using the application-provided value forStdVideoH264PpsFlags::entropy_coding_mode_flagin the PPS when that value is1. -
VK_VIDEO_ENCODE_H264_STD_DIRECT_8X8_INFERENCE_FLAG_UNSET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH264SpsFlags::direct_8x8_inference_flagin the SPS when that value is0. -
VK_VIDEO_ENCODE_H264_STD_CONSTRAINED_INTRA_PRED_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH264PpsFlags::constrained_intra_pred_flagin the PPS when that value is1. -
VK_VIDEO_ENCODE_H264_STD_DEBLOCKING_FILTER_DISABLED_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoEncodeH264SliceHeader::disable_deblocking_filter_idcin the H.264 slice header parameters when that value isSTD_VIDEO_H264_DISABLE_DEBLOCKING_FILTER_IDC_DISABLED. -
VK_VIDEO_ENCODE_H264_STD_DEBLOCKING_FILTER_ENABLED_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoEncodeH264SliceHeader::disable_deblocking_filter_idcin the H.264 slice header parameters when that value isSTD_VIDEO_H264_DISABLE_DEBLOCKING_FILTER_IDC_ENABLED. -
VK_VIDEO_ENCODE_H264_STD_DEBLOCKING_FILTER_PARTIAL_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoEncodeH264SliceHeader::disable_deblocking_filter_idcin the H.264 slice header parameters when that value isSTD_VIDEO_H264_DISABLE_DEBLOCKING_FILTER_IDC_PARTIAL. -
VK_VIDEO_ENCODE_H264_STD_SLICE_QP_DELTA_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoEncodeH264SliceHeader::slice_qp_deltain the H.264 slice header parameters when that value is identical across the slices of the encoded frame. -
VK_VIDEO_ENCODE_H264_STD_DIFFERENT_SLICE_QP_DELTA_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoEncodeH264SliceHeader::slice_qp_deltain the H.264 slice header parameters when that value is different across the slices of the encoded frame.
These capability flags provide information to the application about specific H.264 syntax element values that the implementation supports without having to override them and do not otherwise restrict the values that the application can specify for any of the mentioned H.264 syntax elements.
// Provided by VK_KHR_video_encode_h264
typedef VkFlags VkVideoEncodeH264StdFlagsKHR;
VkVideoEncodeH264StdFlagsKHR is a bitmask type for setting a mask of
zero or more VkVideoEncodeH264StdFlagBitsKHR.
42.16.7. H.264 Encode Quality Level Properties
When calling vkGetPhysicalDeviceVideoEncodeQualityLevelPropertiesKHR
with pVideoProfile->videoCodecOperation specified as
VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR, the
VkVideoEncodeH264QualityLevelPropertiesKHR structure must be included
in the pNext chain of the VkVideoEncodeQualityLevelPropertiesKHR
structure to retrieve additional video encode quality level properties
specific to H.264 encoding.
The VkVideoEncodeH264QualityLevelPropertiesKHR structure is defined as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264QualityLevelPropertiesKHR {
VkStructureType sType;
void* pNext;
VkVideoEncodeH264RateControlFlagsKHR preferredRateControlFlags;
uint32_t preferredGopFrameCount;
uint32_t preferredIdrPeriod;
uint32_t preferredConsecutiveBFrameCount;
uint32_t preferredTemporalLayerCount;
VkVideoEncodeH264QpKHR preferredConstantQp;
uint32_t preferredMaxL0ReferenceCount;
uint32_t preferredMaxL1ReferenceCount;
VkBool32 preferredStdEntropyCodingModeFlag;
} VkVideoEncodeH264QualityLevelPropertiesKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
preferredRateControlFlagsis a bitmask of VkVideoEncodeH264RateControlFlagBitsKHR values indicating the preferred flags to use for VkVideoEncodeH264RateControlInfoKHR::flags. -
preferredGopFrameCountindicates the preferred value to use for VkVideoEncodeH264RateControlInfoKHR::gopFrameCount. -
preferredIdrPeriodindicates the preferred value to use for VkVideoEncodeH264RateControlInfoKHR::idrPeriod. -
preferredConsecutiveBFrameCountindicates the preferred value to use for VkVideoEncodeH264RateControlInfoKHR::consecutiveBFrameCount. -
preferredTemporalLayerCountindicates the preferred value to use for VkVideoEncodeH264RateControlInfoKHR::temporalLayerCount. -
preferredConstantQpindicates the preferred values to use for VkVideoEncodeH264NaluSliceInfoKHR::constantQpfor each picture type when using rate control modeVK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR. -
preferredMaxL0ReferenceCountindicates the preferred maximum number of reference pictures to use in the reference list L0. -
preferredMaxL1ReferenceCountindicates the preferred maximum number of reference pictures to use in the reference list L1. -
preferredStdEntropyCodingModeFlagindicates the preferred value to use forentropy_coding_mode_flaginStdVideoH264PpsFlags.
42.16.8. H.264 Encode Session
Additional parameters can be specified when creating a video session with an
H.264 encode profile by including an instance of the
VkVideoEncodeH264SessionCreateInfoKHR structure in the pNext
chain of VkVideoSessionCreateInfoKHR.
The VkVideoEncodeH264SessionCreateInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264SessionCreateInfoKHR {
VkStructureType sType;
const void* pNext;
VkBool32 useMaxLevelIdc;
StdVideoH264LevelIdc maxLevelIdc;
} VkVideoEncodeH264SessionCreateInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
useMaxLevelIdcindicates whether the value ofmaxLevelIdcshould be used by the implementation. When it is set toVK_FALSE, the implementation ignores the value ofmaxLevelIdcand uses the value of VkVideoEncodeH264CapabilitiesKHR::maxLevelIdc, as reported by vkGetPhysicalDeviceVideoCapabilitiesKHR for the video profile. -
maxLevelIdcis aStdVideoH264LevelIdcvalue specifying the upper bound on the H.264 level for the video bitstreams produced by the created video session, where enum constantSTD_VIDEO_H264_LEVEL_IDC_<major>_<minor>identifies H.264 level<major>.<minor>as defined in section A.3 of the ITU-T H.264 Specification.
42.16.9. H.264 Encode Parameter Sets
Video session parameters objects created with
the video codec operation VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR
can contain the following types of parameters:
- H.264 Sequence Parameter Sets (SPS)
-
Represented by
StdVideoH264SequenceParameterSetstructures and interpreted as follows:-
reserved1andreserved2are used only for padding purposes and are otherwise ignored; -
seq_parameter_set_idis used as the key of the SPS entry; -
level_idcis one of the enum constantsSTD_VIDEO_H264_LEVEL_IDC_<major>_<minor>identifying the H.264 level<major>.<minor>as defined in section A.3 of the ITU-T H.264 Specification; -
if
flags.seq_scaling_matrix_present_flagis set, then theStdVideoH264ScalingListsstructure pointed to bypScalingListsis interpreted as follows:-
scaling_list_present_maskis a bitmask where bit index i corresponds toseq_scaling_list_present_flag[i]as defined in section 7.4.2.1 of the ITU-T H.264 Specification; -
use_default_scaling_matrix_maskis a bitmask where bit index i corresponds toUseDefaultScalingMatrix4x4Flag[i], when i < 6, or corresponds toUseDefaultScalingMatrix8x8Flag[i-6], otherwise, as defined in section 7.3.2.1 of the ITU-T H.264 Specification; -
ScalingList4x4andScalingList8x8correspond to the identically named syntax elements defined in section 7.3.2.1 of the ITU-T H.264 Specification;
-
-
if
flags.vui_parameters_present_flagis set, thenpSequenceParameterSetVuiis a pointer to aStdVideoH264SequenceParameterSetVuistructure that is interpreted as follows:-
reserved1is used only for padding purposes and is otherwise ignored; -
if
flags.nal_hrd_parameters_present_flagorflags.vcl_hrd_parameters_present_flagis set, then theStdVideoH264HrdParametersstructure pointed to bypHrdParametersis interpreted as follows:-
reserved1is used only for padding purposes and is otherwise ignored; -
all other members of
StdVideoH264HrdParametersare interpreted as defined in section E.2.2 of the ITU-T H.264 Specification;
-
-
all other members of
StdVideoH264SequenceParameterSetVuiare interpreted as defined in section E.2.1 of the ITU-T H.264 Specification;
-
-
all other members of
StdVideoH264SequenceParameterSetare interpreted as defined in section 7.4.2.1 of the ITU-T H.264 Specification.
-
- H.264 Picture Parameter Sets (PPS)
-
Represented by
StdVideoH264PictureParameterSetstructures and interpreted as follows:-
the pair constructed from
seq_parameter_set_idandpic_parameter_set_idis used as the key of the PPS entry; -
if
flags.pic_scaling_matrix_present_flagis set, then theStdVideoH264ScalingListsstructure pointed to bypScalingListsis interpreted as follows:-
scaling_list_present_maskis a bitmask where bit index i corresponds topic_scaling_list_present_flag[i]as defined in section 7.4.2.2 of the ITU-T H.264 Specification; -
use_default_scaling_matrix_maskis a bitmask where bit index i corresponds toUseDefaultScalingMatrix4x4Flag[i], when i < 6, or corresponds toUseDefaultScalingMatrix8x8Flag[i-6], otherwise, as defined in section 7.3.2.2 of the ITU-T H.264 Specification; -
ScalingList4x4andScalingList8x8correspond to the identically named syntax elements defined in section 7.3.2.2 of the ITU-T H.264 Specification;
-
-
all other members of
StdVideoH264PictureParameterSetare interpreted as defined in section 7.4.2.2 of the ITU-T H.264 Specification.
-
Implementations may override any of these parameters according to the semantics defined in the Video Encode Parameter Overrides section before storing the resulting H.264 parameter sets into the video session parameters object. Applications need to use the vkGetEncodedVideoSessionParametersKHR command to determine whether any implementation overrides happened and to retrieve the encoded H.264 parameter sets in order to be able to produce a compliant H.264 video bitstream.
Such H.264 parameter set overrides may also have cascading effects on the
implementation overrides applied to the encoded bitstream produced by video
encode operations.
If the implementation supports the
VK_VIDEO_ENCODE_FEEDBACK_BITSTREAM_HAS_OVERRIDES_BIT_KHR
video encode feedback query flag, then the
application can use such queries to retrieve feedback about whether any
implementation overrides have been applied to the encoded bitstream.
When a video session parameters object is
created with the codec operation
VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR, the
VkVideoSessionParametersCreateInfoKHR::pNext chain must include
a VkVideoEncodeH264SessionParametersCreateInfoKHR structure specifying
the capacity and initial contents of the object.
The VkVideoEncodeH264SessionParametersCreateInfoKHR structure is
defined as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264SessionParametersCreateInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t maxStdSPSCount;
uint32_t maxStdPPSCount;
const VkVideoEncodeH264SessionParametersAddInfoKHR* pParametersAddInfo;
} VkVideoEncodeH264SessionParametersCreateInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
maxStdSPSCountis the maximum number of H.264 SPS entries the createdVkVideoSessionParametersKHRcan contain. -
maxStdPPSCountis the maximum number of H.264 PPS entries the createdVkVideoSessionParametersKHRcan contain. -
pParametersAddInfoisNULLor a pointer to a VkVideoEncodeH264SessionParametersAddInfoKHR structure specifying H.264 parameters to add upon object creation.
The VkVideoEncodeH264SessionParametersAddInfoKHR structure is defined
as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264SessionParametersAddInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t stdSPSCount;
const StdVideoH264SequenceParameterSet* pStdSPSs;
uint32_t stdPPSCount;
const StdVideoH264PictureParameterSet* pStdPPSs;
} VkVideoEncodeH264SessionParametersAddInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
stdSPSCountis the number of elements in thepStdSPSsarray. -
pStdSPSsis a pointer to an array ofStdVideoH264SequenceParameterSetstructures describing the H.264 SPS entries to add. -
stdPPSCountis the number of elements in thepStdPPSsarray. -
pStdPPSsis a pointer to an array ofStdVideoH264PictureParameterSetstructures describing the H.264 PPS entries to add.
This structure can be specified in the following places:
-
In the
pParametersAddInfomember of the VkVideoEncodeH264SessionParametersCreateInfoKHR structure specified in thepNextchain of VkVideoSessionParametersCreateInfoKHR used to create a video session parameters object. In this case, if the video codec operation the video session parameters object is created with isVK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR, then it defines the set of initial parameters to add to the created object (see Creating Video Session Parameters). -
In the
pNextchain of VkVideoSessionParametersUpdateInfoKHR. In this case, if the video codec operation the video session parameters object to be updated was created with isVK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR, then it defines the set of parameters to add to it (see Updating Video Session Parameters).
The VkVideoEncodeH264SessionParametersGetInfoKHR structure is defined
as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264SessionParametersGetInfoKHR {
VkStructureType sType;
const void* pNext;
VkBool32 writeStdSPS;
VkBool32 writeStdPPS;
uint32_t stdSPSId;
uint32_t stdPPSId;
} VkVideoEncodeH264SessionParametersGetInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
writeStdSPSindicates whether the encoded H.264 sequence parameter set identified bystdSPSIdis requested to be retrieved. -
writeStdPPSindicates whether the encoded H.264 picture parameter set identified by the pair constructed fromstdSPSIdandstdPPSIdis requested to be retrieved. -
stdSPSIdspecifies the H.264 sequence parameter set ID used to identify the retrieved H.264 sequence and/or picture parameter set(s). -
stdPPSIdspecifies the H.264 picture parameter set ID used to identify the retrieved H.264 picture parameter set whenwriteStdPPSis set toVK_TRUE.
When this structure is specified in the pNext chain of the
VkVideoEncodeSessionParametersGetInfoKHR structure passed to
vkGetEncodedVideoSessionParametersKHR, the command will write encoded
parameter data to the output buffer in the following order:
-
The H.264 sequence parameter set identified by
stdSPSId, ifwriteStdSPSis set toVK_TRUE. -
The H.264 picture parameter set identified by the pair constructed from
stdSPSIdandstdPPSId, ifwriteStdPPSis set toVK_TRUE.
The VkVideoEncodeH264SessionParametersFeedbackInfoKHR structure is
defined as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264SessionParametersFeedbackInfoKHR {
VkStructureType sType;
void* pNext;
VkBool32 hasStdSPSOverrides;
VkBool32 hasStdPPSOverrides;
} VkVideoEncodeH264SessionParametersFeedbackInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
hasStdSPSOverridesindicates whether any of the parameters of the requested H.264 sequence parameter set, if one was requested via VkVideoEncodeH264SessionParametersGetInfoKHR::writeStdSPS, were overridden by the implementation. -
hasStdPPSOverridesindicates whether any of the parameters of the requested H.264 picture parameter set, if one was requested via VkVideoEncodeH264SessionParametersGetInfoKHR::writeStdPPS, were overridden by the implementation.
42.16.10. H.264 Encoding Parameters
The VkVideoEncodeH264PictureInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264PictureInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t naluSliceEntryCount;
const VkVideoEncodeH264NaluSliceInfoKHR* pNaluSliceEntries;
const StdVideoEncodeH264PictureInfo* pStdPictureInfo;
VkBool32 generatePrefixNalu;
} VkVideoEncodeH264PictureInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
naluSliceEntryCountis the number of elements inpNaluSliceEntries. -
pNaluSliceEntriesis a pointer to an array ofnaluSliceEntryCountVkVideoEncodeH264NaluSliceInfoKHR structures specifying the parameters of the individual H.264 slices to encode for the input picture. -
pStdPictureInfois a pointer to aStdVideoEncodeH264PictureInfostructure specifying H.264 picture information. -
generatePrefixNalucontrols whether prefix NALUs are generated before slice NALUs into the target bitstream, as defined in sections 7.3.2.12 and 7.4.2.12 of the ITU-T H.264 Specification.
This structure is specified in the pNext chain of the
VkVideoEncodeInfoKHR structure passed to vkCmdEncodeVideoKHR to
specify the codec-specific picture information for an H.264
encode operation.
- Encode Input Picture Information
-
When this structure is specified in the
pNextchain of the VkVideoEncodeInfoKHR structure passed to vkCmdEncodeVideoKHR, the information related to the encode input picture is defined as follows:-
The image subregion used is determined according to the H.264 Encode Picture Data Access section.
-
The encode input picture is associated with the H.264 picture information provided in
pStdPictureInfo.
-
- Std Picture Information
-
The members of the
StdVideoEncodeH264PictureInfostructure pointed to bypStdPictureInfoare interpreted as follows:-
flags.reservedandreserved1are used only for padding purposes and are otherwise ignored; -
flags.IdrPicFlagas defined in section 7.4.1 of the ITU-T H.264 Specification; -
flags.is_referenceas defined in section 3.136 of the ITU-T H.264 Specification; -
seq_parameter_set_idandpic_parameter_set_idare used to identify the active parameter sets, as described below; -
primary_pic_typeas defined in section 7.4.2 of the ITU-T H.264 Specification; -
PicOrderCntas defined in section 8.2 of the ITU-T H.264 Specification; -
temporal_idas defined in section G.7.4.1.1 of the ITU-T H.264 Specification; -
if
pRefListsis notNULL, then it is a pointer to aStdVideoEncodeH264ReferenceListsInfostructure that is interpreted as follows:-
flags.reservedis used only for padding purposes and is otherwise ignored; -
ref_pic_list_modification_flag_l0andref_pic_list_modification_flag_l1as defined in section 7.4.3.1 of the ITU-T H.264 Specification; -
num_ref_idx_l0_active_minus1andnum_ref_idx_l1_active_minus1as defined in section 7.4.3 of the ITU-T H.264 Specification; -
RefPicList0andRefPicList1as defined in section 8.2.4 of the ITU-T H.264 Specification where each element of these arrays either identifies an active reference picture using its DPB slot index or contains the valueSTD_VIDEO_H264_NO_REFERENCE_PICTUREto indicate “no reference picture”; -
if
refList0ModOpCountis not zero, thenpRefList0ModOperationsis a pointer to an array ofrefList0ModOpCountnumber ofStdVideoEncodeH264RefListModEntrystructures specifying the modification parameters for the reference list L0 as defined in section 7.4.3.1 of the ITU-T H.264 Specification; -
if
refList1ModOpCountis not zero, thenpRefList1ModOperationsis a pointer to an array ofrefList1ModOpCountnumber ofStdVideoEncodeH264RefListModEntrystructures specifying the modification parameters for the reference list L1 as defined in section 7.4.3.1 of the ITU-T H.264 Specification; -
if
refPicMarkingOpCountis not zero, thenrefPicMarkingOperationsis a pointer to an array ofrefPicMarkingOpCountnumber ofStdVideoEncodeH264RefPicMarkingEntrystructures specifying the reference picture marking parameters as defined in section 7.4.3.3 of the ITU-T H.264 Specification;
-
-
all other members are interpreted as defined in section 7.4.3 of the ITU-T H.264 Specification.
-
Reference picture setup is controlled by the value of
StdVideoEncodeH264PictureInfo::flags.is_reference.
If it is set and a reconstructed
picture is specified, then the latter is used as the target of picture
reconstruction to activate the DPB slot
specified in pEncodeInfo->pSetupReferenceSlot->slotIndex.
If StdVideoEncodeH264PictureInfo::flags.is_reference is not set,
but a reconstructed picture is
specified, then the corresponding picture reference associated with the
DPB slot is invalidated, as described in the
DPB Slot States section.
- Active Parameter Sets
-
The members of the
StdVideoEncodeH264PictureInfostructure pointed to bypStdPictureInfoare used to select the active parameter sets to use from the bound video session parameters object, as follows:-
The active SPS is the SPS identified by the key specified in
StdVideoEncodeH264PictureInfo::seq_parameter_set_id. -
The active PPS is the PPS identified by the key specified by the pair constructed from
StdVideoEncodeH264PictureInfo::seq_parameter_set_idandStdVideoEncodeH264PictureInfo::pic_parameter_set_id.
-
H.264 encoding uses explicit weighted sample prediction for a slice, as
defined in section 8.4.2.3 of the ITU-T H.264 Specification,
if any of the following conditions are true for the active
PPS and the pStdSliceHeader member of the
corresponding element of pNaluSliceEntries:
-
pStdSliceHeader->slice_typeisSTD_VIDEO_H264_SLICE_TYPE_Pandweighted_pred_flagis enabled in the active PPS. -
pStdSliceHeader->slice_typeisSTD_VIDEO_H264_SLICE_TYPE_Bandweighted_bipred_idcin the active PPS equalsSTD_VIDEO_H264_WEIGHTED_BIPRED_IDC_EXPLICIT.
The VkVideoEncodeH264NaluSliceInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264NaluSliceInfoKHR {
VkStructureType sType;
const void* pNext;
int32_t constantQp;
const StdVideoEncodeH264SliceHeader* pStdSliceHeader;
} VkVideoEncodeH264NaluSliceInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
constantQpis the QP to use for the slice if the current rate control mode configured for the video session isVK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR. -
pStdSliceHeaderis a pointer to aStdVideoEncodeH264SliceHeaderstructure specifying H.264 slice header parameters for the slice.
- Std Slice Header Parameters
-
The members of the
StdVideoEncodeH264SliceHeaderstructure pointed to bypStdSliceHeaderare interpreted as follows:-
flags.reservedandreserved1are used only for padding purposes and are otherwise ignored; -
if
pWeightTableis notNULL, then it is a pointer to aStdVideoEncodeH264WeightTablethat is interpreted as follows:-
flags.reservedis used only for padding purposes and is otherwise ignored; -
all other members of
StdVideoEncodeH264WeightTableare interpreted as defined in section 7.4.3.2 of the ITU-T H.264 Specification;
-
-
all other members are interpreted as defined in section 7.4.3 of the ITU-T H.264 Specification.
-
The VkVideoEncodeH264DpbSlotInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264DpbSlotInfoKHR {
VkStructureType sType;
const void* pNext;
const StdVideoEncodeH264ReferenceInfo* pStdReferenceInfo;
} VkVideoEncodeH264DpbSlotInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
pStdReferenceInfois a pointer to aStdVideoEncodeH264ReferenceInfostructure specifying H.264 reference information.
This structure is specified in the pNext chain of
VkVideoEncodeInfoKHR::pSetupReferenceSlot, if not NULL, and
the pNext chain of the elements of
VkVideoEncodeInfoKHR::pReferenceSlots to specify the
codec-specific reference picture information for an H.264
encode operation.
- Active Reference Picture Information
-
When this structure is specified in the
pNextchain of the elements of VkVideoEncodeInfoKHR::pReferenceSlots, one element is added to the list of active reference pictures used by the video encode operation for each element of VkVideoEncodeInfoKHR::pReferenceSlotsas follows:-
The image subregion used is determined according to the H.264 Encode Picture Data Access section.
-
The reference picture is associated with the DPB slot index specified in the
slotIndexmember of the corresponding element of VkVideoEncodeInfoKHR::pReferenceSlots. -
The reference picture is associated with the H.264 reference information provided in
pStdReferenceInfo.
-
- Reconstructed Picture Information
-
When this structure is specified in the
pNextchain of VkVideoEncodeInfoKHR::pSetupReferenceSlot, the information related to the reconstructed picture is defined as follows:-
The image subregion used is determined according to the H.264 Encode Picture Data Access section.
-
If reference picture setup is requested, then the reconstructed picture is used to activate the DPB slot with the index specified in VkVideoEncodeInfoKHR::
pSetupReferenceSlot->slotIndex. -
The reconstructed picture is associated with the H.264 reference information provided in
pStdReferenceInfo.
-
- Std Reference Information
-
The members of the
StdVideoEncodeH264ReferenceInfostructure pointed to bypStdReferenceInfoare interpreted as follows:-
flags.reservedis used only for padding purposes and is otherwise ignored; -
flags.used_for_long_term_referenceis used to indicate whether the picture is marked as “used for long-term reference” as defined in section 8.2.5.1 of the ITU-T H.264 Specification; -
primary_pic_typeas defined in section 7.4.2 of the ITU-T H.264 Specification; -
long_term_pic_numandlong_term_frame_idxas defined in section 7.4.3 of the ITU-T H.264 Specification; -
temporal_idas defined in section G.7.4.1.1 of the ITU-T H.264 Specification; -
all other members are interpreted as defined in section 8.2 of the ITU-T H.264 Specification.
-
42.16.11. H.264 Encode Rate Control
Group of Pictures
In case of H.264 encoding it is common practice to follow a regular pattern of different picture types in display order when encoding subsequent frames. This pattern is referred to as the group of pictures (GOP).
A regular GOP is defined by the following parameters:
-
The number of frames in the GOP;
-
The number of consecutive B frames between I and/or P frames in display order.
GOPs are further classified as open and closed GOPs.
Frame types in an open GOP follow each other in display order according to the following algorithm:
-
The first frame is always an I frame.
-
This is followed by a number of consecutive B frames, as defined above.
-
If the number of frames in the GOP is not reached yet, then the next frame is a P frame and the algorithm continues from step 2.
In case of a closed GOP, an IDR frame is used at a certain period.
It is also typical for H.264 encoding to use specific reference picture usage patterns across the frames of the GOP. The two most common reference patterns used are as follows:
- Flat Reference Pattern
-
-
Each P frame uses the last non-B frame, in display order, as reference.
-
Each B frame uses the last non-B frame, in display order, as its backward reference, and uses the next non-B frame, in display order, as its forward reference.
-
- Dyadic Reference Pattern
-
-
Each P frame uses the last non-B frame, in display order, as reference.
-
The following algorithm is applied to the sequence of consecutive B frames between I and/or P frames in display order:
-
The B frame in the middle of this sequence uses the frame preceding the sequence as its backward reference, and uses the frame following the sequence as its forward reference.
-
The algorithm is executed recursively for the following frame sequences:
-
The B frames of the original sequence preceding the frame in the middle, if any.
-
The B frames of the original sequence following the frame in the middle, if any.
-
-
-
The application can provide guidance to the implementation’s rate control algorithm about the structure of the GOP used by the application. Any such guidance about the GOP and its structure does not mandate that specific GOP structure to be used by the application, as the picture type of individual encoded pictures is still application-controlled, however, any deviation from the provided guidance may result in undesired rate control behavior including, but not limited, to the implementation not being able to conform to the expected average or target bitrates, or other rate control parameters specified by the application.
When an H.264 encode session is used to encode multiple temporal layers, it is also common practice to follow a regular pattern for the H.264 temporal ID for the encoded pictures in display order when encoding subsequent frames. This pattern is referred to as the temporal GOP. The most common temporal layer pattern used is as follows:
- Dyadic Temporal Layer Pattern
-
-
The number of frames in the temporal GOP is 2n-1, where n is the number of temporal layers.
-
The ith frame in the temporal GOP uses temporal ID t, if and only if the index of the least significant bit set in i equals n-t-1, except for the first frame, which is the only frame in the temporal GOP using temporal ID zero.
-
The ith frame in the temporal GOP uses the rth frame as reference, where r is calculated from i by clearing the least significant bit set in it, except for the first frame in the temporal GOP, which uses the first frame of the previous temporal GOP, if any, as reference.
-
|
Note
Multi-layer rate control and multi-layer coding are typically used for streaming cases where low latency is expected, hence B pictures with forward prediction are usually not used. |
The VkVideoEncodeH264RateControlInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264RateControlInfoKHR {
VkStructureType sType;
const void* pNext;
VkVideoEncodeH264RateControlFlagsKHR flags;
uint32_t gopFrameCount;
uint32_t idrPeriod;
uint32_t consecutiveBFrameCount;
uint32_t temporalLayerCount;
} VkVideoEncodeH264RateControlInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
flagsis a bitmask of VkVideoEncodeH264RateControlFlagBitsKHR specifying H.264 rate control flags. -
gopFrameCountis the number of frames within a group of pictures (GOP) intended to be used by the application. If it is set to 0, the rate control algorithm may assume an implementation-dependent GOP length. If it is set toUINT32_MAX, the GOP length is treated as infinite. -
idrPeriodis the interval, in terms of number of frames, between two IDR frames (see IDR period). If it is set to 0, the rate control algorithm may assume an implementation-dependent IDR period. If it is set toUINT32_MAX, the IDR period is treated as infinite. -
consecutiveBFrameCountis the number of consecutive B frames between I and/or P frames within the GOP. -
temporalLayerCountspecifies the number of H.264 temporal layers that the application intends to use.
When an instance of this structure is included in the pNext chain of
the VkVideoCodingControlInfoKHR structure passed to the
vkCmdControlVideoCodingKHR command, and
VkVideoCodingControlInfoKHR::flags includes
VK_VIDEO_CODING_CONTROL_ENCODE_RATE_CONTROL_BIT_KHR, the parameters in
this structure are used as guidance for the implementation’s rate control
algorithm (see Video Coding Control).
If flags includes
VK_VIDEO_ENCODE_H264_RATE_CONTROL_ATTEMPT_HRD_COMPLIANCE_BIT_KHR, then
the rate control state is reset to an initial state to meet HRD compliance
requirements.
Otherwise the new rate control state may be applied without a reset
depending on the implementation and the specified rate control parameters.
|
Note
It would be possible to infer the picture type to be used when encoding a
frame, on the basis of the values provided for |
Bits which can be set in
VkVideoEncodeH264RateControlInfoKHR::flags, specifying H.264
rate control flags, are:
// Provided by VK_KHR_video_encode_h264
typedef enum VkVideoEncodeH264RateControlFlagBitsKHR {
VK_VIDEO_ENCODE_H264_RATE_CONTROL_ATTEMPT_HRD_COMPLIANCE_BIT_KHR = 0x00000001,
VK_VIDEO_ENCODE_H264_RATE_CONTROL_REGULAR_GOP_BIT_KHR = 0x00000002,
VK_VIDEO_ENCODE_H264_RATE_CONTROL_REFERENCE_PATTERN_FLAT_BIT_KHR = 0x00000004,
VK_VIDEO_ENCODE_H264_RATE_CONTROL_REFERENCE_PATTERN_DYADIC_BIT_KHR = 0x00000008,
VK_VIDEO_ENCODE_H264_RATE_CONTROL_TEMPORAL_LAYER_PATTERN_DYADIC_BIT_KHR = 0x00000010,
} VkVideoEncodeH264RateControlFlagBitsKHR;
-
VK_VIDEO_ENCODE_H264_RATE_CONTROL_ATTEMPT_HRD_COMPLIANCE_BIT_KHRspecifies that rate control should attempt to produce an HRD compliant bitstream, as defined in annex C of the ITU-T H.264 Specification. -
VK_VIDEO_ENCODE_H264_RATE_CONTROL_REGULAR_GOP_BIT_KHRspecifies that the application intends to use a regular GOP structure according to the parameters specified in thegopFrameCount,idrPeriod, andconsecutiveBFrameCountmembers of the VkVideoEncodeH264RateControlInfoKHR structure. -
VK_VIDEO_ENCODE_H264_RATE_CONTROL_REFERENCE_PATTERN_FLAT_BIT_KHRspecifies that the application intends to follow a flat reference pattern in the GOP. -
VK_VIDEO_ENCODE_H264_RATE_CONTROL_REFERENCE_PATTERN_DYADIC_BIT_KHRspecifies that the application intends to follow a dyadic reference pattern in the GOP. -
VK_VIDEO_ENCODE_H264_RATE_CONTROL_TEMPORAL_LAYER_PATTERN_DYADIC_BIT_KHRspecifies that the application intends to follow a dyadic temporal layer pattern.
// Provided by VK_KHR_video_encode_h264
typedef VkFlags VkVideoEncodeH264RateControlFlagsKHR;
VkVideoEncodeH264RateControlFlagsKHR is a bitmask type for setting a
mask of zero or more VkVideoEncodeH264RateControlFlagBitsKHR.
Rate Control Layers
The VkVideoEncodeH264RateControlLayerInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264RateControlLayerInfoKHR {
VkStructureType sType;
const void* pNext;
VkBool32 useMinQp;
VkVideoEncodeH264QpKHR minQp;
VkBool32 useMaxQp;
VkVideoEncodeH264QpKHR maxQp;
VkBool32 useMaxFrameSize;
VkVideoEncodeH264FrameSizeKHR maxFrameSize;
} VkVideoEncodeH264RateControlLayerInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
useMinQpindicates whether the QP values determined by rate control will be clamped to the lower bounds on the QP values specified inminQp. -
minQpspecifies the lower bounds on the QP values, for each picture type, that the implementation’s rate control algorithm will use whenuseMinQpis set toVK_TRUE. -
useMaxQpindicates whether the QP values determined by rate control will be clamped to the upper bounds on the QP values specified inmaxQp. -
maxQpspecifies the upper bounds on the QP values, for each picture type, that the implementation’s rate control algorithm will use whenuseMaxQpis set toVK_TRUE. -
useMaxFrameSizeindicates whether the implementation’s rate control algorithm should use the values specified inmaxFrameSizeas the upper bounds on the encoded frame size for each picture type. -
maxFrameSizespecifies the upper bounds on the encoded frame size, for each picture type, whenuseMaxFrameSizeis set toVK_TRUE.
When used, the values in minQp and maxQp guarantee that the
effective QP values used by the implementation will respect those lower and
upper bounds, respectively.
However, limiting the range of QP values that the implementation is able to
use will also limit the capabilities of the implementation’s rate control
algorithm to comply to other constraints.
In particular, the implementation may not be able to comply to the
following:
-
The average and/or peak bitrate values to be used for the encoded bitstream specified in the
averageBitrateandmaxBitratemembers of the VkVideoEncodeRateControlLayerInfoKHR structure. -
The upper bounds on the encoded frame size, for each picture type, specified in the
maxFrameSizemember ofVkVideoEncodeH264RateControlLayerInfoKHR.
|
Note
In general, applications need to configure rate control parameters appropriately in order to be able to get the desired rate control behavior, as described in the Video Encode Rate Control section. |
When an instance of this structure is included in the pNext chain of a
VkVideoEncodeRateControlLayerInfoKHR structure specified in one of the
elements of the pLayers array member of the
VkVideoEncodeRateControlInfoKHR structure passed to the
vkCmdControlVideoCodingKHR command,
VkVideoCodingControlInfoKHR::flags includes
VK_VIDEO_CODING_CONTROL_ENCODE_RATE_CONTROL_BIT_KHR, and the bound
video session was created with the video codec operation
VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR, it specifies the
H.264-specific rate control parameters of the rate control layer
corresponding to that element of pLayers.
The VkVideoEncodeH264QpKHR structure is defined as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264QpKHR {
int32_t qpI;
int32_t qpP;
int32_t qpB;
} VkVideoEncodeH264QpKHR;
-
qpIis the QP to be used for I pictures. -
qpPis the QP to be used for P pictures. -
qpBis the QP to be used for B pictures.
The VkVideoEncodeH264FrameSizeKHR structure is defined as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264FrameSizeKHR {
uint32_t frameISize;
uint32_t framePSize;
uint32_t frameBSize;
} VkVideoEncodeH264FrameSizeKHR;
-
frameISizeis the size in bytes to be used for I pictures. -
framePSizeis the size in bytes to be used for P pictures. -
frameBSizeis the size in bytes to be used for B pictures.
GOP Remaining Frames
Besides session level rate control configuration, the application can specify the number of frames per frame type remaining in the group of pictures (GOP).
The VkVideoEncodeH264GopRemainingFrameInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h264
typedef struct VkVideoEncodeH264GopRemainingFrameInfoKHR {
VkStructureType sType;
const void* pNext;
VkBool32 useGopRemainingFrames;
uint32_t gopRemainingI;
uint32_t gopRemainingP;
uint32_t gopRemainingB;
} VkVideoEncodeH264GopRemainingFrameInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
useGopRemainingFramesindicates whether the implementation’s rate control algorithm should use the values specified ingopRemainingI,gopRemainingP, andgopRemainingB. IfuseGopRemainingFramesisVK_FALSE, then the values ofgopRemainingI,gopRemainingP, andgopRemainingBare ignored. -
gopRemainingIspecifies the number of I frames the implementation’s rate control algorithm should assume to be remaining in the GOP prior to executing the video encode operation. -
gopRemainingPspecifies the number of P frames the implementation’s rate control algorithm should assume to be remaining in the GOP prior to executing the video encode operation. -
gopRemainingBspecifies the number of B frames the implementation’s rate control algorithm should assume to be remaining in the GOP prior to executing the video encode operation.
Setting useGopRemainingFrames to VK_TRUE and including this
structure in the pNext chain of VkVideoBeginCodingInfoKHR is
only mandatory if the
VkVideoEncodeH264CapabilitiesKHR::requiresGopRemainingFrames
reported for the used video profile is VK_TRUE.
However, implementations may use these remaining frame counts, when
specified, even when it is not required.
In particular, when the application does not use a
regular GOP structure, these values may provide
additional guidance for the implementation’s rate control algorithm.
The VkVideoEncodeH264CapabilitiesKHR::prefersGopRemainingFrames
capability is also used to indicate that the implementation’s rate control
algorithm may operate more accurately if the application specifies the
remaining frame counts using this structure.
As with other rate control guidance values, if the effective order and number of frames encoded by the application are not in line with the remaining frame counts specified in this structure at any given point, then the behavior of the implementation’s rate control algorithm may deviate from the one expected by the application.
42.16.12. H.264 Encode Requirements
This section described the required H.264 encoding capabilities for
physical devices that have at least one queue family that supports the video
codec operation VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR, as
returned by vkGetPhysicalDeviceQueueFamilyProperties2 in
VkQueueFamilyVideoPropertiesKHR::videoCodecOperations.
| Video Std Header Name | Version |
|---|---|
|
1.0.0 |
| Video Capability | Requirement | Requirement Type1 |
|---|---|---|
|
- |
min |
|
4096 |
max |
|
4096 |
max |
|
(64,64) |
max |
|
- |
max |
|
- |
min |
|
0 |
min |
|
0 |
min |
|
- |
min |
|
- |
min |
|
64000 |
min |
|
1 |
min |
|
(64,64) |
max |
|
|
min |
|
- |
min |
|
|
min |
|
1 |
min |
|
0 |
min |
|
0 |
min |
|
0 |
min |
|
1 |
min |
|
- |
implementation-dependent |
|
- |
max |
|
- |
min |
|
- |
implementation-dependent |
|
- |
implementation-dependent |
|
- |
min |
- 1
-
The Requirement Type column specifies the requirement is either the minimum value all implementations must support, the maximum value all implementations must support, or the exact value all implementations must support. For bitmasks a minimum value is the least bits all implementations must set, but they may have additional bits set beyond this minimum.
42.17. H.265 Encode Operations
Video encode operations using an H.265 encode profile can be used to encode elementary video stream sequences compliant to the ITU-T H.265 Specification.
|
Note
Refer to the Preamble for information on how the Khronos Intellectual Property Rights Policy relates to normative references to external materials not created by Khronos. |
This process is performed according to the video encode operation steps with the codec-specific semantics defined in section 8 of the ITU-T H.265 Specification as follows:
-
Syntax elements, derived values, and other parameters are applied from the following structures:
-
The
StdVideoH265VideoParameterSetstructure corresponding to the active VPS specifying the H.265 video parameter set. -
The
StdVideoH265SequenceParameterSetstructure corresponding to the active SPS specifying the H.265 sequence parameter set. -
The
StdVideoH265PictureParameterSetstructure corresponding to the active PPS specifying the H.265 picture parameter set. -
The
StdVideoEncodeH265PictureInfostructure specifying the H.265 picture information. -
The
StdVideoEncodeH265SliceSegmentHeaderstructures specifying the H.265 slice segment header parameters for each encoded H.265 slice segment. -
The
StdVideoEncodeH265ReferenceInfostructures specifying the H.265 reference information corresponding to the optional reconstructed picture and any active reference pictures.
-
-
The encoded bitstream data is written to the destination video bitstream buffer range as defined in the H.265 Encode Bitstream Data Access section.
-
Picture data in the video picture resources corresponding to the used encode input picture, active reference pictures, and optional reconstructed picture is accessed as defined in the H.265 Encode Picture Data Access section.
-
The decision on reference picture setup is made according to the parameters specified in the H.265 picture information.
If the parameters adhere to the syntactic and semantic requirements defined in the corresponding sections of the ITU-T H.265 Specification, as described above, and the DPB slots associated with the active reference pictures all refer to valid picture references, then the video encode operation will complete successfully. Otherwise, the video encode operation may complete unsuccessfully.
42.17.1. H.265 Encode Parameter Overrides
Implementations may override, unless otherwise specified, any of the H.265 encode parameters specified in the following Video Std structures:
-
StdVideoH265VideoParameterSet -
StdVideoH265SequenceParameterSet -
StdVideoH265PictureParameterSet -
StdVideoEncodeH265PictureInfo -
StdVideoEncodeH265SliceSegmentHeader -
StdVideoEncodeH265ReferenceInfo
All such H.265 encode parameter overrides must fulfill the conditions defined in the Video Encode Parameter Overrides section.
In addition, implementations must not override any of the following H.265 encode parameters:
-
StdVideoEncodeH265PictureInfo::pic_type -
StdVideoEncodeH265SliceSegmentHeader::slice_type
In case of H.265 encode parameters stored in video session parameters objects, applications need to use the vkGetEncodedVideoSessionParametersKHR command to determine whether any implementation overrides happened. If the query indicates that implementation overrides were applied, then the application needs to retrieve and use the encoded H.265 parameter sets in the bitstream in order to be able to produce a compliant H.265 video bitstream using the H.265 encode parameters stored in the video session parameters object.
In case of any H.265 encode parameters stored in the encoded bitstream
produced by video encode operations, if the implementation supports the
VK_VIDEO_ENCODE_FEEDBACK_BITSTREAM_HAS_OVERRIDES_BIT_KHR
video encode feedback query flag, the
application can use such queries to retrieve feedback about whether any
implementation overrides have been applied to those H.265 encode parameters.
42.17.2. H.265 Encode Bitstream Data Access
Each video encode operation writes one or more VCL NAL units comprising of
slice segment headers and data of the encoded picture, in the format defined
in sections 7.3.6 and 7.3.8, according to the semantics defined in sections
7.4.7 and 7.4.9 of the ITU-T H.265 Specification,
respectively.
The number of VCL NAL units written is specified by
VkVideoEncodeH265PictureInfoKHR::naluSliceSegmentEntryCount.
42.17.3. H.265 Encode Picture Data Access
Accesses to image data within a video picture resource happen at the
granularity indicated by
VkVideoCapabilitiesKHR::pictureAccessGranularity, as returned by
vkGetPhysicalDeviceVideoCapabilitiesKHR for the used video profile.
Accordingly, the complete image subregion of a encode
input picture, reference picture, or
reconstructed picture accessed by video coding
operations using an H.265 encode profile is defined
as the set of texels within the coordinate range:
-
([0,
endX),[0,endY))
Where:
-
endXequalscodedExtent.widthrounded up to the nearest integer multiple ofpictureAccessGranularity.widthand clamped to the width of the image subresource referred to by the corresponding VkVideoPictureResourceInfoKHR structure; -
endY equals
codedExtent.heightrounded up to the nearest integer multiple ofpictureAccessGranularity.heightand clamped to the height of the image subresource referred to by the corresponding VkVideoPictureResourceInfoKHR structure;
Where codedExtent is the member of the
VkVideoPictureResourceInfoKHR structure corresponding to the picture.
In case of video encode operations using an H.265
encode profile, any access to a picture at the coordinates
(x,y), as defined by the ITU-T H.265
Specification, is an access to the image subresource
referred to by the corresponding
VkVideoPictureResourceInfoKHR structure at the texel coordinates
(x,y).
Implementations may choose not to access some or all texels within particular reference pictures available to a video encode operation (e.g. due to video encode parameter overrides restricting the effective set of used reference pictures, or if the encoding algorithm chooses not to use certain subregions of the reference picture data for sample prediction).
42.17.4. H.265 Frame, Picture, Slice Segments, and Tiles
H.265 pictures consist of one or more slices, slice segments, and tiles, as defined in section 6.3.1 of the ITU-T H.265 Specification.
Video encode operations using an H.265 encode
profile can encode slice segments of different types, as defined in
section 7.4.7.1 of the ITU-T H.265 Specification, by
specifying the corresponding enumeration constant value in
StdVideoEncodeH265SliceSegmentHeader::slice_type in the
H.265 slice segment header
parameters from the Video Std enumeration type StdVideoH265SliceType:
-
STD_VIDEO_H265_SLICE_TYPE_Bindicates that the slice segment is part of a B slice as defined in section 3.12 of the ITU-T H.265 Specification. -
STD_VIDEO_H265_SLICE_TYPE_Pindicates that the slice segment is part of a P slice as defined in section 3.111 of the ITU-T H.265 Specification. -
STD_VIDEO_H265_SLICE_TYPE_Iindicates that the slice segment is part of an I slice as defined in section 3.74 of the ITU-T H.265 Specification.
Pictures constructed from such slice segments can be of different types, as
defined in section 7.4.3.5 of the ITU-T H.265 Specification.
Video encode operations using an H.265 encode
profile can encode pictures of a specific type by specifying the
corresponding enumeration constant value in
StdVideoEncodeH265PictureInfo::pic_type in the
H.265 picture information from the Video Std
enumeration type StdVideoH265PictureType:
-
STD_VIDEO_H265_PICTURE_TYPE_Pindicates that the picture is a P picture. A frame consisting of a P picture is also referred to as a P frame. -
STD_VIDEO_H265_PICTURE_TYPE_Bindicates that the picture is a B picture. A frame consisting of a B picture is also referred to as a B frame. -
STD_VIDEO_H265_PICTURE_TYPE_Iindicates that the picture is an I picture. A frame consisting of an I picture is also referred to as an I frame. -
STD_VIDEO_H265_PICTURE_TYPE_IDRindicates that the picture is a special type of I picture called an IDR picture as defined in section 3.67 of the ITU-T H.265 Specification. A frame consisting of an IDR picture is also referred to as an IDR frame.
42.17.5. H.265 Encode Profile
A video profile supporting H.265 video encode operations is specified by
setting VkVideoProfileInfoKHR::videoCodecOperation to
VK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR and adding a
VkVideoEncodeH265ProfileInfoKHR structure to the
VkVideoProfileInfoKHR::pNext chain.
The VkVideoEncodeH265ProfileInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265ProfileInfoKHR {
VkStructureType sType;
const void* pNext;
StdVideoH265ProfileIdc stdProfileIdc;
} VkVideoEncodeH265ProfileInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
stdProfileIdcis aStdVideoH265ProfileIdcvalue specifying the H.265 codec profile IDC, as defined in section A.3 of the ITU-T H.265 Specification.
42.17.6. H.265 Encode Capabilities
When calling vkGetPhysicalDeviceVideoCapabilitiesKHR to query the
capabilities for an H.265 encode profile, the
VkVideoCapabilitiesKHR::pNext chain must include a
VkVideoEncodeH265CapabilitiesKHR structure that will be filled with
the profile-specific capabilities.
The VkVideoEncodeH265CapabilitiesKHR structure is defined as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265CapabilitiesKHR {
VkStructureType sType;
void* pNext;
VkVideoEncodeH265CapabilityFlagsKHR flags;
StdVideoH265LevelIdc maxLevelIdc;
uint32_t maxSliceSegmentCount;
VkExtent2D maxTiles;
VkVideoEncodeH265CtbSizeFlagsKHR ctbSizes;
VkVideoEncodeH265TransformBlockSizeFlagsKHR transformBlockSizes;
uint32_t maxPPictureL0ReferenceCount;
uint32_t maxBPictureL0ReferenceCount;
uint32_t maxL1ReferenceCount;
uint32_t maxSubLayerCount;
VkBool32 expectDyadicTemporalSubLayerPattern;
int32_t minQp;
int32_t maxQp;
VkBool32 prefersGopRemainingFrames;
VkBool32 requiresGopRemainingFrames;
VkVideoEncodeH265StdFlagsKHR stdSyntaxFlags;
} VkVideoEncodeH265CapabilitiesKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
flagsis a bitmask of VkVideoEncodeH265CapabilityFlagBitsKHR indicating supported H.265 encoding capabilities. -
maxLevelIdcis aStdVideoH265LevelIdcvalue indicating the maximum H.265 level supported by the profile, where enum constantSTD_VIDEO_H265_LEVEL_IDC_<major>_<minor>identifies H.265 level<major>.<minor>as defined in section A.4 of the ITU-T H.265 Specification. -
maxSliceSegmentCountindicates the maximum number of slice segments that can be encoded for a single picture. Further restrictions may apply to the number of slice segments that can be encoded for a single picture depending on other capabilities and codec-specific rules. -
maxTilesindicates the maximum number of H.265 tile columns and rows, as defined in sections 3.175 and 3.176 of the ITU-T H.265 Specification that can be encoded for a single picture. Further restrictions may apply to the number of H.265 tiles that can be encoded for a single picture depending on other capabilities and codec-specific rules. -
ctbSizesis a bitmask of VkVideoEncodeH265CtbSizeFlagBitsKHR describing the supported CTB sizes. -
transformBlockSizesis a bitmask of VkVideoEncodeH265TransformBlockSizeFlagBitsKHR describing the supported transform block sizes. -
maxPPictureL0ReferenceCountindicates the maximum number of reference pictures the implementation supports in the reference list L0 for P pictures.NoteAs implementations may override the reference lists,
maxPPictureL0ReferenceCountdoes not limit the number of elements that the application can specify in the L0 reference list for P pictures. However, ifmaxPPictureL0ReferenceCountis zero, then the use of P pictures is not allowed. In case of H.265 encoding, backward-only predictive pictures can be encoded even if P pictures are not supported, as the ITU-T H.265 Specification supports generalized P & B frames (also known as low delay B frames) whereas B frames can refer to past frames through both the L0 and L1 reference lists. -
maxBPictureL0ReferenceCountindicates the maximum number of reference pictures the implementation supports in the reference list L0 for B pictures. -
maxL1ReferenceCountindicates the maximum number of reference pictures the implementation supports in the reference list L1 if encoding of B pictures is supported.NoteAs implementations may override the reference lists,
maxBPictureL0ReferenceCountandmaxL1ReferenceCountdoes not limit the number of elements that the application can specify in the L0 and L1 reference lists for B pictures. However, ifmaxBPictureL0ReferenceCountandmaxL1ReferenceCountare both zero, then the use of B pictures is not allowed. -
maxSubLayerCountindicates the maximum number of H.265 sub-layers supported by the implementation. -
expectDyadicTemporalSubLayerPatternindicates that the implementation’s rate control algorithms expect the application to use a dyadic temporal sub-layer pattern when encoding multiple temporal sub-layers. -
minQpindicates the minimum QP value supported. -
maxQpindicates the maximum QP value supported. -
prefersGopRemainingFramesindicates that the implementation’s rate control algorithm prefers the application to specify the number of frames of each type remaining in the current group of pictures when beginning a video coding scope. -
requiresGopRemainingFramesindicates that the implementation’s rate control algorithm requires the application to specify the number of frames of each type remaining in the current group of pictures when beginning a video coding scope. -
stdSyntaxFlagsis a bitmask of VkVideoEncodeH265StdFlagBitsKHR indicating capabilities related to H.265 syntax elements.
Bits which may be set in
VkVideoEncodeH265CapabilitiesKHR::flags, indicating the H.265
encoding capabilities supported, are:
// Provided by VK_KHR_video_encode_h265
typedef enum VkVideoEncodeH265CapabilityFlagBitsKHR {
VK_VIDEO_ENCODE_H265_CAPABILITY_HRD_COMPLIANCE_BIT_KHR = 0x00000001,
VK_VIDEO_ENCODE_H265_CAPABILITY_PREDICTION_WEIGHT_TABLE_GENERATED_BIT_KHR = 0x00000002,
VK_VIDEO_ENCODE_H265_CAPABILITY_ROW_UNALIGNED_SLICE_SEGMENT_BIT_KHR = 0x00000004,
VK_VIDEO_ENCODE_H265_CAPABILITY_DIFFERENT_SLICE_SEGMENT_TYPE_BIT_KHR = 0x00000008,
VK_VIDEO_ENCODE_H265_CAPABILITY_B_FRAME_IN_L0_LIST_BIT_KHR = 0x00000010,
VK_VIDEO_ENCODE_H265_CAPABILITY_B_FRAME_IN_L1_LIST_BIT_KHR = 0x00000020,
VK_VIDEO_ENCODE_H265_CAPABILITY_PER_PICTURE_TYPE_MIN_MAX_QP_BIT_KHR = 0x00000040,
VK_VIDEO_ENCODE_H265_CAPABILITY_PER_SLICE_SEGMENT_CONSTANT_QP_BIT_KHR = 0x00000080,
VK_VIDEO_ENCODE_H265_CAPABILITY_MULTIPLE_TILES_PER_SLICE_SEGMENT_BIT_KHR = 0x00000100,
VK_VIDEO_ENCODE_H265_CAPABILITY_MULTIPLE_SLICE_SEGMENTS_PER_TILE_BIT_KHR = 0x00000200,
} VkVideoEncodeH265CapabilityFlagBitsKHR;
-
VK_VIDEO_ENCODE_H265_CAPABILITY_HRD_COMPLIANCE_BIT_KHRindicates if the implementation may be able to generate HRD compliant bitstreams if any of thenal_hrd_parameters_present_flag,vcl_hrd_parameters_present_flag, orsub_pic_hrd_params_present_flagmembers ofStdVideoH265HrdFlagsare set to1in the HRD parameters of the active VPS or active SPS, or ifStdVideoH265SpsVuiFlags::vui_hrd_parameters_present_flagis set to1in the active SPS. -
VK_VIDEO_ENCODE_H265_CAPABILITY_PREDICTION_WEIGHT_TABLE_GENERATED_BIT_KHRindicates that if theweighted_pred_flagor theweighted_bipred_flagmember ofStdVideoH265PpsFlagsis set to1in the active PPS when encoding a P picture or B picture, respectively, then the implementation is able to internally decide syntax forpred_weight_table, as defined in section 7.4.7.3 of the ITU-T H.265 Specification, and the application is not required to provide a weight table in the H.265 slice segment header parameters. -
VK_VIDEO_ENCODE_H265_CAPABILITY_ROW_UNALIGNED_SLICE_SEGMENT_BIT_KHRindicates that each slice segment in a frame with a single or multiple tiles per slice may begin or finish at any offset in a CTB row. If not supported, all slice segments in such a frame must begin at the start of a CTB row (and hence each slice segment must finish at the end of a CTB row). Also indicates that each slice segment in a frame with multiple slices per tile may begin or finish at any offset within the enclosing tile’s CTB row. If not supported, slice segments in such a frame must begin at the start of the enclosing tile’s CTB row (and hence each slice segment must finish at the end of the enclosing tile’s CTB row). -
VK_VIDEO_ENCODE_H265_CAPABILITY_DIFFERENT_SLICE_SEGMENT_TYPE_BIT_KHRindicates that when a frame is encoded with multiple slice segments, the implementation allows encoding each slice segment with a differentStdVideoEncodeH265SliceSegmentHeader::slice_typespecified in the H.265 slice segment header parameters. If not supported, all slice segments of the frame must be encoded with the sameslice_typewhich corresponds to the picture type of the frame. -
VK_VIDEO_ENCODE_H265_CAPABILITY_B_FRAME_IN_L0_LIST_BIT_KHRindicates support for using a B frame as L0 reference, as specified inStdVideoEncodeH265ReferenceListsInfo::RefPicList0in the H.265 picture information. -
VK_VIDEO_ENCODE_H265_CAPABILITY_B_FRAME_IN_L1_LIST_BIT_KHRindicates support for using a B frame as L1 reference, as specified inStdVideoEncodeH265ReferenceListsInfo::RefPicList1in the H.265 picture information. -
VK_VIDEO_ENCODE_H265_CAPABILITY_PER_PICTURE_TYPE_MIN_MAX_QP_BIT_KHRindicates support for specifying different QP values in the members of VkVideoEncodeH265QpKHR. -
VK_VIDEO_ENCODE_H265_CAPABILITY_PER_SLICE_SEGMENT_CONSTANT_QP_BIT_KHRindicates support for specifying different constant QP values for each slice segment. -
VK_VIDEO_ENCODE_H265_CAPABILITY_MULTIPLE_TILES_PER_SLICE_SEGMENT_BIT_KHRindicates if encoding multiple tiles per slice segment, as defined in section 6.3.1 of the ITU-T H.265 Specification, is supported. If this capability flag is not present, then the implementation is only able to encode a single tile for each slice segment. -
VK_VIDEO_ENCODE_H265_CAPABILITY_MULTIPLE_SLICE_SEGMENTS_PER_TILE_BIT_KHRindicates if encoding multiple slice segments per tile, as defined in section 6.3.1 of the ITU-T H.265 Specification, is supported. If this capability flag is not present, then the implementation is only able to encode a single slice segment for each tile.
// Provided by VK_KHR_video_encode_h265
typedef VkFlags VkVideoEncodeH265CapabilityFlagsKHR;
VkVideoEncodeH265CapabilityFlagsKHR is a bitmask type for setting a
mask of zero or more VkVideoEncodeH265CapabilityFlagBitsKHR.
Bits which may be set in
VkVideoEncodeH265CapabilitiesKHR::stdSyntaxFlags, indicating the
capabilities related to the H.265 syntax elements, are:
// Provided by VK_KHR_video_encode_h265
typedef enum VkVideoEncodeH265StdFlagBitsKHR {
VK_VIDEO_ENCODE_H265_STD_SEPARATE_COLOR_PLANE_FLAG_SET_BIT_KHR = 0x00000001,
VK_VIDEO_ENCODE_H265_STD_SAMPLE_ADAPTIVE_OFFSET_ENABLED_FLAG_SET_BIT_KHR = 0x00000002,
VK_VIDEO_ENCODE_H265_STD_SCALING_LIST_DATA_PRESENT_FLAG_SET_BIT_KHR = 0x00000004,
VK_VIDEO_ENCODE_H265_STD_PCM_ENABLED_FLAG_SET_BIT_KHR = 0x00000008,
VK_VIDEO_ENCODE_H265_STD_SPS_TEMPORAL_MVP_ENABLED_FLAG_SET_BIT_KHR = 0x00000010,
VK_VIDEO_ENCODE_H265_STD_INIT_QP_MINUS26_BIT_KHR = 0x00000020,
VK_VIDEO_ENCODE_H265_STD_WEIGHTED_PRED_FLAG_SET_BIT_KHR = 0x00000040,
VK_VIDEO_ENCODE_H265_STD_WEIGHTED_BIPRED_FLAG_SET_BIT_KHR = 0x00000080,
VK_VIDEO_ENCODE_H265_STD_LOG2_PARALLEL_MERGE_LEVEL_MINUS2_BIT_KHR = 0x00000100,
VK_VIDEO_ENCODE_H265_STD_SIGN_DATA_HIDING_ENABLED_FLAG_SET_BIT_KHR = 0x00000200,
VK_VIDEO_ENCODE_H265_STD_TRANSFORM_SKIP_ENABLED_FLAG_SET_BIT_KHR = 0x00000400,
VK_VIDEO_ENCODE_H265_STD_TRANSFORM_SKIP_ENABLED_FLAG_UNSET_BIT_KHR = 0x00000800,
VK_VIDEO_ENCODE_H265_STD_PPS_SLICE_CHROMA_QP_OFFSETS_PRESENT_FLAG_SET_BIT_KHR = 0x00001000,
VK_VIDEO_ENCODE_H265_STD_TRANSQUANT_BYPASS_ENABLED_FLAG_SET_BIT_KHR = 0x00002000,
VK_VIDEO_ENCODE_H265_STD_CONSTRAINED_INTRA_PRED_FLAG_SET_BIT_KHR = 0x00004000,
VK_VIDEO_ENCODE_H265_STD_ENTROPY_CODING_SYNC_ENABLED_FLAG_SET_BIT_KHR = 0x00008000,
VK_VIDEO_ENCODE_H265_STD_DEBLOCKING_FILTER_OVERRIDE_ENABLED_FLAG_SET_BIT_KHR = 0x00010000,
VK_VIDEO_ENCODE_H265_STD_DEPENDENT_SLICE_SEGMENTS_ENABLED_FLAG_SET_BIT_KHR = 0x00020000,
VK_VIDEO_ENCODE_H265_STD_DEPENDENT_SLICE_SEGMENT_FLAG_SET_BIT_KHR = 0x00040000,
VK_VIDEO_ENCODE_H265_STD_SLICE_QP_DELTA_BIT_KHR = 0x00080000,
VK_VIDEO_ENCODE_H265_STD_DIFFERENT_SLICE_QP_DELTA_BIT_KHR = 0x00100000,
} VkVideoEncodeH265StdFlagBitsKHR;
-
VK_VIDEO_ENCODE_H265_STD_SEPARATE_COLOR_PLANE_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265SpsFlags::separate_colour_plane_flagin the SPS when that value is1. -
VK_VIDEO_ENCODE_H265_STD_SAMPLE_ADAPTIVE_OFFSET_ENABLED_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265SpsFlags::sample_adaptive_offset_enabled_flagin the SPS when that value is1. -
VK_VIDEO_ENCODE_H265_STD_SCALING_LIST_DATA_PRESENT_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value for thescaling_list_enabled_flagandsps_scaling_list_data_present_flagmembers ofStdVideoH265SpsFlagsin the SPS, and the application-provided value forStdVideoH265PpsFlags::pps_scaling_list_data_present_flagin the PPS when those values are1. -
VK_VIDEO_ENCODE_H265_STD_PCM_ENABLED_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265SpsFlags::pcm_enable_flagin the SPS when that value is1. -
VK_VIDEO_ENCODE_H265_STD_SPS_TEMPORAL_MVP_ENABLED_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265SpsFlags::sps_temporal_mvp_enabled_flagin the SPS when that value is1. -
VK_VIDEO_ENCODE_H265_STD_INIT_QP_MINUS26_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265PictureParameterSet::init_qp_minus26in the PPS when that value is non-zero. -
VK_VIDEO_ENCODE_H265_STD_WEIGHTED_PRED_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265PpsFlags::weighted_pred_flagin the PPS when that value is1. -
VK_VIDEO_ENCODE_H265_STD_WEIGHTED_BIPRED_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265PpsFlags::weighted_bipred_flagin the PPS when that value is1. -
VK_VIDEO_ENCODE_H265_STD_LOG2_PARALLEL_MERGE_LEVEL_MINUS2_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265PictureParameterSet::log2_parallel_merge_level_minus2in the PPS when that value is non-zero. -
VK_VIDEO_ENCODE_H265_STD_SIGN_DATA_HIDING_ENABLED_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265PpsFlags::sign_data_hiding_enabled_flagin the PPS when that value is1. -
VK_VIDEO_ENCODE_H265_STD_TRANSFORM_SKIP_ENABLED_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265PpsFlags::transform_skip_enabled_flagin the PPS when that value is1. -
VK_VIDEO_ENCODE_H265_STD_TRANSFORM_SKIP_ENABLED_FLAG_UNSET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265PpsFlags::transform_skip_enabled_flagin the PPS when that value is0. -
VK_VIDEO_ENCODE_H265_STD_PPS_SLICE_CHROMA_QP_OFFSETS_PRESENT_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265PpsFlags::pps_slice_chroma_qp_offsets_present_flagin the PPS when that value is1. -
VK_VIDEO_ENCODE_H265_STD_TRANSQUANT_BYPASS_ENABLED_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265PpsFlags::transquant_bypass_enabled_flagin the PPS when that value is1. -
VK_VIDEO_ENCODE_H265_STD_CONSTRAINED_INTRA_PRED_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265PpsFlags::constrained_intra_pred_flagin the PPS when that value is1. -
VK_VIDEO_ENCODE_H265_STD_ENTROPY_CODING_SYNC_ENABLED_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265PpsFlags::entropy_coding_sync_enabled_flagin the PPS when that value is1. -
VK_VIDEO_ENCODE_H265_STD_DEBLOCKING_FILTER_OVERRIDE_ENABLED_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265PpsFlags::deblocking_filter_override_enabled_flagin the PPS when that value is1. -
VK_VIDEO_ENCODE_H265_STD_DEPENDENT_SLICE_SEGMENTS_ENABLED_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoH265PpsFlags::dependent_slice_segments_enabled_flagin the PPS when that value is1. -
VK_VIDEO_ENCODE_H265_STD_DEPENDENT_SLICE_SEGMENT_FLAG_SET_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoEncodeH265SliceSegmentHeader::dependent_slice_segment_flagin the H.265 slice segment header parameters when that value is1. -
VK_VIDEO_ENCODE_H265_STD_SLICE_QP_DELTA_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoEncodeH265SliceSegmentHeader::slice_qp_deltain the H.265 slice segment header parameters when that value is identical across the slice segments of the encoded frame. -
VK_VIDEO_ENCODE_H265_STD_DIFFERENT_SLICE_QP_DELTA_BIT_KHRindicates whether the implementation supports using the application-provided value forStdVideoEncodeH265SliceSegmentHeader::slice_qp_deltain the H.265 slice segment header parameters when that value is different across the slice segments of the encoded frame.
These capability flags provide information to the application about specific H.265 syntax element values that the implementation supports without having to override them and do not otherwise restrict the values that the application can specify for any of the mentioned H.265 syntax elements.
// Provided by VK_KHR_video_encode_h265
typedef VkFlags VkVideoEncodeH265StdFlagsKHR;
VkVideoEncodeH265StdFlagsKHR is a bitmask type for setting a mask of
zero or more VkVideoEncodeH265StdFlagBitsKHR.
Bits which may be set in
VkVideoEncodeH265CapabilitiesKHR::ctbSizes, indicating the CTB
sizes supported by the implementation, are:
// Provided by VK_KHR_video_encode_h265
typedef enum VkVideoEncodeH265CtbSizeFlagBitsKHR {
VK_VIDEO_ENCODE_H265_CTB_SIZE_16_BIT_KHR = 0x00000001,
VK_VIDEO_ENCODE_H265_CTB_SIZE_32_BIT_KHR = 0x00000002,
VK_VIDEO_ENCODE_H265_CTB_SIZE_64_BIT_KHR = 0x00000004,
} VkVideoEncodeH265CtbSizeFlagBitsKHR;
-
VK_VIDEO_ENCODE_H265_CTB_SIZE_16_BIT_KHRspecifies that a CTB size of 16x16 is supported. -
VK_VIDEO_ENCODE_H265_CTB_SIZE_32_BIT_KHRspecifies that a CTB size of 32x32 is supported. -
VK_VIDEO_ENCODE_H265_CTB_SIZE_64_BIT_KHRspecifies that a CTB size of 64x64 is supported.
// Provided by VK_KHR_video_encode_h265
typedef VkFlags VkVideoEncodeH265CtbSizeFlagsKHR;
VkVideoEncodeH265CtbSizeFlagsKHR is a bitmask type for setting a mask
of zero or more VkVideoEncodeH265CtbSizeFlagBitsKHR.
Implementations must support at least one of
VkVideoEncodeH265CtbSizeFlagBitsKHR.
Bits which may be set in
VkVideoEncodeH265CapabilitiesKHR::transformBlockSizes,
indicating the transform block sizes supported by the implementation, are:
// Provided by VK_KHR_video_encode_h265
typedef enum VkVideoEncodeH265TransformBlockSizeFlagBitsKHR {
VK_VIDEO_ENCODE_H265_TRANSFORM_BLOCK_SIZE_4_BIT_KHR = 0x00000001,
VK_VIDEO_ENCODE_H265_TRANSFORM_BLOCK_SIZE_8_BIT_KHR = 0x00000002,
VK_VIDEO_ENCODE_H265_TRANSFORM_BLOCK_SIZE_16_BIT_KHR = 0x00000004,
VK_VIDEO_ENCODE_H265_TRANSFORM_BLOCK_SIZE_32_BIT_KHR = 0x00000008,
} VkVideoEncodeH265TransformBlockSizeFlagBitsKHR;
-
VK_VIDEO_ENCODE_H265_TRANSFORM_BLOCK_SIZE_4_BIT_KHRspecifies that a transform block size of 4x4 is supported. -
VK_VIDEO_ENCODE_H265_TRANSFORM_BLOCK_SIZE_8_BIT_KHRspecifies that a transform block size of 8x8 is supported. -
VK_VIDEO_ENCODE_H265_TRANSFORM_BLOCK_SIZE_16_BIT_KHRspecifies that a transform block size of 16x16 is supported. -
VK_VIDEO_ENCODE_H265_TRANSFORM_BLOCK_SIZE_32_BIT_KHRspecifies that a transform block size of 32x32 is supported.
// Provided by VK_KHR_video_encode_h265
typedef VkFlags VkVideoEncodeH265TransformBlockSizeFlagsKHR;
VkVideoEncodeH265TransformBlockSizeFlagsKHR is a bitmask type for
setting a mask of zero or more
VkVideoEncodeH265TransformBlockSizeFlagBitsKHR.
Implementations must support at least one of
VkVideoEncodeH265TransformBlockSizeFlagBitsKHR.
42.17.7. H.265 Encode Quality Level Properties
When calling vkGetPhysicalDeviceVideoEncodeQualityLevelPropertiesKHR
with pVideoProfile->videoCodecOperation specified as
VK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR, the
VkVideoEncodeH265QualityLevelPropertiesKHR structure must be included
in the pNext chain of the VkVideoEncodeQualityLevelPropertiesKHR
structure to retrieve additional video encode quality level properties
specific to H.265 encoding.
The VkVideoEncodeH265QualityLevelPropertiesKHR structure is defined as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265QualityLevelPropertiesKHR {
VkStructureType sType;
void* pNext;
VkVideoEncodeH265RateControlFlagsKHR preferredRateControlFlags;
uint32_t preferredGopFrameCount;
uint32_t preferredIdrPeriod;
uint32_t preferredConsecutiveBFrameCount;
uint32_t preferredSubLayerCount;
VkVideoEncodeH265QpKHR preferredConstantQp;
uint32_t preferredMaxL0ReferenceCount;
uint32_t preferredMaxL1ReferenceCount;
} VkVideoEncodeH265QualityLevelPropertiesKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
preferredRateControlFlagsis a bitmask of VkVideoEncodeH265RateControlFlagBitsKHR values indicating the preferred flags to use for VkVideoEncodeH265RateControlInfoKHR::flags. -
preferredGopFrameCountindicates the preferred value to use for VkVideoEncodeH265RateControlInfoKHR::gopFrameCount. -
preferredIdrPeriodindicates the preferred value to use for VkVideoEncodeH265RateControlInfoKHR::idrPeriod. -
preferredConsecutiveBFrameCountindicates the preferred value to use for VkVideoEncodeH265RateControlInfoKHR::consecutiveBFrameCount. -
preferredSubLayerCountindicates the preferred value to use for VkVideoEncodeH265RateControlInfoKHR::subLayerCount. -
preferredConstantQpindicates the preferred values to use for VkVideoEncodeH265NaluSliceSegmentInfoKHR::constantQpfor each picture type when using rate control modeVK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR. -
preferredMaxL0ReferenceCountindicates the preferred maximum number of reference pictures to use in the reference list L0. -
preferredMaxL1ReferenceCountindicates the preferred maximum number of reference pictures to use in the reference list L1.
42.17.8. H.265 Encode Session
Additional parameters can be specified when creating a video session with an
H.265 encode profile by including an instance of the
VkVideoEncodeH265SessionCreateInfoKHR structure in the pNext
chain of VkVideoSessionCreateInfoKHR.
The VkVideoEncodeH265SessionCreateInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265SessionCreateInfoKHR {
VkStructureType sType;
const void* pNext;
VkBool32 useMaxLevelIdc;
StdVideoH265LevelIdc maxLevelIdc;
} VkVideoEncodeH265SessionCreateInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
useMaxLevelIdcindicates whether the value ofmaxLevelIdcshould be used by the implementation. When it is set toVK_FALSE, the implementation ignores the value ofmaxLevelIdcand uses the value of VkVideoEncodeH265CapabilitiesKHR::maxLevelIdc, as reported by vkGetPhysicalDeviceVideoCapabilitiesKHR for the video profile. -
maxLevelIdcis aStdVideoH265LevelIdcvalue specifying the upper bound on the H.265 level for the video bitstreams produced by the created video session, where enum constantSTD_VIDEO_H265_LEVEL_IDC_<major>_<minor>identifies H.265 level<major>.<minor>as defined in section A.4 of the ITU-T H.265 Specification.
42.17.9. H.265 Encode Parameter Sets
Video session parameters objects created with
the video codec operation VK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR
can contain the following types of parameters:
- H.265 Video Parameter Sets (VPS)
-
Represented by
StdVideoH265VideoParameterSetstructures and interpreted as follows:-
reserved1,reserved2, andreserved3are used only for padding purposes and are otherwise ignored; -
vps_video_parameter_set_idis used as the key of the VPS entry; -
the
max_latency_increase_plus1,max_dec_pic_buffering_minus1, andmax_num_reorder_picsmembers of theStdVideoH265DecPicBufMgrstructure pointed to bypDecPicBufMgrcorrespond tovps_max_latency_increase_plus1,vps_max_dec_pic_buffering_minus1, andvps_max_num_reorder_pics, respectively, as defined in section 7.4.3.1 of the ITU-T H.265 Specification; -
the
StdVideoH265HrdParametersstructure pointed to bypHrdParametersis interpreted as follows:-
reservedis used only for padding purposes and is otherwise ignored; -
flags.fixed_pic_rate_general_flagis a bitmask where bit index i corresponds tofixed_pic_rate_general_flag[i]as defined in section E.3.2 of the ITU-T H.265 Specification; -
flags.fixed_pic_rate_within_cvs_flagis a bitmask where bit index i corresponds tofixed_pic_rate_within_cvs_flag[i]as defined in section E.3.2 of the ITU-T H.265 Specification; -
flags.low_delay_hrd_flagis a bitmask where bit index i corresponds tolow_delay_hrd_flag[i]as defined in section E.3.2 of the ITU-T H.265 Specification; -
if
flags.nal_hrd_parameters_present_flagis set, thenpSubLayerHrdParametersNalis a pointer to an array ofvps_max_sub_layers_minus1+ 1 number ofStdVideoH265SubLayerHrdParametersstructures wherevps_max_sub_layers_minus1is the corresponding member of the encompassingStdVideoH265VideoParameterSetstructure and each element is interpreted as follows:-
cbr_flagis a bitmask where bit index i corresponds tocbr_flag[i]as defined in section E.3.3 of the ITU-T H.265 Specification; -
all other members of the
StdVideoH265SubLayerHrdParametersstructure are interpreted as defined in section E.3.3 of the ITU-T H.265 Specification;
-
-
if
flags.vcl_hrd_parameters_present_flagis set, thenpSubLayerHrdParametersVclis a pointer to an array ofvps_max_sub_layers_minus1+ 1 number ofStdVideoH265SubLayerHrdParametersstructures wherevps_max_sub_layers_minus1is the corresponding member of the encompassingStdVideoH265VideoParameterSetstructure and each element is interpreted as follows:-
cbr_flagis a bitmask where bit index i corresponds tocbr_flag[i]as defined in section E.3.3 of the ITU-T H.265 Specification; -
all other members of the
StdVideoH265SubLayerHrdParametersstructure are interpreted as defined in section E.3.3 of the ITU-T H.265 Specification;
-
-
all other members of
StdVideoH265HrdParametersare interpreted as defined in section E.3.2 of the ITU-T H.265 Specification;
-
-
the
StdVideoH265ProfileTierLevelstructure pointed to bypProfileTierLevelare interpreted as follows:-
general_level_idcis one of the enum constantsSTD_VIDEO_H265_LEVEL_IDC_<major>_<minor>identifying the H.265 level<major>.<minor>as defined in section A.4 of the ITU-T H.265 Specification; -
all other members of
StdVideoH265ProfileTierLevelare interpreted as defined in section 7.4.4 of the ITU-T H.265 Specification;
-
-
all other members of
StdVideoH265VideoParameterSetare interpreted as defined in section 7.4.3.1 of the ITU-T H.265 Specification.
-
- H.265 Sequence Parameter Sets (SPS)
-
Represented by
StdVideoH265SequenceParameterSetstructures and interpreted as follows:-
reserved1andreserved2are used only for padding purposes and are otherwise ignored; -
the pair constructed from
sps_video_parameter_set_idandsps_seq_parameter_set_idis used as the key of the SPS entry; -
the
StdVideoH265ProfileTierLevelstructure pointed to bypProfileTierLevelare interpreted as follows:-
general_level_idcis one of the enum constantsSTD_VIDEO_H265_LEVEL_IDC_<major>_<minor>identifying the H.265 level<major>.<minor>as defined in section A.4 of the ITU-T H.265 Specification; -
all other members of
StdVideoH265ProfileTierLevelare interpreted as defined in section 7.4.4 of the ITU-T H.265 Specification;
-
-
the
max_latency_increase_plus1,max_dec_pic_buffering_minus1, andmax_num_reorder_picsmembers of theStdVideoH265DecPicBufMgrstructure pointed to bypDecPicBufMgrcorrespond tosps_max_latency_increase_plus1,sps_max_dec_pic_buffering_minus1, andsps_max_num_reorder_pics, respectively, as defined in section 7.4.3.2 of the ITU-T H.265 Specification; -
if
flags.sps_scaling_list_data_present_flagis set, then theStdVideoH265ScalingListsstructure pointed to bypScalingListsis interpreted as follows:-
ScalingList4x4,ScalingList8x8,ScalingList16x16, andScalingList32x32correspond toScalingList[0],ScalingList[1],ScalingList[2], andScalingList[3], respectively, as defined in section 7.3.4 of the ITU-T H.265 Specification; -
ScalingListDCCoef16x16andScalingListDCCoef32x32correspond toscaling_list_dc_coef_minus8[0]andscaling_list_dc_coef_minus8[1], respectively, as defined in section 7.3.4 of the ITU-T H.265 Specification;
-
-
pShortTermRefPicSetis a pointer to an array ofnum_short_term_ref_pic_setsnumber ofStdVideoH265ShortTermRefPicSetstructures where each element is interpreted as follows:-
reserved1,reserved2, andreserved3are used only for padding purposes and are otherwise ignored; -
used_by_curr_pic_flagis a bitmask where bit index i corresponds toused_by_curr_pic_flag[i]as defined in section 7.4.8 of the ITU-T H.265 Specification; -
use_delta_flagis a bitmask where bit index i corresponds touse_delta_flag[i]as defined in section 7.4.8 of the ITU-T H.265 Specification; -
used_by_curr_pic_s0_flagis a bitmask where bit index i corresponds toused_by_curr_pic_s0_flag[i]as defined in section 7.4.8 of the ITU-T H.265 Specification; -
used_by_curr_pic_s1_flagis a bitmask where bit index i corresponds toused_by_curr_pic_s1_flag[i]as defined in section 7.4.8 of the ITU-T H.265 Specification; -
all other members of
StdVideoH265ShortTermRefPicSetare interpreted as defined in section 7.4.8 of the ITU-T H.265 Specification;
-
-
if
flags.long_term_ref_pics_present_flagis set then theStdVideoH265LongTermRefPicsSpsstructure pointed to bypLongTermRefPicsSpsis interpreted as follows:-
used_by_curr_pic_lt_sps_flagis a bitmask where bit index i corresponds toused_by_curr_pic_lt_sps_flag[i]as defined in section 7.4.3.2 of the ITU-T H.265 Specification; -
all other members of
StdVideoH265LongTermRefPicsSpsare interpreted as defined in section 7.4.3.2 of the ITU-T H.265 Specification;
-
-
if
flags.vui_parameters_present_flagis set, then theStdVideoH265SequenceParameterSetVuistructure pointed to bypSequenceParameterSetVuiis interpreted as follows:-
reserved1,reserved2, andreserved3are used only for padding purposes and are otherwise ignored; -
the
StdVideoH265HrdParametersstructure pointed to bypHrdParametersis interpreted as follows:-
flags.fixed_pic_rate_general_flagis a bitmask where bit index i corresponds tofixed_pic_rate_general_flag[i]as defined in section E.3.2 of the ITU-T H.265 Specification; -
flags.fixed_pic_rate_within_cvs_flagis a bitmask where bit index i corresponds tofixed_pic_rate_within_cvs_flag[i]as defined in section E.3.2 of the ITU-T H.265 Specification; -
flags.low_delay_hrd_flagis a bitmask where bit index i corresponds tolow_delay_hrd_flag[i]as defined in section E.3.2 of the ITU-T H.265 Specification; -
if
flags.nal_hrd_parameters_present_flagis set, thenpSubLayerHrdParametersNalis a pointer to an array ofsps_max_sub_layers_minus1+ 1 number ofStdVideoH265SubLayerHrdParametersstructures wheresps_max_sub_layers_minus1is the corresponding member of the encompassingStdVideoH265SequenceParameterSetstructure and each element is interpreted as follows:-
cbr_flagis a bitmask where bit index i corresponds tocbr_flag[i]as defined in section E.3.3 of the ITU-T H.265 Specification; -
all other members of the
StdVideoH265SubLayerHrdParametersstructure are interpreted as defined in section E.3.3 of the ITU-T H.265 Specification;
-
-
if
flags.vcl_hrd_parameters_present_flagis set, thenpSubLayerHrdParametersVclis a pointer to an array ofsps_max_sub_layers_minus1+ 1 number ofStdVideoH265SubLayerHrdParametersstructures wheresps_max_sub_layers_minus1is the corresponding member of the encompassingStdVideoH265SequenceParameterSetstructure and each element is interpreted as follows:-
cbr_flagis a bitmask where bit index i corresponds tocbr_flag[i]as defined in section E.3.3 of the ITU-T H.265 Specification; -
all other members of the
StdVideoH265SubLayerHrdParametersstructure are interpreted as defined in section E.3.3 of the ITU-T H.265 Specification;
-
-
all other members of
StdVideoH265HrdParametersare interpreted as defined in section E.3.2 of the ITU-T H.265 Specification;
-
-
all other members of
pSequenceParameterSetVuiare interpreted as defined in section E.3.1 of the ITU-T H.265 Specification;
-
-
if
flags.sps_palette_predictor_initializer_present_flagis set, then thePredictorPaletteEntriesmember of theStdVideoH265PredictorPaletteEntriesstructure pointed to bypPredictorPaletteEntriesis interpreted as defined in section 7.4.9.13 of the ITU-T H.265 Specification; -
all other members of
StdVideoH265SequenceParameterSetare interpreted as defined in section 7.4.3.1 of the ITU-T H.265 Specification.
-
- H.265 Picture Parameter Sets (PPS)
-
Represented by
StdVideoH265PictureParameterSetstructures and interpreted as follows:-
reserved1,reserved2, andreserved3are used only for padding purposes and are otherwise ignored; -
the triplet constructed from
sps_video_parameter_set_id,pps_seq_parameter_set_id, andpps_pic_parameter_set_idis used as the key of the PPS entry; -
if
flags.pps_scaling_list_data_present_flagis set, then theStdVideoH265ScalingListsstructure pointed to bypScalingListsis interpreted as follows:-
ScalingList4x4,ScalingList8x8,ScalingList16x16, andScalingList32x32correspond toScalingList[0],ScalingList[1],ScalingList[2], andScalingList[3], respectively, as defined in section 7.3.4 of the ITU-T H.265 Specification; -
ScalingListDCCoef16x16andScalingListDCCoef32x32correspond toscaling_list_dc_coef_minus8[0]andscaling_list_dc_coef_minus8[1], respectively, as defined in section 7.3.4 of the ITU-T H.265 Specification;
-
-
if
flags.pps_palette_predictor_initializer_present_flagis set, then thePredictorPaletteEntriesmember of theStdVideoH265PredictorPaletteEntriesstructure pointed to bypPredictorPaletteEntriesis interpreted as defined in section 7.4.9.13 of the ITU-T H.265 Specification; -
all other members of
StdVideoH265PictureParameterSetare interpreted as defined in section 7.4.3.3 of the ITU-T H.265 Specification.
-
Implementations may override any of these parameters according to the semantics defined in the Video Encode Parameter Overrides section before storing the resulting H.265 parameter sets into the video session parameters object. Applications need to use the vkGetEncodedVideoSessionParametersKHR command to determine whether any implementation overrides happened and to retrieve the encoded H.265 parameter sets in order to be able to produce a compliant H.265 video bitstream.
Such H.265 parameter set overrides may also have cascading effects on the
implementation overrides applied to the encoded bitstream produced by video
encode operations.
If the implementation supports the
VK_VIDEO_ENCODE_FEEDBACK_BITSTREAM_HAS_OVERRIDES_BIT_KHR
video encode feedback query flag, then the
application can use such queries to retrieve feedback about whether any
implementation overrides have been applied to the encoded bitstream.
When a video session parameters object is
created with the codec operation
VK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR, the
VkVideoSessionParametersCreateInfoKHR::pNext chain must include
a VkVideoEncodeH265SessionParametersCreateInfoKHR structure specifying
the capacity and initial contents of the object.
The VkVideoEncodeH265SessionParametersCreateInfoKHR structure is
defined as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265SessionParametersCreateInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t maxStdVPSCount;
uint32_t maxStdSPSCount;
uint32_t maxStdPPSCount;
const VkVideoEncodeH265SessionParametersAddInfoKHR* pParametersAddInfo;
} VkVideoEncodeH265SessionParametersCreateInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
maxStdVPSCountis the maximum number of H.265 VPS entries the createdVkVideoSessionParametersKHRcan contain. -
maxStdSPSCountis the maximum number of H.265 SPS entries the createdVkVideoSessionParametersKHRcan contain. -
maxStdPPSCountis the maximum number of H.265 PPS entries the createdVkVideoSessionParametersKHRcan contain. -
pParametersAddInfoisNULLor a pointer to a VkVideoEncodeH265SessionParametersAddInfoKHR structure specifying H.265 parameters to add upon object creation.
The VkVideoEncodeH265SessionParametersAddInfoKHR structure is defined
as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265SessionParametersAddInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t stdVPSCount;
const StdVideoH265VideoParameterSet* pStdVPSs;
uint32_t stdSPSCount;
const StdVideoH265SequenceParameterSet* pStdSPSs;
uint32_t stdPPSCount;
const StdVideoH265PictureParameterSet* pStdPPSs;
} VkVideoEncodeH265SessionParametersAddInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
stdVPSCountis the number of elements in thepStdVPSsarray. -
pStdVPSsis a pointer to an array ofStdVideoH265VideoParameterSetstructures describing the H.265 VPS entries to add. -
stdSPSCountis the number of elements in thepStdSPSsarray. -
pStdSPSsis a pointer to an array ofStdVideoH265SequenceParameterSetstructures describing the H.265 SPS entries to add. -
stdPPSCountis the number of elements in thepStdPPSsarray. -
pStdPPSsis a pointer to an array ofStdVideoH265PictureParameterSetstructures describing the H.265 PPS entries to add.
This structure can be specified in the following places:
-
In the
pParametersAddInfomember of the VkVideoEncodeH265SessionParametersCreateInfoKHR structure specified in thepNextchain of VkVideoSessionParametersCreateInfoKHR used to create a video session parameters object. In this case, if the video codec operation the video session parameters object is created with isVK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR, then it defines the set of initial parameters to add to the created object (see Creating Video Session Parameters). -
In the
pNextchain of VkVideoSessionParametersUpdateInfoKHR. In this case, if the video codec operation the video session parameters object to be updated was created with isVK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR, then it defines the set of parameters to add to it (see Updating Video Session Parameters).
The VkVideoEncodeH265SessionParametersGetInfoKHR structure is defined
as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265SessionParametersGetInfoKHR {
VkStructureType sType;
const void* pNext;
VkBool32 writeStdVPS;
VkBool32 writeStdSPS;
VkBool32 writeStdPPS;
uint32_t stdVPSId;
uint32_t stdSPSId;
uint32_t stdPPSId;
} VkVideoEncodeH265SessionParametersGetInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
writeStdVPSindicates whether the encoded H.265 video parameter set identified bystdVPSIdis requested to be retrieved. -
writeStdSPSindicates whether the encoded H.265 sequence parameter set identified by the pair constructed fromstdVPSIdandstdSPSIdis requested to be retrieved. -
writeStdPPSindicates whether the encoded H.265 picture parameter set identified by the triplet constructed fromstdVPSId,stdSPSId, andstdPPSIdis requested to be retrieved. -
stdVPSIdspecifies the H.265 video parameter set ID used to identify the retrieved H.265 video, sequence, and/or picture parameter set(s). -
stdSPSIdspecifies the H.265 sequence parameter set ID used to identify the retrieved H.265 sequence and/or picture parameter set(s) whenwriteStdSPSand/orwriteStdPPSis set toVK_TRUE. -
stdPPSIdspecifies the H.265 picture parameter set ID used to identify the retrieved H.265 picture parameter set whenwriteStdPPSis set toVK_TRUE.
When this structure is specified in the pNext chain of the
VkVideoEncodeSessionParametersGetInfoKHR structure passed to
vkGetEncodedVideoSessionParametersKHR, the command will write encoded
parameter data to the output buffer in the following order:
-
The H.265 video parameter set identified by
stdVPSId, ifwriteStdVPSis set toVK_TRUE. -
The H.265 sequence parameter set identified by the pair constructed from
stdVPSIdandstdSPSId, ifwriteStdSPSis set toVK_TRUE. -
The H.265 picture parameter set identified by the triplet constructed from
stdVPSId,stdSPSId, andstdPPSId, ifwriteStdPPSis set toVK_TRUE.
The VkVideoEncodeH265SessionParametersFeedbackInfoKHR structure is
defined as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265SessionParametersFeedbackInfoKHR {
VkStructureType sType;
void* pNext;
VkBool32 hasStdVPSOverrides;
VkBool32 hasStdSPSOverrides;
VkBool32 hasStdPPSOverrides;
} VkVideoEncodeH265SessionParametersFeedbackInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
hasStdVPSOverridesindicates whether any of the parameters of the requested H.265 video parameter set, if one was requested via VkVideoEncodeH265SessionParametersGetInfoKHR::writeStdVPS, were overridden by the implementation. -
hasStdSPSOverridesindicates whether any of the parameters of the requested H.265 sequence parameter set, if one was requested via VkVideoEncodeH265SessionParametersGetInfoKHR::writeStdSPS, were overridden by the implementation. -
hasStdPPSOverridesindicates whether any of the parameters of the requested H.265 picture parameter set, if one was requested via VkVideoEncodeH265SessionParametersGetInfoKHR::writeStdPPS, were overridden by the implementation.
42.17.10. H.265 Encoding Parameters
The VkVideoEncodeH265PictureInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265PictureInfoKHR {
VkStructureType sType;
const void* pNext;
uint32_t naluSliceSegmentEntryCount;
const VkVideoEncodeH265NaluSliceSegmentInfoKHR* pNaluSliceSegmentEntries;
const StdVideoEncodeH265PictureInfo* pStdPictureInfo;
} VkVideoEncodeH265PictureInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
naluSliceSegmentEntryCountis the number of elements inpNaluSliceSegmentEntries. -
pNaluSliceSegmentEntriesis a pointer to an array ofnaluSliceSegmentEntryCountVkVideoEncodeH265NaluSliceSegmentInfoKHR structures specifying the parameters of the individual H.265 slice segments to encode for the input picture. -
pStdPictureInfois a pointer to aStdVideoEncodeH265PictureInfostructure specifying H.265 picture information.
This structure is specified in the pNext chain of the
VkVideoEncodeInfoKHR structure passed to vkCmdEncodeVideoKHR to
specify the codec-specific picture information for an H.265
encode operation.
- Encode Input Picture Information
-
When this structure is specified in the
pNextchain of the VkVideoEncodeInfoKHR structure passed to vkCmdEncodeVideoKHR, the information related to the encode input picture is defined as follows:-
The image subregion used is determined according to the H.265 Encode Picture Data Access section.
-
The encode input picture is associated with the H.265 picture information provided in
pStdPictureInfo.
-
- Std Picture Information
-
The members of the
StdVideoEncodeH265PictureInfostructure pointed to bypStdPictureInfoare interpreted as follows:-
flags.reservedandreserved1are used only for padding purposes and are otherwise ignored; -
flags.is_referenceas defined in section 3.132 of the ITU-T H.265 Specification; -
flags.IrapPicFlagas defined in section 3.73 of the ITU-T H.265 Specification; -
flags.used_for_long_term_referenceis used to indicate whether the picture is marked as “used for long-term reference” as defined in section 8.3.2 of the ITU-T H.265 Specification; -
flags.discardable_flagandcross_layer_bla_flagas defined in section F.7.4.7.1 of the ITU-T H.265 Specification; -
pic_typeas defined in section 7.4.3.5 of the ITU-T H.265 Specification; -
sps_video_parameter_set_id,pps_seq_parameter_set_id, andpps_pic_parameter_set_idare used to identify the active parameter sets, as described below; -
PicOrderCntValas defined in section 8.3.1 of the ITU-T H.265 Specification; -
TemporalIdas defined in section 7.4.2.2 of the ITU-T H.265 Specification; -
if
pRefListsis notNULL, then it is a pointer to aStdVideoEncodeH265ReferenceListsInfostructure that is interpreted as follows:-
flags.reservedis used only for padding purposes and is otherwise ignored; -
ref_pic_list_modification_flag_l0andref_pic_list_modification_flag_l1as defined in section 7.4.7.2 of the ITU-T H.265 Specification; -
num_ref_idx_l0_active_minus1andnum_ref_idx_l1_active_minus1as defined in section 7.4.7.1 of the ITU-T H.265 Specification; -
RefPicList0andRefPicList1as defined in section 8.3.4 of the ITU-T H.265 Specification where each element of these arrays either identifies an active reference picture using its DPB slot index or contains the valueSTD_VIDEO_H265_NO_REFERENCE_PICTUREto indicate “no reference picture”; -
list_entry_l0andlist_entry_l1as defined in section 7.4.7.2 of the ITU-T H.265 Specification;
-
-
if
flags.short_term_ref_pic_set_sps_flagis set, then theStdVideoH265ShortTermRefPicSetstructure pointed to bypShortTermRefPicSetis interpreted as defined for the elements of thepShortTermRefPicSetarray specified in H.265 sequence parameter sets. -
if
flags.long_term_ref_pics_present_flagis set in the active SPS, then theStdVideoEncodeH265LongTermRefPicsstructure pointed to bypLongTermRefPicsis interpreted as follows:-
used_by_curr_pic_lt_flagis a bitmask where bit index i corresponds toused_by_curr_pic_lt_flag[i]as defined in section 7.4.7.1 of the ITU-T H.265 Specification; -
all other members of
StdVideoEncodeH265LongTermRefPicsare interpreted as defined in section 7.4.7.1 of the ITU-T H.265 Specification;
-
-
all other members are interpreted as defined in section 7.4.7.1 of the ITU-T H.265 Specification.
-
Reference picture setup is controlled by the value of
StdVideoEncodeH265PictureInfo::flags.is_reference.
If it is set and a reconstructed
picture is specified, then the latter is used as the target of picture
reconstruction to activate the DPB slot
specified in pEncodeInfo->pSetupReferenceSlot->slotIndex.
If StdVideoEncodeH265PictureInfo::flags.is_reference is not set,
but a reconstructed picture is
specified, then the corresponding picture reference associated with the
DPB slot is invalidated, as described in the
DPB Slot States section.
- Active Parameter Sets
-
The members of the
StdVideoEncodeH265PictureInfostructure pointed to bypStdPictureInfoare used to select the active parameter sets to use from the bound video session parameters object, as follows:-
The active VPS is the VPS identified by the key specified in
StdVideoEncodeH265PictureInfo::sps_video_parameter_set_id. -
The active SPS is the SPS identified by the key specified by the pair constructed from
StdVideoEncodeH265PictureInfo::sps_video_parameter_set_idandStdVideoEncodeH265PictureInfo::pps_seq_parameter_set_id. -
The active PPS is the PPS identified by the key specified by the triplet constructed from
StdVideoEncodeH265PictureInfo::sps_video_parameter_set_id,StdVideoEncodeH265PictureInfo::pps_seq_parameter_set_id, andStdVideoEncodeH265PictureInfo::pps_pic_parameter_set_id.
-
H.265 encoding uses explicit weighted sample prediction for a slice
segment, as defined in section 8.5.3.3.4 of the ITU-T H.265
Specification, if any of the following conditions are true for the active
PPS and the pStdSliceSegmentHeader member of the
corresponding element of pNaluSliceSegmentEntries:
-
pStdSliceSegmentHeader->slice_typeisSTD_VIDEO_H265_SLICE_TYPE_Pandweighted_pred_flagis enabled in the active PPS. -
pStdSliceSegmentHeader->slice_typeisSTD_VIDEO_H265_SLICE_TYPE_Bandweighted_bipred_flagis enabled in the active PPS.
The number of H.265 tiles, as defined in section 3.174 of the ITU-T H.265 Specification, is derived from the
num_tile_columns_minus1 and num_tile_rows_minus1 members of the
active PPS as follows:
-
(
num_tile_columns_minus1+ 1) × (num_tile_rows_minus1+ 1)
The VkVideoEncodeH265NaluSliceSegmentInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265NaluSliceSegmentInfoKHR {
VkStructureType sType;
const void* pNext;
int32_t constantQp;
const StdVideoEncodeH265SliceSegmentHeader* pStdSliceSegmentHeader;
} VkVideoEncodeH265NaluSliceSegmentInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
constantQpis the QP to use for the slice segment if the current rate control mode configured for the video session isVK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR. -
pStdSliceSegmentHeaderis a pointer to aStdVideoEncodeH265SliceSegmentHeaderstructure specifying H.265 slice segment header parameters for the slice segment.
- Std Slice Segment Header Parameters
-
The members of the
StdVideoEncodeH265SliceSegmentHeaderstructure pointed to bypStdSliceSegmentHeaderare interpreted as follows:-
flags.reservedandreserved1are used only for padding purposes and are otherwise ignored; -
if
pWeightTableis notNULL, then it is a pointer to aStdVideoEncodeH265WeightTablethat is interpreted as follows:-
flags.luma_weight_l0_flag,flags.chroma_weight_l0_flag,flags.luma_weight_l1_flag, andflags.chroma_weight_l1_flagare bitmasks where bit index i corresponds toluma_weight_l0_flag[i],chroma_weight_l0_flag[i],luma_weight_l1_flag[i], andchroma_weight_l1_flag[i], respectively, as defined in section 7.4.7.3 of the ITU-T H.265 Specification; -
all other members of
StdVideoEncodeH265WeightTableare interpreted as defined in section 7.4.7.3 of the ITU-T H.265 Specification;
-
-
all other members are interpreted as defined in section 7.4.7.1 of the ITU-T H.265 Specification.
-
The VkVideoEncodeH265DpbSlotInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265DpbSlotInfoKHR {
VkStructureType sType;
const void* pNext;
const StdVideoEncodeH265ReferenceInfo* pStdReferenceInfo;
} VkVideoEncodeH265DpbSlotInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
pStdReferenceInfois a pointer to aStdVideoEncodeH265ReferenceInfostructure specifying H.265 reference information.
This structure is specified in the pNext chain of
VkVideoEncodeInfoKHR::pSetupReferenceSlot, if not NULL, and
the pNext chain of the elements of
VkVideoEncodeInfoKHR::pReferenceSlots to specify the
codec-specific reference picture information for an H.265
encode operation.
- Active Reference Picture Information
-
When this structure is specified in the
pNextchain of the elements of VkVideoEncodeInfoKHR::pReferenceSlots, one element is added to the list of active reference pictures used by the video encode operation for each element of VkVideoEncodeInfoKHR::pReferenceSlotsas follows:-
The image subregion used is determined according to the H.265 Encode Picture Data Access section.
-
The reference picture is associated with the DPB slot index specified in the
slotIndexmember of the corresponding element of VkVideoEncodeInfoKHR::pReferenceSlots. -
The reference picture is associated with the H.265 reference information provided in
pStdReferenceInfo.
-
- Reconstructed Picture Information
-
When this structure is specified in the
pNextchain of VkVideoEncodeInfoKHR::pSetupReferenceSlot, the information related to the reconstructed picture is defined as follows:-
The image subregion used is determined according to the H.265 Encode Picture Data Access section.
-
If reference picture setup is requested, then the reconstructed picture is used to activate the DPB slot with the index specified in VkVideoEncodeInfoKHR::
pSetupReferenceSlot->slotIndex. -
The reconstructed picture is associated with the H.265 reference information provided in
pStdReferenceInfo.
-
- Std Reference Information
-
The members of the
StdVideoEncodeH265ReferenceInfostructure pointed to bypStdReferenceInfoare interpreted as follows:-
flags.reservedis used only for padding purposes and is otherwise ignored; -
flags.used_for_long_term_referenceis used to indicate whether the picture is marked as “used for long-term reference” as defined in section 8.3.2 of the ITU-T H.265 Specification; -
flags.unused_for_referenceis used to indicate whether the picture is marked as “unused for reference” as defined in section 8.3.2 of the ITU-T H.265 Specification; -
pic_typeas defined in section 7.4.3.5 of the ITU-T H.265 Specification; -
PicOrderCntValas defined in section 8.3.1 of the ITU-T H.265 Specification; -
TemporalIdas defined in section 7.4.2.2 of the ITU-T H.265 Specification.
-
42.17.11. H.265 Encode Rate Control
Group of Pictures
In case of H.265 encoding it is common practice to follow a regular pattern of different picture types in display order when encoding subsequent frames. This pattern is referred to as the group of pictures (GOP).
A regular GOP is defined by the following parameters:
-
The number of frames in the GOP;
-
The number of consecutive B frames between I and/or P frames in display order.
GOPs are further classified as open and closed GOPs.
Frame types in an open GOP follow each other in display order according to the following algorithm:
-
The first frame is always an I frame.
-
This is followed by a number of consecutive B frames, as defined above.
-
If the number of frames in the GOP is not reached yet, then the next frame is a P frame and the algorithm continues from step 2.
In case of a closed GOP, an IDR frame is used at a certain period.
It is also typical for H.265 encoding to use specific reference picture usage patterns across the frames of the GOP. The two most common reference patterns used are as follows:
- Flat Reference Pattern
-
-
Each P frame uses the last non-B frame, in display order, as reference.
-
Each B frame uses the last non-B frame, in display order, as its backward reference, and uses the next non-B frame, in display order, as its forward reference.
-
- Dyadic Reference Pattern
-
-
Each P frame uses the last non-B frame, in display order, as reference.
-
The following algorithm is applied to the sequence of consecutive B frames between I and/or P frames in display order:
-
The B frame in the middle of this sequence uses the frame preceding the sequence as its backward reference, and uses the frame following the sequence as its forward reference.
-
The algorithm is executed recursively for the following frame sequences:
-
The B frames of the original sequence preceding the frame in the middle, if any.
-
The B frames of the original sequence following the frame in the middle, if any.
-
-
-
The application can provide guidance to the implementation’s rate control algorithm about the structure of the GOP used by the application. Any such guidance about the GOP and its structure does not mandate that specific GOP structure to be used by the application, as the picture type of individual encoded pictures is still application-controlled, however, any deviation from the provided guidance may result in undesired rate control behavior including, but not limited, to the implementation not being able to conform to the expected average or target bitrates, or other rate control parameters specified by the application.
When an H.265 encode session is used to encode multiple temporal sub-layers, it is also common practice to follow a regular pattern for the H.265 temporal ID for the encoded pictures in display order when encoding subsequent frames. This pattern is referred to as the temporal GOP. The most common temporal layer pattern used is as follows:
- Dyadic Temporal Sub-Layer Pattern
-
-
The number of frames in the temporal GOP is 2n-1, where n is the number of temporal sub-layers.
-
The ith frame in the temporal GOP uses temporal ID t, if and only if the index of the least significant bit set in i equals n-t-1, except for the first frame, which is the only frame in the temporal GOP using temporal ID zero.
-
The ith frame in the temporal GOP uses the rth frame as reference, where r is calculated from i by clearing the least significant bit set in it, except for the first frame in the temporal GOP, which uses the first frame of the previous temporal GOP, if any, as reference.
-
|
Note
Multi-layer rate control and multi-layer coding are typically used for streaming cases where low latency is expected, hence B pictures with forward prediction are usually not used. |
The VkVideoEncodeH265RateControlInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265RateControlInfoKHR {
VkStructureType sType;
const void* pNext;
VkVideoEncodeH265RateControlFlagsKHR flags;
uint32_t gopFrameCount;
uint32_t idrPeriod;
uint32_t consecutiveBFrameCount;
uint32_t subLayerCount;
} VkVideoEncodeH265RateControlInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
flagsis a bitmask of VkVideoEncodeH265RateControlFlagBitsKHR specifying H.265 rate control flags. -
gopFrameCountis the number of frames within a group of pictures (GOP) intended to be used by the application. If it is set to 0, the rate control algorithm may assume an implementation-dependent GOP length. If it is set toUINT32_MAX, the GOP length is treated as infinite. -
idrPeriodis the interval, in terms of number of frames, between two IDR frames (see IDR period). If it is set to 0, the rate control algorithm may assume an implementation-dependent IDR period. If it is set toUINT32_MAX, the IDR period is treated as infinite. -
consecutiveBFrameCountis the number of consecutive B frames between I and/or P frames within the GOP. -
temporalLayerCountspecifies the number of H.265 sub-layers that the application intends to use.
When an instance of this structure is included in the pNext chain of
the VkVideoCodingControlInfoKHR structure passed to the
vkCmdControlVideoCodingKHR command, and
VkVideoCodingControlInfoKHR::flags includes
VK_VIDEO_CODING_CONTROL_ENCODE_RATE_CONTROL_BIT_KHR, the parameters in
this structure are used as guidance for the implementation’s rate control
algorithm (see Video Coding Control).
If flags includes
VK_VIDEO_ENCODE_H265_RATE_CONTROL_ATTEMPT_HRD_COMPLIANCE_BIT_KHR, then
the rate control state is reset to an initial state to meet HRD compliance
requirements.
Otherwise the new rate control state may be applied without a reset
depending on the implementation and the specified rate control parameters.
|
Note
It would be possible to infer the picture type to be used when encoding a
frame, on the basis of the values provided for |
Bits which can be set in
VkVideoEncodeH265RateControlInfoKHR::flags, specifying H.265
rate control flags, are:
// Provided by VK_KHR_video_encode_h265
typedef enum VkVideoEncodeH265RateControlFlagBitsKHR {
VK_VIDEO_ENCODE_H265_RATE_CONTROL_ATTEMPT_HRD_COMPLIANCE_BIT_KHR = 0x00000001,
VK_VIDEO_ENCODE_H265_RATE_CONTROL_REGULAR_GOP_BIT_KHR = 0x00000002,
VK_VIDEO_ENCODE_H265_RATE_CONTROL_REFERENCE_PATTERN_FLAT_BIT_KHR = 0x00000004,
VK_VIDEO_ENCODE_H265_RATE_CONTROL_REFERENCE_PATTERN_DYADIC_BIT_KHR = 0x00000008,
VK_VIDEO_ENCODE_H265_RATE_CONTROL_TEMPORAL_SUB_LAYER_PATTERN_DYADIC_BIT_KHR = 0x00000010,
} VkVideoEncodeH265RateControlFlagBitsKHR;
-
VK_VIDEO_ENCODE_H265_RATE_CONTROL_ATTEMPT_HRD_COMPLIANCE_BIT_KHRspecifies that rate control should attempt to produce an HRD compliant bitstream, as defined in annex C of the ITU-T H.265 Specification. -
VK_VIDEO_ENCODE_H265_RATE_CONTROL_REGULAR_GOP_BIT_KHRspecifies that the application intends to use a regular GOP structure according to the parameters specified in thegopFrameCount,idrPeriod, andconsecutiveBFrameCountmembers of the VkVideoEncodeH265RateControlInfoKHR structure. -
VK_VIDEO_ENCODE_H265_RATE_CONTROL_REFERENCE_PATTERN_FLAT_BIT_KHRspecifies that the application intends to follow a flat reference pattern in the GOP. -
VK_VIDEO_ENCODE_H265_RATE_CONTROL_REFERENCE_PATTERN_DYADIC_BIT_KHRspecifies that the application intends to follow a dyadic reference pattern in the GOP. -
VK_VIDEO_ENCODE_H265_RATE_CONTROL_TEMPORAL_SUB_LAYER_PATTERN_DYADIC_BIT_KHRspecifies that the application intends to follow a dyadic temporal sub-layer pattern.
// Provided by VK_KHR_video_encode_h265
typedef VkFlags VkVideoEncodeH265RateControlFlagsKHR;
VkVideoEncodeH265RateControlFlagsKHR is a bitmask type for setting a
mask of zero or more VkVideoEncodeH265RateControlFlagBitsKHR.
Rate Control Layers
The VkVideoEncodeH265RateControlLayerInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265RateControlLayerInfoKHR {
VkStructureType sType;
const void* pNext;
VkBool32 useMinQp;
VkVideoEncodeH265QpKHR minQp;
VkBool32 useMaxQp;
VkVideoEncodeH265QpKHR maxQp;
VkBool32 useMaxFrameSize;
VkVideoEncodeH265FrameSizeKHR maxFrameSize;
} VkVideoEncodeH265RateControlLayerInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
useMinQpindicates whether the QP values determined by rate control will be clamped to the lower bounds on the QP values specified inminQp. -
minQpspecifies the lower bounds on the QP values, for each picture type, that the implementation’s rate control algorithm will use whenuseMinQpis set toVK_TRUE. -
useMaxQpindicates whether the QP values determined by rate control will be clamped to the upper bounds on the QP values specified inmaxQp. -
maxQpspecifies the upper bounds on the QP values, for each picture type, that the implementation’s rate control algorithm will use whenuseMaxQpis set toVK_TRUE. -
useMaxFrameSizeindicates whether the implementation’s rate control algorithm should use the values specified inmaxFrameSizeas the upper bounds on the encoded frame size for each picture type. -
maxFrameSizespecifies the upper bounds on the encoded frame size, for each picture type, whenuseMaxFrameSizeis set toVK_TRUE.
When used, the values in minQp and maxQp guarantee that the
effective QP values used by the implementation will respect those lower and
upper bounds, respectively.
However, limiting the range of QP values that the implementation is able to
use will also limit the capabilities of the implementation’s rate control
algorithm to comply to other constraints.
In particular, the implementation may not be able to comply to the
following:
-
The average and/or peak bitrate values to be used for the encoded bitstream specified in the
averageBitrateandmaxBitratemembers of the VkVideoEncodeRateControlLayerInfoKHR structure. -
The upper bounds on the encoded frame size, for each picture type, specified in the
maxFrameSizemember ofVkVideoEncodeH265RateControlLayerInfoKHR.
|
Note
In general, applications need to configure rate control parameters appropriately in order to be able to get the desired rate control behavior, as described in the Video Encode Rate Control section. |
When an instance of this structure is included in the pNext chain of a
VkVideoEncodeRateControlLayerInfoKHR structure specified in one of the
elements of the pLayers array member of the
VkVideoEncodeRateControlInfoKHR structure passed to the
vkCmdControlVideoCodingKHR command,
VkVideoCodingControlInfoKHR::flags includes
VK_VIDEO_CODING_CONTROL_ENCODE_RATE_CONTROL_BIT_KHR, and the bound
video session was created with the video codec operation
VK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR, it specifies the
H.265-specific rate control parameters of the rate control layer
corresponding to that element of pLayers.
The VkVideoEncodeH265QpKHR structure is defined as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265QpKHR {
int32_t qpI;
int32_t qpP;
int32_t qpB;
} VkVideoEncodeH265QpKHR;
-
qpIis the QP to be used for I pictures. -
qpPis the QP to be used for P pictures. -
qpBis the QP to be used for B pictures.
The VkVideoEncodeH265FrameSizeKHR structure is defined as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265FrameSizeKHR {
uint32_t frameISize;
uint32_t framePSize;
uint32_t frameBSize;
} VkVideoEncodeH265FrameSizeKHR;
GOP Remaining Frames
Besides session level rate control configuration, the application can specify the number of frames per frame type remaining in the group of pictures (GOP).
The VkVideoEncodeH265GopRemainingFrameInfoKHR structure is defined as:
// Provided by VK_KHR_video_encode_h265
typedef struct VkVideoEncodeH265GopRemainingFrameInfoKHR {
VkStructureType sType;
const void* pNext;
VkBool32 useGopRemainingFrames;
uint32_t gopRemainingI;
uint32_t gopRemainingP;
uint32_t gopRemainingB;
} VkVideoEncodeH265GopRemainingFrameInfoKHR;
-
sTypeis a VkStructureType value identifying this structure. -
pNextisNULLor a pointer to a structure extending this structure. -
useGopRemainingFramesindicates whether the implementation’s rate control algorithm should use the values specified ingopRemainingI,gopRemainingP, andgopRemainingB. IfuseGopRemainingFramesisVK_FALSE, then the values ofgopRemainingI,gopRemainingP, andgopRemainingBare ignored. -
gopRemainingIspecifies the number of I frames the implementation’s rate control algorithm should assume to be remaining in the GOP prior to executing the video encode operation. -
gopRemainingPspecifies the number of P frames the implementation’s rate control algorithm should assume to be remaining in the GOP prior to executing the video encode operation. -
gopRemainingBspecifies the number of B frames the implementation’s rate control algorithm should assume to be remaining in the GOP prior to executing the video encode operation.
Setting useGopRemainingFrames to VK_TRUE and including this
structure in the pNext chain of VkVideoBeginCodingInfoKHR is
only mandatory if the
VkVideoEncodeH265CapabilitiesKHR::requiresGopRemainingFrames
reported for the used video profile is VK_TRUE.
However, implementations may use these remaining frame counts, when
specified, even when it is not required.
In particular, when the application does not use a
regular GOP structure, these values may provide
additional guidance for the implementation’s rate control algorithm.
The VkVideoEncodeH265CapabilitiesKHR::prefersGopRemainingFrames
capability is also used to indicate that the implementation’s rate control
algorithm may operate more accurately if the application specifies the
remaining frame counts using this structure.
As with other rate control guidance values, if the effective order and number of frames encoded by the application are not in line with the remaining frame counts specified in this structure at any given point, then the behavior of the implementation’s rate control algorithm may deviate from the one expected by the application.
42.17.12. H.265 Encode Requirements
This section described the required H.265 encoding capabilities for
physical devices that have at least one queue family that supports the video
codec operation VK_VIDEO_CODEC_OPERATION_ENCODE_H265_BIT_KHR, as
returned by vkGetPhysicalDeviceQueueFamilyProperties2 in
VkQueueFamilyVideoPropertiesKHR::videoCodecOperations.
| Video Std Header Name | Version |
|---|---|
|
1.0.0 |
| Video Capability | Requirement | Requirement Type1 |
|---|---|---|
|
- |
min |
|
4096 |
max |
|
4096 |
max |
|
(64,64) |
max |
|
- |
max |
|
- |
min |
|
0 |
min |
|
0 |
min |
|
- |
min |
|
- |
min |
|
128000 |
min |
|
1 |
min |
|
(64,64) |
max |
|
|
min |
|
- |
min |
|
|
min |
|
1 |
min |
|
(1,1) |
min |
|
at least one bit set |
implementation-dependent |
|
at least one bit set |
implementation-dependent |
|
0 |
min |
|
0 |
min |
|
0 |
min |
|
1 |
min |
|
- |
implementation-dependent |
|
- |
max |
|
- |
min |
|
- |
implementation-dependent |
|
- |
implementation-dependent |
|
- |
min |
- 1
-
The Requirement Type column specifies the requirement is either the minimum value all implementations must support, the maximum value all implementations must support, or the exact value all implementations must support. For bitmasks a minimum value is the least bits all implementations must set, but they may have additional bits set beyond this minimum.