CN101617539A - Based on computation complexity in the digital media coder of conversion and precision control - Google Patents

Abstract

The Digital Media encoder/decoder comprises and signaling in the computation complexity of the decoding place various patterns relevant with precision.Encoder can send the syntax elements of indication in the arithmetic precision (for example, using 16 or 32 bit arithmetics) of the transform operation of decoding place execution.Encoder can also signal in decoder output place whether use convergent-divergent, and this permits the wideer dynamic range of the intermediate data of decoding place, but owing to the convergent-divergent computing has increased computation complexity.

Description

Computational Complexity and Precision Control in Transform-Based Digital Media Codecs

background

Block Transform Based Coding

Transform coding is a compression technique used in many digital media (eg, audio, image, and video) compression systems. Uncompressed digital images and video are typically represented or captured as samples of primitives or colors at various locations in an image or video frame arranged in a two-dimensional (2D) grid. This is called a spatial domain representation of an image or video. For example, a typical format for images consists of a stream of 24-bit color primitive samples arranged as a grid. Each sample is a number representing a color component at a pixel location in the grid in a color space such as RGB or YIQ. Various image and video systems may use sampling at various color, spatial and temporal resolutions. Similarly, digital audio is often represented as a stream of time-sampled audio signals. For example, a typical audio format consists of a stream of 16-bit audio signal amplitude samples taken at regular time intervals.

Uncompressed digital audio, image and video signals can consume large amounts of storage and transmission capacity. Transform coding reduces digital audio, The size of images and videos. This generally produces less perceptible degradation of the digital signal than reducing the color or spatial resolution of images or video in the spatial domain or audio in the temporal domain.

)。More specifically, a typical block-transform-based encoder/decoder system 100 (also referred to as a "codec") shown in FIG. 1 divides the pixels of an uncompressed digital image into fixed-size two-dimensional blocks (X ₁ , . . . X _n ), each block may overlap other blocks. A linear transform 120-121 that performs a space-frequency analysis is applied to each block, which converts the samples spaced apart from each other within the block into a set of frequency (or transform) coefficients that generally represent the strength of the digital signal in the corresponding frequency band over the block interval . For compression, the transform coefficients may be selectively quantized 130 (i.e., to reduce resolution such as by discarding the least significant bits of coefficient values or mapping values in a higher resolution digital set to a lower resolution), and also entropy Encoding or variable length encoding 130 into a compressed data stream. On decoding, the transform coefficients are inverse transformed 170-171 in order to nearly reconstruct the original color/space sampled image/video signal (reconstruction block

).

A block transform 120-121 may be defined as a mathematical operation on a vector x of size N. Most commonly, this operation is a linear multiplication, resulting in a transform domain output y=Mx, where M is the transformation matrix. When the input data is arbitrarily long, it is segmented into vectors of size N, and a block transformation is applied to each segment. For the purpose of data compression, the reversible block transform is chosen. In other words, matrix M is invertible. In multiple dimensions (eg, for images and videos), block transformations are often implemented as separable operations. Matrix multiplication is applied separably along each dimension of the data (ie, rows and columns).

For compression, the transform coefficients (components of the vector y) can be selectively quantized (i.e., to reduce resolution such as by discarding the least significant bits of the coefficient values or mapping values in higher resolution digit sets to lower resolutions) , and can also be entropy coded or variable length coded into a compressed data stream.

When decoding in the decoder 150, as shown in FIG. 1, the inverse process of these operations (dequantization/entropy decoding 160 and inverse block transformation 170-171) is applied on the decoder 150 side. When reconstructing the data, the inverse matrix M ^-1 (inverse transform 170-171) is applied as a multiplier to the transform domain data. When applied to transform-domain data, the inverse transform nearly reconstructs the original time-domain or space-domain digital media.

In many block transform-based coding applications, the transform is ideally reversible to support both lossy and lossless compression depending on the quantization factor. A codec utilizing reversible transforms can reproduce the input data exactly when decoded if, for example, there is no quantization (commonly denoted quantization factor 1). However, the requirement of invertibility in these applications constrains the choice of transforms for designing codecs.

Many image and video compression systems, such as MPEG and Windows Media, utilize discrete cosine transform (DCT) based transforms. DCT is known to have good energy compression properties leading to near-optimal data compression. In these compression systems, an inverse DCT (IDCT) is employed in the reconstruction loop in both the encoder and decoder of the compression system to reconstruct the individual image blocks.

Quantify

Quantization is the primary mechanism by which most image and video codecs control compressed image quality and compression ratio. According to one possible definition, quantization is the term used for approximately irreversible mapping functions commonly used in lossy compression, where there is a specified set of possible output values, and each member of the set of possible output values has A set of associated input values for a selection of output values. Various quantization techniques have been developed, including scalar or vector, uniform or non-uniform, with or without dead zone, and adaptive or non-adaptive quantization.

The quantization operation is essentially a biased division according to the quantization parameter QP, which is performed at the encoder. The inverse quantization or multiplication operation is a multiplication with the QP, which is performed at the decoder. Together, these processes introduce a loss of the original transform coefficient data, which manifests as compression errors or artifacts in the decoded image.

overview

The following detailed description presents tools and techniques for controlling the computational complexity and precision of decoding using digital media codecs. In one aspect of the technique, the encoder signals at the decoder to use one of scaled or unscaled precision modes. In scaled precision mode, the input image is premultiplied (eg multiplied by 8) at the encoder. The output at the decoder is also scaled by rounding and division. In unscaled precision mode, this scaling operation is not applied. In unscaled precision mode, the encoder or decoder can handle a smaller dynamic range of transform coefficients and thus has lower computational complexity.

In another aspect of this technique, the codec can also signal to the decoder the precision required to perform the transform operations. In one implementation, an element of the bitstream syntax signals whether to employ lower precision arithmetic operations for the transform at the decoder.

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. Other features and advantages of the present invention will become apparent after continuing to read the following detailed description of the embodiments with reference to the accompanying drawings.

Brief description of the drawings

Fig. 1 is a block diagram of a conventional block transform based codec in the prior art.

Figure 2 is a flow diagram of a representative encoder including block mode encoding.

Figure 3 is a flowchart of a representative decoder including block mode encoding.

4 is a diagram of an inverse lapped transform including a kernel transform and a post-filtering (overlap) operation in one implementation of the representative encoder/decoder of FIGS. 2 and 3 .

5 is a diagram identifying input data points for a transformation operation.

6 is a block diagram of a suitable computing environment for implementing the media encoder/decoder of FIGS. 2 and 3 .

A detailed description

The following description relates to techniques for controlling the precision and computational complexity of transform-based digital media codecs. The following description describes one example implementation of the techniques in the context of a digital media compression system or codec. The digital media system encodes digital media data in a compressed form for transmission or storage and decodes the data for playback or other processing. For purposes of illustration, this exemplary compression system involving computational complexity and precision control is an image or video compression system. Alternatively, this technique may also be incorporated into compression systems or codecs for other digital media data. Computational complexity and precision control techniques do not require digital media compression systems to encode compressed digital media data in a specific encoding format.

1.1. Encoder/Decoder

2 and 3 are generalized illustrations of processes employed in a representative 2-dimensional (2D) data encoder 200 and decoder 300 . The figure presents a generalized or simplified illustration of a compression system incorporating a 2D data encoder and decoder that uses computational complexity and precision control techniques to achieve the compression. In alternative compression systems using control techniques, more or fewer processes than shown in this representative encoder and decoder can be used for 2D data compression. For example, some encoders/decoders may also include color conversion, color formats, scalable encoding, lossless encoding, macroblock modes, etc. Compression systems (encoder and decoder) may provide lossless and/or lossy compression of 2D data, depending on quantization based on quantization parameters that may vary from lossless to lossy.

The 2D data encoder 200 produces a compressed bitstream 220, which is a more compact representation (for a typical input) of the 2D data 210 provided as input to the encoder. For example, a 2D data input may be an image, a frame of a video sequence, or other data having two dimensions. The 2D data encoder divides the input data frame into blocks (shown generally in Figure 2 as partitions 230), which in the implementation shown are non-overlapping 4x4 pixel blocks that form a regular pattern across the plane of the frame. These blocks are grouped into clusters called macroblocks, which in this representative encoder are 16x16 pixels in size. Macroblocks are in turn grouped into regular structures called tiles. The tiles can also form a regular pattern on the image, such that tiles in horizontal rows are of uniform height and aligned, while tiles in vertical columns are of uniform width and aligned. In this representative encoder, tiles can be of any size that is a multiple of 16 in the horizontal and/or vertical directions. Alternative encoder implementations may divide the image into blocks, macroblocks, tiles, or other units of other sizes and structures.

A "forward overlap" operator 240 is applied to each edge between blocks, after which a block transform 250 is used to transform each 4x4 block. The block transform 250 may be obtained by Srinivasan in U.S. Patent Application No. 1, entitled "Reversible Transform For Lossy And Lossless 2-D Data Compression", filed on December 17, 2004 by Srinivasan. The reversible, unscaled 2D transform described in 11/015,707. The overlap operator 240 may be submitted by Tu et al. on December 17, 2004, entitled "Reversible Overlap Operator for Efficient Lossless Data Compression" (reversible overlap operator for efficient lossless data compression) U.S. Patent Application No. 11/ 015,148; and "Reversible 2-Dimensional Pre-/Post-Filter for Lapped Biorthogonal Transform" by Tu et al., filed Jan. 14, 2005 ) of US Patent Application No. 11/035,991 described in the reversible overlap operator. Alternatively, a discrete cosine transform or other block transform and overlap operator may be used. After transformation, the DC coefficients 260 of each 4x4 transform block are subjected to a similar processing chain (blocking, forward overlap, followed by 4x4 block transform). The resulting DC transform coefficients and AC transform coefficients are quantized 270 , entropy encoded 280 and packetized 290 .

The decoder performs the reverse process. On the decoder side, the transform coefficient bits are extracted 310 from their respective packets, from which the coefficients themselves are decoded 320 and dequantized 330 . The DC coefficients 340 are regenerated by applying an inverse transform, and "inverse overlapping" the planes of the DC coefficients using a suitable smoothing operator applied across the DC block edges. Subsequently, the entire data is regenerated by applying a 4x4 inverse transform 350 to the DC coefficients, and the AC coefficients are decoded 342 from the bitstream. Finally, inverse overlap filtering is performed 360 on the block edges in the resulting image plane. This produces a reconstructed 2D data output.

In an exemplary implementation, encoder 200 (FIG. 2) compresses an input image into a compressed bitstream 220 (e.g., a file), and decoder 300 (FIG. 3) reconstructs the original image based on whether lossless or lossy encoding was used. input or its approximation. The encoding process involves applying the forward lapped transform (LT) discussed below, which is achieved with invertible 2-dimensional pre/post filtering, also described more fully below. The decoding process involves applying an inverse lapped transform (ILT) using reversible 2-dimensional pre/post filtering.

The shown LT and ILT are in the exact sense the inverse of each other, and thus may be collectively referred to as an invertible lapped transform. As a reversible transform, the LT/ILT pair can be used for lossless image compression.

The input data 210 compressed by the illustrated encoder 200/decoder 300 can be in various color formats (e.g., RGB/YUV 4:4:4, YUV 4:2:2, or YUV 4:2:0 color image formats )Image. In general, an input image always has a luminance (Y) component. If it is an RGB/YUV 4:4:4, YUV 4:2:2 or YUV 4:2:0 image, the image also has chroma components such as U and V components. These individual color planes or components of an image may have different spatial resolutions. In case of an input image in eg YUV 4:2:0 color format, the U and V components have half the width and height of the Y component.

As described above, the encoder 200 blocks an input image or picture into macroblocks. In an exemplary implementation, the encoder 200 blocks the input image into 16×16 pixel regions (referred to as “macroblocks”) in the Y channel (depending on the color format, this could be 16×16 in the U and V channels, 16×8 or 8×8 area). Each macroblock color plane is blockized into regions or blocks of 4x4 pixels. Therefore, for this exemplary encoder implementation, a macroblock is composed of various color formats in the following manner:

1. For grayscale images, each macroblock contains 16 4x4 luma (Y) blocks.

2. For a YUV 4:2:0 format color image, each macroblock contains 16 4×4 Y blocks, and 4 4×4 chroma (U and V) blocks each.

3. For YUV 4:2:2 format color images, each macroblock contains 16 4×4 Y blocks, and 8 4×4 chrominance (U and V) blocks each.

4. For RGB or YUV 4:4:4 color images, each macroblock contains 16 blocks for each of the Y, U, and V channels.

Thus, after transformation, a macroblock in this representative encoder 200/decoder 300 has three frequency subbands: a DC subband (DC macroblock), a low-pass subband (low-pass macroblock), and a high-pass subband (Qualcomm macroblock). In this representative system, the lowpass and/or highpass subbands are optional in the bitstream - these subbands can be dropped entirely.

Furthermore, compressed data can be stuffed into the bitstream in one of two orders: spatial order and frequency order. For spatial order, different subbands of the same macroblock within a tile are ordered together, and the resulting bitstream for each tile is written into one packet. For frequency order, the same subbands from different macroblocks within a tile are grouped together, and thus the bitstream of the tile is written in the following three groups: DC tile grouping, low-pass tile grouping, and high-pass tile grouping block grouping. Additionally, there may be other data layers.

Therefore, for this representative system, images are organized in the following "dimensions":

Spatial dimension: frame→tile→macroblock;

Frequency Dimensions: DC|Low Pass|High Pass; and

Channel dimensions: luma|chroma0|chroma1... (eg, Y|U|V).

Arrows above indicate hierarchy, while vertical bars indicate divisions.

Although this representative system organizes compressed digital media data along spatial, frequency, and channel dimensions, the flexible quantization methods described here can be applied to alternative encoders/coders that organize their data along fewer, more, or other dimensions. decoder system. For example, this flexible quantization method can be applied to encodings using larger numbers of frequency bands, color channels in other formats (eg, YIQ, RGB, etc.), additional image channels (eg, for stereo vision or other multi-camera arrays).

2. Inverse core and overlapping transformation

overview

In one implementation of the encoder 200/decoder 300, the inverse transform at the decoder side takes the form of a two-stage lapped transform. Proceed as follows:

• An inverse core transform (ICT) is applied to each 4x4 block corresponding to the reconstructed DC and low-pass coefficients arranged in a planar array called a DC plane.

- Optionally apply post-filtering operations to uniformly span the 4x4 region of the block in the DC plane. In addition, the post-filter is applied to the border 2×4 and 4×2 regions, while the four corner regions are not changed.

• The resulting array contains DC coefficients corresponding to a 4x4 block of the first stage transform. The DC coefficients are (symbolically) copied to a larger array, and the reconstructed high-pass coefficients are filled into the remaining positions.

• Apply ICT to each 4x4 block.

- Optionally apply post-filtering operations to uniformly span the 4x4 region of the block in the DC plane. In addition, the post-filter is applied to the border 2×4 and 4×2 regions, while the four corner regions are not changed.

This process is illustrated in FIG. 4 .

Application of the post-filter is governed by the OVERLAP_INFO (overlap information) syntax element in the compressed bitstream 220 . OVERLAP_INFO can take three values:

• If OVERLAP_INFO = 0, no post-filtering is performed.

• If OVERLAP_INFO = 1, only external post-filtering is performed.

• If OVERLAP_INFO = 2, perform internal and external post-filtering.

inverse kernel transform

The Kernel Transform (CT) is inspired by what is conventionally known as the 4x4 Discrete Cosine Transform (DCT), but it is fundamentally different. The first key difference is that DCT is linear while CT is non-linear. The second key difference is due to the fact that it is defined on the real numbers, the DCT is not a lossless operation in integer-to-integer space. CT is defined on integers and is lossless in that space. The third key difference is that 2D DCT is a separable operation. CT is specifically indivisible.

The entire inverse transformation process can be written as a cascade of three basic 2×2 transformation operations, which are:

2×2 Hadamard transform: T_h

Inverse 1D rotation: InvT_odd

Inverse 2D rotation: InvT_odd_odd

These transformations are implemented as inseparable operations and are described first, followed by the description of the entire ICT.

2D 2×2 Hadamard Transform T_h

As shown in the following pseudocode table, the encoder/decoder implements the 2D 2×2 Hadamard transform T_h. R is a rounding factor whose value can only be 0 or 1. T_h is involuntary (i.e., applying T-h twice to the data vector [a b c d] successfully restores the original value of [a b c d], assuming R is unchanged between the two applications). The inverse T_h is T_h itself.

Inverse 1D rotation InvT_odd

The lossless inverse of T_odd is defined by the pseudocode in the table below.

Inverse 2D rotation InvT_odd_odd

The inverse 2D rotation InvT_odd_odd is defined by the pseudocode in the table below.

ICT operation

The correspondence between the 2×2 data and the previously listed pseudocode is shown in FIG. 5 . Here we introduce color coding using four gray levels to indicate four data points to facilitate the transformation description in the next section.

2D 4×4 point ICT is constructed using T_h, inverse T_odd and inverse T_odd_odd. Note that the inverse T_h is T_h itself. ICT consists of two phases, which are shown in the following pseudocode. Each stage consists of four 2x2 transforms that can be done in any order or simultaneously within that stage.

If the input data block is $\left[\begin{array}{cccc}a& b& c& d\\ e& f& g& h\\ i& j& k& l\\ m& \mathrm{no}& o& p\end{array}\right],$ Then 4×4_IPCT_1stStage() and 4×4_IPCT_2ndStage() are defined as follows:

The function 2×2_ICT is the same as T_h.

Post Filtering Overview

Four operators define the post-filter used in the inverse lapped transform. They are:

· 4×4 post-filter

· 4-point post filter

· 2×2 post-filter

· 2-point post filter

The post filter uses T_h, InvT_odd_odd, invScale and invRotate. invRotate and invScale are defined in the following tables respectively.

4×4 post filter

Initially, when OVERLAP_INFO is 1 or 2, a 4x4 post-filter is applied to all block concatenations (uniformly across an area of 4 blocks) in all color planes. Similarly, when OVERLAP_INFO is 2, a 4×4 filter is applied to all block connections in the DC plane of all planes, and when OVERLAP_INFO is 2 and the color format is YUV 4:2:0 or YUV 4:2:2, A 4x4 filter is applied to all block concatenations in the DC plane of the luma plane only.

If the input data is $\left[\begin{array}{cccc}a& b& c& d\\ e& f& g& h\\ i& j& k& l\\ m& \mathrm{no}& o& p\end{array}\right],$ Then the 4×4 post filter 4×4PostFilter(a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p) is defined in the following table:

4 point post filter

Applies a linear 4-point filter to the edges of the 2×4 and 4×2 regions across the border of the image. If the input data is [a b c d], the 4-point post filter 4PostFilter(a, b, c, d) is defined in the following table.

2×2 post filter

A 2×2 post-filter is applied to regions of blocks in the DC plane spanning YUV 4:2:0 and chroma channels of YUV 4:2:2 data. If the input data is $\left[\begin{array}{cc}a& b\\ c& d\end{array}\right],$ Then the 2×2 post filter 2×2PostFilter(a, b, c, d) is defined in the following table:

2 point post filter

Applies a 2-point post-filter to 2×1 and 1×2 samples across block boundaries. The 2-point post filter 2PostFilter(a, b) is defined in the following table:

The signaling of the precision required for the transform operations used to perform the lapped transform described above can be performed in the header of the compressed data structure. In this example implementation, LONG_WORD_FLAG and NO_SCALED_FLAGS are syntax elements sent in the compressed bitstream (eg, in the picture header) to signal the decoder the precision and computational complexity to apply.

3. Precision and word length

The example encoder/decoder performs integer arithmetic. Additionally, the sample encoder/decoder supports lossless encoding and decoding. Therefore, the required host machine precision for this example encoder/decoder is integer.

However, the integer operations defined in this example encoder/decoder result in round-off errors for lossy encoding. These errors are small by design, however, they cause a drop in the rate-distortion curve. For the purpose of improving encoding performance by reducing round-off errors, the example encoder/decoder defines a second machine precision. In this mode, the input is premultiplied by 8 (ie, shifted left by 3 bits), and the final output is divided by 8 and rounded (ie, shifted right by 3 bits). These operations are performed at the front end of the encoder and at the back end of the decoder, and are largely invisible to the rest of the process. Furthermore, the quantization levels are scaled accordingly so that streams created with the main machine precision and decoded with the second machine precision (and vice versa) produce acceptable images.

Second machine precision cannot be used when lossless compression is required. The machine precision used when creating the archive is explicitly marked in the header.

Second machine precision is equal to using scaling arithmetic in the codec, and thus this mode is called scaled. Main machine precision is said to be unscaled.

This example encoder/decoder is designed to provide good encoding and decoding speed. The design goal of this example encoder/decoder is that for an 8-bit input, the data values at the encoder and decoder do not exceed 16-bit signed values. (However, intermediate operations within the transform stage can exceed this number.) This is true for both machine-precision modes.

In contrast, when the second machine precision is selected, the range span of intermediate values is 8 bits. Because main machine precision avoids premultiplication by 8, its range spans 8-3=5 bits.

The first example encoder/decoder uses two different word sizes for intermediate values. These word lengths are 16 and 32 bits.

Second Example Bitstream Syntax and Semantics

The second example bitstream syntax and semantics is layered and includes the following layers: picture, tile, macroblock, and block.

Image (IMAGE)

IMAGE(){ Descriptor

IMAGE_HEADER variable struct

bAlphaPlane=FALSE

IMAGE_PLANE_HEADER variable struct

if(ALPHACHANNEL_FLAG){

bAlphaPlane=TRUE

IMAGE_PLANE_HEADER Variable Struct

}

INDEX_TABLE mutable struct

TILE struct variable

}

Image header (IMAGE_HEADER)

IMAGE_HEADER(){ Descriptor

GDISIGNATURE 64 uimsbf

RESERVED1 4 4 uimsbf

RESERVED2 4 4 uimsbf

TILING_FLAG 1 bool

FREQUENCYMODE_BITSTREAM_FLAG 1 uimsbf

IMAGE_ORIENTATION 3 uimsbf

INDEXTABLE_PRESENT_FLAG 1 uimsbf

OVERLAP_INFO 2 uimsbf

SHORT_HEADER_FLAG 1 bool

LONG_WORD_FLAG 1 bool

WINDOWING_FLAG l bool

TRIM_FLEXBITS_FLAG 1 bool

RESERVED3 3 3 uimsbf

ALPHACHANNEL_FLAG 1 bool

SOURCE_CLR_FMT 4 uimsbf

SOURCE_BITDEPTH 4 uimsbf

If(SHORT_HEADER_FLAG){

WIDTH_MINUS1 16 uimsbf

HEIGHT_MINUS1 16 uimsbf

}

else {

WIDTH_MINUS1 32 uimsbf

HEIGHT_MINUS1 32 uimsbf

}

if(TILING_FLAG){

NUM_VERT_TILES_MINUS1 12 uimsbf

NUM_HORIZ_TILES_MINUS1 12 uimsbf

}

for(n=0;n<

NUM_VERT_TILES_MINUS1; n++){

If(SHORT_HEADER_FLAG)

8 uimsbf

WIDTH_IN_MB_OF_TILE_MINUS1[n]

else

...

WIDTH_IN_MB_OF_TILE_MINUS1[n]

}

for(n=0;n<

NUM_HORIZ_TILES_MINUS1; n++){

If(SHORT_HEADER_FLAG)

8 uimsbf

HEIGHT_IN_MB_OF_TILE_MINUS1[n]

else

...

HEIGHT_IN_MB_OF_TILE_MINUS1[n]

}

if(WINDOWING_FLAG){

NUM_TOP_EXTRAPIXELS 6 uimsbf

NUM_LEFT_EXTRAPIXELS 6 uimsbf

NUM_BOTTOM_EXTRAPIXELS 6 uimsbf

NUM_RIGHT_EXTRAPIXELS 6 uimsbf

}

}

IMAGE_PLANE_HEADER(){ digits description

Symbol

CLR_FMT 3 uimsbf

NO_SCALED_FLAG 1 bool

BANDS_PRESENT 4 uimsbf

If(CLR_FMT==YUV444){

CHROMA_CENTERING 4 uimsbf

COLOR_INTERPRETATION 4 uimsbf

}

Else if(CLR_FMT==NCHANNEL){

NUM_CHANNELS_MINUS 14 uimsbf

COLOR_INTERPRETATION 4 uimsbf

}

if(SOURCE_CLR_FMT==BAYER){

BAYER_PATTERN 2 uimsbf

CHROMA_CENTERING_BAYER 2 uimsbf

COLOR_INTERPRETATION 4 uimsbf

}

if(SOURCE_BITDEPTH ∈

{BD16, BD16S, BD32, BD32S}){

SHIFT_BITS 8 uimsbf

}

If(SOURCE_BITEPTH==BD32F){

LEN_MANTISSA 8 uimsbf

EXP_BIAS 8 uimsbf

}

DC_FRAME_UNIFORM 1 bool

if(DC_FRAME_UNIFORM){

DC_QP() mutable struct

}

if(BANDS_PRESENT!=SB_DC_ONLY){

USE_DC_QP 1 bool

If(USE_DC_QP==FALSE){

LP_FRAME_UNIFORM 1 bool

      if(LP_FRAME_UNIFORM){

NUM_LP_QPS＝1

LP_QP() mutable struct

}

}

if(BANDS_PRESENT!=SB_NO_HIGHPASS){

USE_LP_QP 1 bool

If(USE_LP_QP==FALSE){

                                                

      if(HP_FRAME_UNIFORM){

NUM_HP_QPS＝1

                                                                                                                             

}

}

}

}

FLUSH_BYTE variable

}

Some selected bitstream elements from the second example bitstream syntax and semantics are defined as follows.

Long word flag (LONG_WORD_FLAG) (1 bit)

LONG_WORD_FLAG is a 1-bit syntax element and specifies whether 16-bit integers are used for transform calculations. In this second example bitstream syntax, if LONG_WORD_FLAG == 0 (FALSE (false)), then 16-bit integers and arrays can be used in the external stages of the transformation calculation (intermediate operations in the transformation (such as (3*a+1 )>>1) is performed with higher accuracy). If LONG_WORD_FLAG == TRUE, then 32-bit integers and arrays shall be used for transform calculations.

Note: 32-bit arithmetic can be used to decode images regardless of the value of LONG_WORD_FLAG. This syntax element can be used by the decoder to select the most efficient word size for implementation.

No scaling arithmetic flag (NO_SCALED_FLAG) (1 bit)

NO_SCALED_FLAG is a 1-bit syntax element that specifies whether the transform uses scaling. If NO_SCALED_FLAG==1, scaling should not be performed. If NO_SCALED_FLAG==0, then scaling should be performed. In this case scaling should be performed by appropriately rounding down the output of the final stage (color conversion) by 3 bits.

NOTE: If lossless encoding is desired, NO_SCALED_FLAG should be set to TRUE even if lossless encoding is only used for subregions of the image. Lossy encoding can use either mode.

Note: The rate-distortion performance of lossy encoding is good when scaling is used (ie, NO_SCALED_FLAG==FALSE), especially at low QP.

4. Signaling and use of the long word flag

An example image format of a representative encoder/decoder supports a wide variety of pixel formats, including high dynamic range and wide color gamut formats. Supported data types include signed integer, unsigned integer, fixed-point float, and float-float. Supported bit depths include 8, 16, 24 and 32 bits per color channel. An example image format allows lossless compression of images using up to 24 bits per color channel, and lossy compression of images using up to 32 bits per color channel.

At the same time, this example image format is designed to provide high image quality and compression efficiency, and to allow low-complexity encoding and decoding implementations.

To support low-complexity implementations, the transformations in the example image formats are designed to minimize expansion of the dynamic range. The two-stage transform only increases the dynamic range by 5 bits. Thus, if the image bit depth is 8 bits per color channel, 16-bit arithmetic may be sufficient to perform all transform operations at the decoder. For other bit depths, transform operations may require higher precision arithmetic.

The computational complexity of decoding a particular bitstream can be reduced if the precision required to perform the transform operation is known at the decoder. This information can be signaled to the decoder using a syntax element (eg, a 1-bit flag in the picture header). The described signaling techniques and syntax elements can reduce the computational complexity of decoding a bitstream.

In one example implementation, the 1-bit syntax element LONG_WORD_FLAG is used. For example, if LONG_WORD_FLAG==FALSE, 16-bit integers and arrays may be used for the external stages of transform calculations, and if LONG_WORD_FLAG==TRUE, 32-bit integers and arrays should be used for transform calculations.

In one implementation of this representative encoder/decoder, transform operations can be performed in-place on 16-bit wide words, but intermediate operations within the transform (as given by computing b+=(3*a+1)>>1 The product of 3*a of the "lifting" step out) is performed with higher accuracy (eg, 18 bits or higher). However, in this example, the intermediate transformed values a and b may themselves be stored in 16-bit integers.

32-bit arithmetic can be used to decode images regardless of the value of the LONG_WORD_FLAG element. The LONG_WORD_FLAG element may be used by an encoder/decoder to select the most efficient word length for implementation. For example, an encoder may choose to set the LONG_WORD_FLAG element to FALSE if it can verify that 16-bit and 32-bit precision transform steps produce the same output value.

5. Signaling and use of NO_SCALED_FLAG

An example image format of a representative encoder/decoder supports a wide variety of pixel formats, including high dynamic range and wide color gamut formats. Meanwhile, the design of this representative encoder/decoder optimizes image quality and compression efficiency, and allows low-complexity encoding and decoding implementation.

As mentioned above, this representative encoder/decoder uses a two-stage layered block-based transform where all transform steps are integer operations. The presence of small round-off errors in these integer operations results in a loss of compression efficiency during lossy compression. To combat this problem, one implementation of this representative encoder/decoder defines two different precision modes for decoder operations: scaled mode and unscaled mode.

In scaled precision mode, the input image is premultiplied by 8 at the encoder (ie, shifted left by 3 bits), and the final output at the decoder is divided by 8 and rounded (ie shifted right by 3 bits). Operations in scaled precision mode minimize round-off errors and yield improved rate-distortion performance.

In unscaled precision mode, there is no such scaling. An encoder or decoder operating in unscaled precision mode has to deal with a smaller dynamic range of transform coefficients and thus has lower computational complexity. However, there is a small penalty in compression efficiency for operating in this mode. Lossless coding (without quantization, i.e. setting the quantization parameter ie QP to 1) can only use the unscaled precision mode to get the guaranteed reversibility.

The precision mode used by the encoder when creating the compressed file is explicitly signaled using NO_SCALED_FLAG in the image header of the compressed bitstream 220 (FIG. 2). It is proposed that the decoder 300 also use the same precision mode for its operations.

NO_SCALED_FLAG is a 1-bit syntax element in the image header specifying the precision mode as follows:

If NO_SCALED_FLAG == TRUE, unscaled mode shall be used for decoder operations.

If NO_SCALED_FLAG == FALSE, then scaling should be used. In this case, the scaling mode should be used for operations by properly rounding the output of the final stage (color conversion) by 3 bits.

The rate-distortion performance of lossy coding is good when using unscaled mode (ie, NO_SCALED_FLAG==FALSE), especially at low QP. However, when using unscaled mode, the computational complexity is lower for two reasons:

Smaller dynamic range expansion in unscaled mode means shorter words can be used for transform calculations, especially in combination with "LONG_WORD_FLAG". In a VLSI implementation, the reduced dynamic range expansion means that gate logic implementing more significant bits can be powered down.

Scaling mode requires an addition operation and a right shift of 3 bits on the decoder side (to implement division by 8). On the encoder side, it requires a left shift of 3 bits. Overall, this is slightly more computationally demanding than unscaled mode.

Also, unscaled mode allows more significant bits to be compressed than scaled mode. For example, using 32-bit arithmetic, unscaled mode permits lossless compression (and decompression) of up to 27 effective bits per sample. In contrast, scaling mode only allows 24-bit compression under the same circumstances. This is because the scaling process introduces three additional bits of dynamic range.

For both precision modes, the data value at the decoder does not exceed 16 signed bits for an 8-bit input. (However, intermediate operations within the transform stage can exceed this number.)

NOTE: The encoder sets NO_SCALED_FLAG to TRUE if lossless encoding is required (QP=1), even if only a subregion of the image requires lossless encoding.

Encoders can use either mode for lossy compression. It is recommended that decoders use the precision mode signaled by NO_SCALED_MODE for their operations. However, scaling the quantization level so that a stream created with a scaled precision mode and decoded with an unscaled precision mode (and vice versa) produces acceptable images in most cases.

6. Scaled arithmetic for increased accuracy

In one implementation of this representative encoder/decoder, the transformations (including color transformations) are integer transformations and are implemented through a series of lifting steps. In these lifting steps, truncation errors impair transformation performance. For the case of lossy compression, to minimize the penalty of truncation errors and thus maximize the transform performance, the input data for the transform needs to be left shifted by a number of bits. However, another highly desirable feature is that if the input image is 8 bits, the output of each transform should be within 16 bits. So the number of left shifts cannot be very large. This representative decoder implements techniques for scaling arithmetic to achieve both goals. Scaled arithmetic techniques maximize transform performance by minimizing the penalty of truncation errors, and still limit the output of each transform step to 16 bits when the input image is 8 bits. This enables simple 16-bit implementations.

The transform used in this representative encoder/decoder is an integer transform and is implemented by a lifting step. Most boosting steps involve right shifts, which introduce truncation errors. Transformations often involve multiple lifting steps, and accumulating truncation errors significantly impairs transformation performance.

One way to reduce the penalty of truncation error is to left shift the input data before the transform at the encoder, and right shift by the same number of bits after the transform (combined with quantization) at the decoder. As mentioned above, the representative encoder/decoder has a two-stage transform structure: optional first-stage overlap + first-stage CT + optional second-stage overlap + second-stage CT. Experiments have shown that a left shift of 3 bits is necessary to minimize truncation error. So, in the lossy case, the input data can be left shifted by 3 bits before color conversion, i.e. multiplied or upscaled by a factor of 8 (e.g. for the above scaling modes).

However, color conversions and transformations enlarge the data. If the input data is shifted left by 3 bits, the output of the second stage 4×4DCT has a dynamic range of 17 bits if the input data is 8 bits (the output of other transformations is still within 16 bits). This is highly undesirable as it prevents 16-bit implementation (which is a highly desired feature). To get around this, before the second stage 4x4CT, the input data is shifted right by 1 bit, and thus the output is also within 16 bits. Since the second-stage 4x4CT is only applied to 1/16 of the data (DC transform coefficients of the first-stage DCT), and the first-stage transform already amplifies this data, the damage of truncation errors is small.

So in the lossy case of an 8-bit image, on the encoder side, the input is left shifted by 3 bits before color conversion, and right shifted by 1 bit before the second stage 4×4CT. On the decoder side, 1 bit is left shifted before the first stage 4x4 IDCT and 3 bits are shifted right after color conversion.

7. Computing environment

The processing techniques described above for computational complexity and precision signaling in digital media codecs can be implemented on any of a variety of digital media encoding and/or decoding systems, including computers (various form factors, including servers, desktop computers, laptop computers, handheld computers, etc.); digital media recorders and players; image and video capture equipment (such as cameras, scanners, etc.); communication equipment (such as telephones, mobile phones, conferencing equipment, etc.); display, printing, or other presentation device; and other examples, etc. Computational complexity and precision signaling techniques in digital media codecs may be implemented with hardware circuitry, firmware controlling digital media processing hardware, and communications software executing on a computer or in other computing environments such as shown in FIG. 6 .

Figure 6 shows one general example of a suitable computing environment (600) in which described embodiments may be implemented. The computing environment (600) is not intended to suggest any limitation as to the scope of use or functionality of the invention, as the invention can be implemented in entirely different general-purpose or special-purpose computing environments.

Referring to Figure 6, the computing environment (600) includes at least one processing unit (610) and memory (620). In Figure 6, this most basic configuration (630) is enclosed within the dashed line. The processing unit (610) executes computer-executable instructions and may be a real or virtual processor. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. The memory (620) may be volatile memory (eg, registers, cache, RAM), non-volatile memory (eg, ROM, EEPROM, flash memory, etc.), or some combination of the two. The memory (602) stores software (680) implementing the described digital media encoding/decoding using computational complexity and precision signaling techniques.

A computing environment can have additional features. For example, computing environment (600) includes storage (640), one or more input devices (650), one or more output devices (660), and one or more communication connections (670). An interconnection mechanism (not shown), such as a bus, controller, or network, interconnects the various components of the computing environment (600). In general, operating system software (not shown) provides an operating environment for other software executing in the computing environment (600) and coordinates the activities of the various components of the computing environment (600).

Storage (640) may be removable or non-removable and includes magnetic disks, tape or cassettes, CD-ROM, CD-RW, DVD, or any other medium that can be used to store information and be accessed within the computing environment (600) . The storage (640) stores instructions for implementing the described software (680) for digital media encoding/decoding using computational complexity and precision signaling techniques.

The input device (650) may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computing environment (600). For audio, the input device (650) may be a sound card or similar device that accepts audio input in analog or digital form from a microphone or microphone array, or a CD-ROM reader that provides audio samples to the computing environment. The output device (660) may be a display, printer, CD recorder, or another device that provides output from the computing environment (600).

A communication connection (670) allows communication with another computing entity over a communication medium. Communication media convey information such as computer-executable instructions, compressed audio or video information, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired or wireless technologies implemented with electrical, optical, RF, infrared, acoustic or other carrier waves.

The encoding/decoding of digital media using flexible quantization techniques described herein may be described in the general context of computer-readable media. Computer readable media can be any available media that can be accessed within a computing environment. By way of example, and not limitation, for the computing environment (600), computer-readable media can include memory (620), storage (640), communication media, and combinations of any of the above.

Digital media encoding/decoding using computational complexity and precision signaling techniques described herein may be in the general context of computer-executable instructions executed in a computing environment on a target real or virtual processor, such as included in a program module described in. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functions of the program modules may be combined as desired in various embodiments or separated among the program modules. Computer-executable instructions for program modules may be executed in local or distributed computing environments.

For purposes of presentation, the detailed description uses terms such as "determine," "generate," "modify," and "apply" to describe computer operations in a computing environment. These terms are high-level abstractions for operations performed by computers and should not be confused with actions performed by humans. The actual computer operations that correspond to these terms vary depending on the implementation.

In view of the many possible embodiments to which the principles of the invention may be applied, the invention is claimed all such embodiments which come within the scope and spirit of the appended claims and their equivalents.

Claims (18)

1. A digital media decoding method, comprising: receiving a compressed digital media bitstream at a digital media decoder; parsing syntax elements from the bitstream that signal arithmetic precision for transform calculations during processing of the digital media data; and Output the reconstructed image. 2. The digital media decoding method of claim 1, wherein the syntax element signals to use one of high arithmetic precision or low arithmetic precision. 3. The digital media decoding method according to claim 2, wherein the high arithmetic precision is 32-bit digital processing, and the low arithmetic precision is 16-bit digital processing. 4. The digital media decoding method according to claim 2, further comprising: decoding a block of transform coefficients from said compressed digital media bitstream; where the syntax element signals use of the high arithmetic precision, applying an inverse transform to the transform coefficients using high arithmetic precision processing; and Where the syntax element signals use of the low arithmetic precision, an inverse transform is applied to the transform coefficients using low arithmetic precision processing. 5. The digital media decoding method according to claim 4, wherein the high arithmetic precision is 32-bit digital processing, and the low arithmetic precision is 16-bit digital processing. 6. The digital media decoding method as claimed in claim 2, further comprising: decoding a block of transform coefficients from said compressed digital media bitstream; The inverse transform is applied to the transform coefficients using high arithmetic precision processing, regardless of the arithmetic precision signaled via the syntax element. 7. A digital media encoding method, comprising: receiving digital media data at a digital media encoder; making a decision whether to use lower precision arithmetic for transform calculations during processing of said digital media data; expressing said decision whether to use lower-precision arithmetic for transform calculations by a syntax element in an encoded bitstream, wherein said syntax element can be used to communicate said decision to a digital media decoder; and The encoded bitstream is output. 8. The digital media encoding method according to claim 7, wherein said making a decision comprises: verifying that said lower precision arithmetic used for transform calculations produces the same decoder output as using higher precision arithmetic for transform calculations; and Based on the verification, a decision is made whether to use the lower precision arithmetic. 9. The digital media encoding method of claim 7, wherein the lower precision arithmetic is 16-bit arithmetic precision. 10. digital media encoding method as claimed in claim 7, is characterized in that, also comprises: making a determination whether to apply scaling of the input digital media data prior to transform encoding; and The decision whether to apply the scaling is represented by a syntax element in the encoded bitstream. 11. The digital media encoding method according to claim 10, wherein said determining whether to apply scaling comprises deciding not to apply scaling to said input digital media data when encoding said digital media data losslessly . 12. A digital media decoding method, comprising: receiving a compressed digital media bitstream at a digital media decoder; parsing syntax elements from the bitstream that signal a precision mode selection for transform calculations during processing of the digital media data; scaling the output of the decoder if a scaled first-precision mode is signaled; where a second precision mode without scaling is signaled, omitting to apply the scaling of the output; and Output the reconstructed image. 13. The digital media decoding method according to claim 12, wherein said scaling the output of said decoder comprises rounding and dividing said output by a certain number. 14. The digital media decoding method according to claim 12, characterized in that, the rounding and division of the output is a rounding and division of the number 8. 15. The digital media decoding method according to claim 12, further comprising: parsing a second syntax element from the bitstream, the second syntax element signaling whether to use lower arithmetic precision for transform calculations during processing of the digital media data; decoding a block of transform coefficients from said compressed digital media bitstream; and In the case of the second precision mode without scaling and the use of a lower arithmetic precision is signaled, the inverse transform process of the transform coefficients is performed using the lower arithmetic precision. 16. The digital media decoding method according to claim 15, wherein the lower arithmetic precision is 16-bit arithmetic precision. 17. The digital media decoding method as claimed in claim 12, wherein said digital media data is encoded using a two-stage transform structure, said two-stage transform structure having a first-stage transform followed by a conversion of said two-stage transform structure. The second stage transformation of the DC coefficient of the first stage transformation, the digital media decoding method also includes: decoding digital media data from said compressed digital media bitstream; applying an inverse second stage transform to the digital media data; applying an inverse first stage transform to the digital media data; performing color conversion of said digital media data; and Wherein, in case the first precision mode using scaling is signaled, said scaling of the output of said decoder comprises: shifting the digital media data to the left by a single bit prior to input to the inverse first-stage transform; After the color conversion, the digital media data is shifted right by 3 bits. 18. The digital media decoding method according to claim 12, wherein the compressed digital media bit stream is encoded according to a syntax pattern defining separate main image planes and alpha image planes of an image, the syntax elements signaling a selection of a precision mode signaled per picture plane, whereby the precision modes of said main picture plane and said alpha picture plane are signaled independently, and said decoding method comprises performing a parsing signaled said action of said syntax element for selection of a precision mode for each image plane, and if said first precision mode using scaling is signaled for said corresponding image plane, scaling said corresponding image plane output of the decoder.

Images