WO2013077508A1

WO2013077508A1 - Device and method for depth map generation and device and method using same for 3d image conversion

Info

Publication number: WO2013077508A1
Application number: PCT/KR2012/003347
Authority: WO
Inventors: 우대식; 김종대; 박재범; 전병기; 정원석
Original assignee: 에스케이플래닛 주식회사; 시모스 미디어텍(주)
Priority date: 2011-11-24
Filing date: 2012-04-30
Publication date: 2013-05-30
Also published as: KR101660808B1; KR20130057586A

Abstract

The present invention relates to a device and method for depth map generation and to a device and method using same for 3D image conversion. The depth map generation device includes: a feature information extraction part extracting at least one piece of feature information with respect to an input image; a depth map initialization part generating a depth map with respect to the input image on the basis of the feature information; an FFT conversion part performing an FFT with respect to the conversion of the input image into a frequency image; and a depth map determination part evaluating a correlation value by using a representative value of the frequency image and the mean value of the initial depth map to determine a final depth map on the basis of the correlation value. According to the present invention, the error of the depth map for expressing a 3D image generated when an image is automatically converted may be corrected.

Description

Depth map generation device and method and stereoscopic image conversion device and method using same

The present invention relates to an apparatus and method for generating a depth map, and to an apparatus and method for converting a stereoscopic image using the same, more particularly, extracting at least one characteristic information of an input image, and based on the characteristic information, After generating an initial depth map, performing a Four Fourier Transform (FFT) on the input image and converting it into a frequency image, obtaining a correlation value using the representative value of the frequency image and the average value of the initial depth map, A depth map generating apparatus and method for determining a final depth map based on the correlation value, and a stereoscopic image converting apparatus and method using the same.

Recently, as interest in 3D images is amplified, research on 3D images is being actively conducted.

In general, it is known that humans feel the most three-dimensional effect by the parallax between both eyes. Thus, 3D imaging can be implemented using these characteristics of humans. For example, by distinguishing a particular subject into a left eye image seen through the viewer's left eye and a right eye image seen through the viewer's right eye, the viewer simultaneously displays the left eye image and the right eye image so that the viewer views the 3D image as a 3D image. I can make it visible. As a result, the 3D image may be implemented by manufacturing a binocular image divided into a left eye image and a right eye image and displaying the same.

In order to convert a monocular 2D image without depth information into a 3D image, it is necessary to add depth information to the 2D image to render.

In general, stereoscopic conversion is divided into manual and automatic methods. The manual method literally creates a depth map while watching the image according to the subjective judgment of a person about all the images. This process is based on the subjective judgment of the person who can predict the depth map while watching the video. Therefore, a person directly produces a depth map for each image, and the error of the depth map is very small. However, a lot of time and effort is required because a person directly intervenes in each video to create a depth map of the video.

Automatic stereoscopic transformation means analyzing the characteristics of an image to extract an appropriate depth map and using it to generate left and right stereoscopic images. In this process, since the image itself does not have information about the depth map, the depth map is generated by using general image characteristics such as edge characteristics, color, brightness characteristics, and vanishing point characteristics of the image. However, these features often do not match the stereoscopic characteristics of the image itself, so the effect of stereoscopic is not large.

In addition, a single image includes various types of image contents. It is virtually impossible to extract a depth map through image processing of each image, and the depth map obtained through image processing also includes many errors.

The error of the depth map obtained through such image processing can be divided into two types.

One is an error or inversion of the depth map in the partial region of the image and a combination thereof, and another is an error or inversion of the depth map in the entire image. Of course, the error of the depth map is not easy to distinguish by technical methods such as image processing.

Therefore, there is a need for a technique capable of automatically detecting an error of a depth map from an objective point of view only with an image.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and an object of the present invention is to provide a depth map generating apparatus capable of correcting an error of a depth map for stereoscopic representation generated during an automatic stereoscopic conversion process of an image; A method and an apparatus and method for converting a stereoscopic image using the same are provided.

Another object of the present invention is to provide an apparatus and method for generating a depth map that can objectively detect and correct an error of a depth map of an image generated through image processing during automatic stereoscopic image conversion, and an apparatus and method for stereoscopic image conversion using the same. have.

Another object of the present invention is to provide a depth map generating apparatus and method for correcting errors in a depth map and minimizing errors in image conversion by converting a 2D image into a 3D image using the corrected depth map. And a stereoscopic image conversion apparatus and method using the same.

According to an aspect of the present invention, a feature information extractor for extracting at least one feature information of an input image, a depth map initializer for generating an initial depth map for the input image based on the feature information, the input image An FFT transform unit converts a frequency image by performing a fast fourier transform (FFT), and obtains a correlation value by using a representative value of the frequency image and an average value of the initial depth map, and finally obtains a correlation value based on the correlation value. There is provided a depth map generator including a depth map determiner for determining a depth map.

The feature information extracting unit extracts feature information including at least one of edge information, color information, luminance information, motion information, and histogram information.

The depth map initialization unit divides a plurality of pixels constituting the input image into at least one block, and then sets an initial depth value for the at least one block to set an initial depth map ( Create a depth map.

The depth map determiner obtains a representative value by summing pixel values corresponding to a high frequency region in the frequency image, and obtains an average value of depth values corresponding to a block region corresponding to the region obtained from the initial depth map. A correlation value is obtained using the representative value and the average value.

The depth map determiner calculates a correlation value (CRV) using the following equation.

[Equation]

CRV = Σ (FFT (n) * Depth (n))

Here, the FFT (n) is a representative value representing the block sharpness of the frequency image, the Depth (n) is the average value of the initial depth map corresponding to the block area matching the block area of the FFT (n), n Is the index of each block.

The depth map determiner may determine the initial depth map as the final depth map when the correlation value is greater than or equal to a predetermined threshold value, and invert the depth values of the initial depth map when the correlation value is not greater than the threshold value. Determine the map as the final depth map.

According to another aspect of the present invention, an image analysis unit for extracting at least one feature information by analyzing a two-dimensional input image, generates an initial depth map for the input image based on the feature information, After converting to a frequency image by performing an FFT, a correlation value is obtained by using a representative value of the frequency image and an average value of the initial depth map, and a depth map setting for determining a final depth map based on the correlation value. A stereoscopic image converting apparatus including a stereoscopic image generator for converting the input image into a three-dimensional stereoscopic image using the final depth map is provided.

The depth map setting unit may include a depth map initializer configured to generate an initial depth map of the input image based on the characteristic information, an FFT converter configured to perform an FFT on the input image, and convert the image into a frequency image; And a depth map determiner that obtains a correlation value by using a representative value and an average value of the initial depth map, and determines a final depth map based on the correlation value.

According to another aspect of the present invention, in the method for generating a depth map by the depth map generating apparatus, extracting at least one characteristic information for the input image, the initial depth for the input image based on the characteristic information Generating a map, performing an FFT on the input image, converting the image into a frequency image, obtaining a correlation value using a representative value of the frequency image and an average value of the initial depth map, and calculating the correlation value. A depth map generation method is provided that includes determining a final depth map as a basis.

The obtaining of the correlation value by using the representative value of the frequency image and the average value of the initial depth map may include obtaining a representative value by adding pixel values corresponding to a high frequency region in the frequency image, and representing the representative value in the initial depth map. Obtaining an average value of depth values corresponding to a block area coinciding with the obtained area, and obtaining a correlation value using the obtained representative value and the average value.

The determining of the final depth map based on the correlation value may include determining the initial depth map as the final depth map when the correlation value is greater than or equal to a predetermined threshold value, and determining the final depth map as the final depth map. And inverting the depth values to determine the inverted depth map as the final depth map.

According to another aspect of the invention, the step of extracting at least one characteristic information for the input image, generating an initial depth map for the input image based on the characteristic information, performing an FFT on the input image Converting the frequency image into a frequency image, obtaining a correlation value using an average value of the representative value of the frequency image and the initial depth map, and determining a final depth map based on the correlation value. A recording medium is provided which is produced by a program and can be read by an electronic device.

According to another aspect of the present invention, in a method for automatically converting a stereoscopic image by a stereoscopic image converting apparatus, analyzing the input two-dimensional input image to extract at least one characteristic information, based on the characteristic information Generating an initial depth map with respect to the input image, determining a final depth map by checking the validity of the initial depth map, and converting the input image into a three-dimensional stereoscopic image using the final depth map. There is provided a stereoscopic image conversion method comprising the step.

The determining of the final depth map may include generating an initial depth map of the input image based on the characteristic information, performing an FFT on the input image, and converting the input image into a frequency image. Obtaining a representative value by summing pixel values corresponding to an area, obtaining an average value of depth values corresponding to a block area corresponding to the area where the representative value is obtained from the initial depth map, and representing the representative value of the frequency image and the initial value Obtaining a correlation value using an average value of the depth map; when the correlation value is equal to or greater than a predetermined threshold, determining the initial depth map as a final depth map; and when the correlation value is not greater than or equal to the threshold, a depth value of the initial depth map. Inverting them to determine the inverted depth map as the final depth map.

According to another aspect of the invention, the step of extracting at least one characteristic information by analyzing the input two-dimensional input image, generating an initial depth map for the input image based on the characteristic information, the initial depth Determining a final depth map by checking whether a map is valid, and converting the input image into a three-dimensional stereoscopic image using the final depth map. A readable recording medium is provided.

Therefore, according to the present invention, it is possible to correct an error of a depth map for the stereoscopic representation that occurs during the automatic stereoscopic conversion of the image.

In addition, an error in a depth map of an image generated through image processing during automatic stereoscopic image conversion may be objectively detected and corrected.

In addition, the error of the depth map may be corrected, and the error of the image conversion may be minimized by converting the 2D image into the 3D image using the corrected depth map.

1 is a block diagram showing the configuration of a stereoscopic image conversion apparatus according to the present invention.

Figure 2 is a block diagram schematically showing the configuration of the depth map generating apparatus according to the present invention.

3 is an exemplary view for explaining the difference between the initial depth map and the final depth map when the input image is converted into a three-dimensional stereoscopic image according to the present invention.

4 is an exemplary diagram for explaining an image of an FFT high frequency component according to the present invention;

5 is an exemplary view of a block representing the 8x8 block in the frequency domain through the FFT according to the present invention.

6 is a diagram illustrating a method for converting a two-dimensional input image into a three-dimensional stereoscopic image by the stereoscopic image conversion apparatus according to the present invention.

7 is a flowchart illustrating a method of generating a depth map by a depth map generating apparatus according to the present invention;

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same or corresponding components will be given the same reference numerals and redundant description thereof will be omitted.

Hereinafter, a depth map representing a stereoscopic image is generated and corrected for an image object satisfying the following two prerequisites.

Precondition 1 is "usually a bright color or a strong brightness is often near." For example, the more distant the mountain in the picture you see, the more the color looks achromatic compared to the nearby mountain. Or, the farther and brighter the object, the less saturated the color becomes. Such a feature is not only a visual characteristic of a person but also a feature of a camera photographing it.

Precondition 2 is "usually, the distance between the clear parts of the image is often close." For example, when the same object is near and far, the sharpness of the object is different, which occurs because of the limitations of the resolution of the camera being visually or photographed.

The present invention intends to transform the stereoscopic and correct it by utilizing the two characteristics as described above. Precondition 1 is a condition for converting three-dimensional, and precondition 2 is a means for correcting this. That is, the depth map is extracted from the color and brightness information of the image, and the error of the depth map is detected and corrected through objective verification of whether the depth map is partially or totally valid.

Referring to FIG. 1, the 3D image converting apparatus 100 includes an image analyzer 110, a depth map setting unit 120, and a 3D image generating unit 130.

The image analyzer 110 extracts at least one characteristic information by analyzing a two-dimensional input image. The characteristic information includes edge information, color information, luminance information, motion information, histogram information, and the like.

The depth map setting unit 120 generates an initial depth map of the input image based on the characteristic information extracted by the image analyzer 110, and performs a fast fourier transform (FFT) on the input image. After converting to a frequency image, a correlation value is obtained using the representative value of the frequency image and the average value of the initial depth map, and a final depth map is determined based on the correlation value.

A detailed description of the depth map setting unit 120 will be described with reference to FIG. 2.

The stereoscopic image generator 130 converts the two-dimensional input image into a three-dimensional stereoscopic image using the final depth map determined by the depth map setting unit 120. For example, the stereoscopic image generation unit 130 may generate parallax information using the final depth map, and generate a 3D stereoscopic image using the parallax information. The 3D stereoscopic image generated in this way looks more stereoscopic as the depth values for each pixel in each frame vary.

Herein, the stereoscopic image generator 130 converts the 2D image into a 3D stereoscopic image using parallax information. However, the stereoscopic image generator 130 converts the input image using the final depth map. The method of converting to a stereoscopic image follows various conventional methods.

The stereoscopic image converting apparatus 100 configured as described above may convert a 2D input image into a 3D image by setting a depth value for the input image based on the characteristic information of the input image.

2 is a block diagram schematically illustrating a configuration of a depth map generating apparatus according to the present invention, and FIG. 3 illustrates a difference between an initial depth map and a final depth map when converting an input image into a 3D stereoscopic image. 4 is an exemplary diagram for explaining an image of an FFT high frequency component according to the present invention, and FIG. 5 is an exemplary diagram of a block in which a 8x8 block according to the present invention is expressed in a frequency domain through an FFT.

In FIG. 1, the depth map setting unit is described, but in FIG. 2, the depth map generator 200 will be described.

Referring to FIG. 2, the depth map generator 200 may include a feature information extractor 210, a depth map initializer 220, a fast fourier transform (FFT) transformer 230, and a depth map determiner 240. Include.

The characteristic information extractor 210 extracts at least one characteristic information with respect to the input image. The input image may be a monocular image.

The feature information extracted by the feature information extractor 210 may be edge information, color information, luminance information, motion information, or histogram information.

The feature information extractor 210 extracts feature information in an image through various analysis methods in units of pixels or blocks in order to collect information that is a basis for generating a depth map.

The depth map initializer 220 generates an initial depth map of the input image based on the feature information extracted by the feature information extractor 210.

That is, the depth map initialization unit 220 divides a plurality of pixels constituting the input image into at least one block, and then initializes an initial depth value of the at least one block. Set to create an initial depth map.

The depth map initializer 220 generates a depth map for each frame of the 2D image based on the extracted characteristic information. That is, the depth map initializer 220 extracts depth values for each pixel of each frame from a depth map of the 2D image. Here, the depth map is a data structure storing depth values of each pixel per frame for the 2D image.

An example of converting an input image into a 3D stereoscopic image using the initial depth map generated by the depth map initialization unit 220 will be described with reference to FIG. 3.

Based on the observer, the spider is placed closest to the observer, the next flower is located, and the background is farthest from the observer. For example, the spider, flower, and background can be separated in three stages according to the position. Do. Although it can be expressed in more detail, it will be divided into three steps as an example for explanation.

This is a subjective human visual characteristic. This analysis is possible, but if it is to be analyzed automatically through image processing, the spider that is considered to be the most front using the precondition 1 is achromatic and dark and is the farthest. Will be an expression.

FIG. 3A illustrates a stereoscopic image converted into a depth map according to Precondition 1. FIG.

In addition, if the depth map generated by the depth map initialization unit 220 is changed like a film image inverted like a print film of a photograph, a normal depth map may be formed.

That is, if the depth map is reversed like a print film of a photograph, the initial depth map of FIG. 3A can be converted into a depth map as shown in FIG. 3B, and FIG. 3B can be a three-dimensional expression with less error than that of FIG.

In the image of FIG. 3B, the spider is bright because the spider is in front, and then the flower area, and the background is dark because there is little change. As a result, when the depth map of FIG. 3B is selected, the stereoscopic representation is possible while improving the error.

Therefore, the depth map generating apparatus 200 should determine whether to select and convert an image to the depth map of FIG. 3A or to convert to a depth map of FIG. 3B.

The depth map generating apparatus 200 performs a fast fourier transform (FFT) on the input image to determine whether to select and convert to the depth map of FIG. 3A or to convert to the depth map of FIG. 3B. The final depth map is generated using the frequency image.

Therefore, the depth map generating apparatus 200 includes an FFT converter 230 and a depth map determiner 240.

The FFT converter 230 performs an FFT on the input image to convert the input image into a frequency image in a frequency domain. That is, the FFT converter 230 converts a spatial input image into a frequency image.

The FFT converter 230 performs an FFT with reference to FIG. 4 for an image obtained by converting an input image into a frequency image. Referring to FIG. 4, the brighter portion has a clearer or stronger outline. In other words, if the high frequency area in the frequency domain, that is, the area expressing sharpness or strong exterior, is expressed in bright colors, the outline of the spider is bright in the image, and then the flower area and the background are dark because there is little change. do.

The image representation as shown in FIG. 4 is a characteristic of the FFT. FIG. 5 (a) represents an arbitrary region of 8x8 pixels in the input image, and FIG. 5 (b) shows (a) as the frequency region through the FFT.

In FIG. 5B, region A refers to a low frequency band and region B refers to a high frequency band. That is, the B region is a high frequency component. If the image has a clear or strong outline, the value of the B region is increased, and the simpler the value of the upper portion of the A region is, the smaller the value of the B region is.

As such, the sum of pixel values belonging to the high frequency region in the frequency image, that is, the sum of the B region representing the sharpness in the 8x8 block belonging to the partial region of the image and the large deviation between each pixel of the image is expressed as 8x8. It is defined as a representative value corresponding to pixel. In this way, when the size value of each block unit is expressed as an image, the image is expressed using the FFT high frequency component as shown in FIG. 4.

The depth map determiner 240 obtains a correlation value by using a representative value of the frequency image generated by the FFT converter 230 and an average value of the initial depth map, and based on the correlation value, a final depth map. Determine.

That is, the depth map determiner 240 obtains a representative value FFT (n) representing a block sharpness in arbitrary 8 × 8 block units in the frequency image formed by the FFT converter 230. Here, the representative value is obtained by summing pixel values corresponding to a high frequency region in the frequency image. Then, the depth map determiner 240 obtains an average value Depth (n) of the 8x8 block of the initial depth map that matches the block area of the FFT (n), and then correlates the representative value with the average value. Obtain the value (Co-Relation Value).

The depth map determiner 240 obtains a correlation value (CRV (Co-Relation Value)) using Equation 1.

[Equation 1]

CRV = Σ (FFT (n) * Depth (n))

Here, the FFT (n) is a representative value representing the block sharpness of the frequency image, Depth (n) is the average value of the initial depth map corresponding to the block area that matches the block area of the FFT (n), n is It means the index of each block. The FFT (n) refers to a sum of pixel values corresponding to a high frequency region in an FFT-converted frequency image, and an average value of the initial depth map corresponds to a block region corresponding to a region obtained by obtaining the representative value in the initial depth map. The mean value of the depth values.

When the correlation value is obtained through Equation 1, the depth map determiner 240 determines the initial depth map as the final depth map when the correlation value is greater than or equal to a predetermined threshold value, and when the correlation value is not greater than or equal to the threshold value. The depth values of the initial depth map are inverted to determine the inverted depth map as the final depth map.

As a result, the depth map determiner 240 is a depth map for automatically converting an image to a stereoscopic image by using a frequency image converted into an image of the frequency domain as shown in FIG. 4. It is concluded that

In addition, the depth map determiner 240 is a precondition if the condition that the color of the precondition 1 is stronger and brighter than the condition determined to be in front and the condition that the sharp part is in front match, that is, the correlation value is greater than or equal to a predetermined threshold. It is determined that the depth map of FIG. 3A generated as 1 is valid.

If the correlation value is not greater than or equal to a predetermined threshold value, the depth map determiner 240 determines that the depth map reflecting the inversion phenomenon is a more suitable depth map as shown in FIG. 3B.

The depth map generation apparatus 200 configured as described above extracts an initial depth map using characteristic information such as color and brightness information of an image, and performs an objective verification on whether the initial depth map is partially or entirely valid. Detects the error and generates the final depth map.

6 is a diagram illustrating a method of converting a two-dimensional input image into a three-dimensional stereoscopic image by the stereoscopic image conversion apparatus according to the present invention.

Referring to FIG. 6, the stereoscopic image conversion apparatus analyzes a two-dimensional input image to extract at least one characteristic information (S602). Here, the characteristic information includes edge information, color information, luminance information, motion information, histogram information, and the like.

After performing the step S602, the 3D image conversion apparatus generates an initial depth map of the input image based on the characteristic information (S604).

Then, the stereoscopic image conversion apparatus determines the final depth map through objective verification of whether the initial depth map is partially or entirely valid (S606).

That is, the stereoscopic image converting apparatus performs an FFT on the input image and converts the image into a frequency image. Then, the stereoscopic image conversion apparatus obtains a representative value by summing pixel values belonging to a high frequency region in the frequency image, and obtains an average value of depth values corresponding to a block region that matches the region where the representative value is obtained in the initial depth map. Then, the 3D image conversion apparatus obtains a correlation value using Equation 1 using the obtained representative value and the average value.

Thereafter, the stereoscopic image conversion apparatus determines a final depth map based on the correlation value.

After performing S606, the stereoscopic image conversion apparatus converts the input image into a 3D stereoscopic image using the determined final depth map (S608).

7 is a flowchart illustrating a method of generating a depth map by a depth map generating apparatus according to the present invention.

Referring to FIG. 7, the depth map generating apparatus extracts at least one characteristic information of an input image, and generates an initial depth map of the input image based on the extracted characteristic information (S702). That is, the depth map generator extracts characteristic information such as edge information, color information, luminance information, motion information, histogram information, and the like. Then, the depth map generating apparatus divides a plurality of pixels constituting the input image into at least one block, and then sets an initial depth value for the at least one block to initialize the depth map. Create a depth map.

After performing the operation S702, the depth map generating apparatus performs an FFT on the input image and converts the frequency image into a frequency image (S704).

After performing S704, the depth map generator determines whether the converted frequency image is an image that satisfies a precondition (S706). In this case, the precondition is a precondition 2, and the precondition 2 is " usually, a clear portion in the image is often close in distance. &Quot;

If the determination result of the step S706 is an image satisfying the precondition, the depth map generator obtains a correlation value using the representative value of the frequency image and the average value of the initial depth map (S708). A detailed description of the method for obtaining the correlation value will be given with reference to FIG. 2.

After performing S708, the depth map generator determines whether the obtained correlation value is greater than or equal to a predetermined threshold value (S710).

When the determination result of S710 is greater than or equal to a threshold, the depth map generator determines the initial depth map as a final depth map (S712).

If the correlation value is not greater than or equal to a threshold as a result of the determination in S710, the depth map generator inverts depth values of the initial depth map (S714), and converts a depth map including the inverted depth values into a final depth map. Determine (S716).

In the present invention, a computer program (also known as a program, software, software application, script, or code) included in a depth map generating device or a stereoscopic image conversion device and executing a method according to the present invention is a compiled or interpreted language or a priori. Or written in any form of programming language, including procedural languages, and may be deployed in any form, including stand-alone programs or modules, components, subroutines, or other units suitable for use in a computer environment. A computer program does not necessarily correspond to a file in a file system. A program may be in a single file provided to the requested program, in multiple interactive files (eg, a file that stores one or more modules, subprograms, or parts of code), or part of a file that holds other programs or data. (Eg, one or more scripts stored in a markup language document). The computer program may be deployed to run on a single computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, such as magnetic disks such as internal hard disks or external disks, magneto-optical disks, and CD-ROMs. And all forms of nonvolatile memory, media and memory devices, including DVD-ROM discs. The processor and memory can be supplemented by or integrated with special purpose logic circuitry.

Implementations of the functional operations and subject matter described herein may be implemented in digital electronic circuitry, in computer software, firmware, or hardware including the structures and structural equivalents disclosed herein, or in a combination of one or more of these. Can be. Implementations of the subject matter described in this specification are one or more modules relating to computer program instructions encoded on a program storage medium of tangible type for controlling or by the operation of one or more computer program products, ie data processing apparatuses. It can be implemented as.

Implementations of the subject matter described herein may include, for example, a backend component such as a data server, or include a middleware component such as, for example, an application server, or a web browser or graphical user, for example, where a user may interact with the implementation of the subject matter described herein. It can be implemented in a computing system that includes a front end component such as a client computer having an interface or any combination of one or more of such back end, middleware or front end components. The components of the system may be interconnected by any form or medium of digital data communication such as, for example, a communication network.

Although the specification includes numerous specific implementation details, these should not be construed as limiting to any invention or the scope of the claims, but rather as a description of features that may be specific to a particular embodiment of a particular invention. It must be understood. Certain features that are described in this specification in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, while the features may operate in a particular combination and may be initially depicted as so claimed, one or more features from the claimed combination may in some cases be excluded from the combination, the claimed combination being a subcombination Or a combination of subcombinations.

Likewise, although the operations are depicted in the drawings in a specific order, it should not be understood that such operations must be performed in the specific order or sequential order shown in order to obtain desirable results or that all illustrated operations must be performed. In certain cases, multitasking and parallel processing may be advantageous. Moreover, the separation of the various system components of the above-described embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems will generally be integrated together into a single software product or packaged into multiple software products. It should be understood that it can.

As such, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without changing the technical spirit or essential features thereof. Therefore, the above-described embodiments are to be understood as illustrative in all respects and not as restrictive. The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

The present invention can objectively detect and correct errors in the overall depth map of an image through image processing during automatic stereoscopic image conversion, and convert the 2D image into a 3D image by using the corrected depth map. The present invention can be applied to a depth map generating apparatus and method and a stereoscopic image converting apparatus and method using the same.

Claims

A feature information extractor configured to extract at least one feature information of the input image;

A depth map initialization unit generating an initial depth map of the input image based on the characteristic information;

An FFT transform unit performing a fast fourier transform (FFT) on the input image to convert the image into a frequency image; And

A depth map determination unit obtaining a correlation value by using a representative value of the frequency image and an average value of the initial depth map, and determining a final depth map based on the correlation value;

Depth map generation device comprising a.
The method of claim 1,

The feature information extractor extracts feature information including at least one of edge information, color information, luminance information, motion information, and histogram information. Map generator.
The method of claim 1,

The depth map initializer divides a plurality of pixels constituting the input image into at least one block, and then sets an initial depth value for the at least one block. Depth map generation device characterized in that for generating a depth map).
The method of claim 1,

The depth map determiner obtains a representative value by summing pixel values corresponding to a high frequency region in the frequency image, and obtains an average value of depth values corresponding to a block region corresponding to the region obtained from the initial depth map. And a correlation value is calculated using the representative value and the average value.
The method of claim 4

The depth map determination unit obtains a correlation value (CRV (Co-Relation Value)) using the following equation.

[Equation]

CRV = Σ (FFT (n) * Depth (n))

Here, the FFT (n) is a representative value representing the block sharpness of the frequency image, the Depth (n) is the average value of the initial depth map corresponding to the block area that matches the block area of the FFT (n), n is each The index of a block.
The method of claim 1,

The depth map determiner determines the initial depth map as a final depth map when the correlation value is greater than or equal to a predetermined threshold value, and inverts the depth values of the initial depth map when the correlation value is not greater than the threshold value to determine the inverted depth map. Depth map generation device characterized in that for determining the final depth map.
An image analyzer extracting at least one characteristic information by analyzing a two-dimensional input image;

After generating an initial depth map of the input image based on the characteristic information, performing an FFT on the input image, converting the image into a frequency image, and using a representative value of the frequency image and an average value of the initial depth map. A depth map setting unit obtaining a correlation value and determining a final depth map based on the correlation value; And

A stereoscopic image generator for converting the input image into a three-dimensional stereoscopic image using the final depth map;

Stereoscopic image conversion device comprising a.
The method of claim 7, wherein

The depth map setting unit,

A depth map initialization unit generating an initial depth map of the input image based on the characteristic information;

An FFT converter which performs an FFT on the input image and converts the frequency image into a frequency image; And

And a depth map determiner which obtains a correlation value by using the representative value of the frequency image and the average value of the initial depth map, and determines a final depth map based on the correlation value.
In the depth map generating apparatus generates a depth map,

Extracting at least one characteristic information with respect to the input image;

Generating an initial depth map of the input image based on the characteristic information;

Performing an FFT on the input image and converting the same to a frequency image;

Obtaining a correlation value using an average value of the representative value of the frequency image and the initial depth map; And

Determining a final depth map based on the correlation value;

Depth map generation method comprising a.
The method of claim 9,

Obtaining a correlation value by using the representative value of the frequency image and the average value of the initial depth map,

Obtaining a representative value by summing pixel values corresponding to a high frequency region in the frequency image;

Obtaining an average value of depth values corresponding to a block area corresponding to the area from which the representative value is obtained in the initial depth map; And

And obtaining a correlation value using the obtained representative value and the average value.
The method of claim 9,

Determining a final depth map based on the correlation value,

If the correlation value is greater than or equal to a predetermined threshold value, the initial depth map is determined as the final depth map. If the correlation value is not greater than the threshold value, the depth values of the initial depth map are inverted to determine the inverted depth map as the final depth map. Depth map generation method characterized in that.
Extracting at least one characteristic information with respect to the input image;

Generating an initial depth map of the input image based on the characteristic information;

Performing an FFT on the input image and converting the same to a frequency image;

Obtaining a correlation value using an average value of the representative value of the frequency image and the initial depth map;

And determining a final depth map based on the correlation value. The recording medium may be recorded by a program and read by an electronic device.
In the stereoscopic image conversion apparatus automatically converts a stereoscopic image,

Extracting at least one characteristic information by analyzing the input two-dimensional input image;

Generating an initial depth map of the input image based on the characteristic information, and determining a final depth map by checking whether the initial depth map is valid; And

Converting the input image into a three-dimensional stereoscopic image using the final depth map;

Stereoscopic image conversion method comprising a.
The method of claim 13,

Determining the final depth map,

Generating an initial depth map of the input image based on the characteristic information;

Performing an FFT on the input image and converting the same to a frequency image;

Obtaining a representative value by summing pixel values corresponding to a high frequency region in the frequency image;

Obtaining an average value of depth values corresponding to a block area corresponding to the area from which the representative value is obtained in the initial depth map;

Obtaining a correlation value using an average value of the representative value of the frequency image and the initial depth map; And

If the correlation value is greater than or equal to a predetermined threshold value, the initial depth map is determined as the final depth map. If the correlation value is not greater than the threshold value, the depth values of the initial depth map are inverted to determine the inverted depth map as the final depth map. Stereoscopic image conversion method comprising the step of performing.
Extracting at least one characteristic information by analyzing the input two-dimensional input image;

Generating an initial depth map of the input image based on the characteristic information, and determining a final depth map by checking whether the initial depth map is valid; And

And converting the input image into a three-dimensional stereoscopic image using the final depth map, wherein the stereoscopic image conversion method is recorded by a program and can be read by an electronic device.