计算机图形学学科研讨会基于图像绘制（IBR）技术综述（A Survey on Image-based Rendering Techniques）石教英浙江大学计算机学院计算机辅助设计与图形学国家重点实验室2005年5月目录一、基于图像绘制（IBR）技术定义二、典型IBR技术演示1、Panorama Mosaics2、Tour-Into-Picture3、Light Field4、Feature-based Morphing三、IBR技术理论基础-Plenoptic Funciton1、An introduction2、how to handle with plenoptic fuction四、IBR技术发展1、Image Matting2、Digital PhotoMontage3、High-Dynamic-Range Image Display4、Plenoptic photography五、基于图像绘制技术定义的扩展TransformRasterization(& Lighting)¾Computer Vision methods to recover models.Image-Based Rendering& ModelingRendering& Modeling VolumeRendering一、基于图像绘制（IBR）技术定义z Problems of triangle-based graphics:•Always starts from scratch.•Millions of sub-pixel triangles.一、基于图像绘制（IBR）技术定义¾Definition of IBR (by Sing Bing Kang)image-based rendering techniques rely on interpolation using the original set of input images or pixel reprojection from source images onto the target image in order to produce a novel virtual view一、基于图像绘制（IBR）技术定义¾Definition of IBR (by Cha Zhang & Tsuhan Chen of CMU) Given a continuous plenoptic function that describes a scene, image-based rendering is a process of two stages: –sampling and rendering. In the sampling stage, samples are taken from the plenoptic function for representation and storage. In the rendering stage, the continuous plenopticfunction is reconstructed with the captured samples.二、典型IBR技术演示¾Paronoma MosaicsDemo二、典型IBR技术演示¾Tour-Into-Picture二、典型IBR技术演示Lightfield Video二、典型IBR技术演示Feature-based Morphing三、IBR技术理论基础-Plenoptic Funciton1. An Introductionz Two ways of describing the world:• A source description: ——The world can be described bygeometric models, texture maps, refection models, lightingand shading models.•An appearance description: ——The world can be describedby the dense array of light rays filling the space, which canbe observed by posing eyes or cameras in the space. Thelight rays can be represented through the Plenoptic function.z The traditional model-based rendering approachadopts the source description method.z The image-based rendering approach adopots theplenoptic function to describe the world.三、IBR 技术理论基础-Plenoptic Funciton ¾IBR: An Old Storyz Plenoptic function z As pointed out by Adelson and Bergen (1991):The world is made of three -dimensional objects, but these objects do not communicate their properties directly to an observer. …The plenoptic function serves as the sole communication link between the physical objects and their corresponding retinal images. It is the intermediary between the world and the eye.),,,,,,(t Z Y X f λφθ三、IBR技术理论基础-Plenoptic Funciton ¾7D Plenoptic Function三、IBR技术理论基础-Plenoptic Funciton2. How to Handle Plenoptic Function?z Two Stages•sampling and reconstruction of sampled signals z Two directions for simplication•Restrain the viewing space of the viewersz View pointz Perceptual•Introduce some source descriptions into IBRz Geometryz Depth三、IBR 技术理论基础-Plenoptic Funciton ¾Restraining in View Spacez Assumption 1: Wavelength•Constant wavelength•RGB•Almost all the practical representations of IBR make this assumption.),,,,,,(t Z Y X f λφθ三、IBR 技术理论基础-Plenoptic Funciton ¾Restraining in View Spacez Assumption 2: Air•Air is translucent•Radiances along a light ray through empty space remain constant.),,,,,,(t Z Y X f λφθ三、IBR 技术理论基础-Plenoptic Funciton ¾Restraining in View Spacez Assumption 3: Time•Static scene•Images captured at different time and positions can be used together to render novel views.•Too large size for dynamic scene),,,,,,(t Z Y X f λφθ三、IBR 技术理论基础-Plenoptic Funciton ¾Restraining in View Spacez Assumption 4: Viewpoint•The viewer is constrained to be on a surface•Acceptable:zHuman eyes are usually at a certain height -level z Human eyes are less sensitive to vertical parallax and lighting changes),,,,,,(t Z Y X f λφθ三、IBR 技术理论基础-Plenoptic Funciton ¾Restraining in View Spacez Assumption 5: Viewpoint•The viewer moves along a certain path.•Reduces 2dimensions from the full plenoptic function.•Too large size for dynamic scene),,,,,,(t Z Y X f λφθ三、IBR 技术理论基础-Plenoptic Funciton ¾Restraining in View Spacez Assumption 6: Viewpoint•The viewer has a fixed position.•Reduces the dimension of the plenoptic function by 3.•No 3D effects can possibly be perceived.•Similarity to regular images and videos.),,,,,,(t Z Y X f λφθ三、IBR技术理论基础-Plenoptic Funciton ¾Various Representations三、IBR技术理论基础-Plenoptic Funciton¾6D –The Surface Plenoptic Function z Assumption 2. As we discussed, whenradiance along a light ray through emptyspace remains constant.z6D•Position on the surface (2D)•Light ray direction(2D)•Time (1D) and wavelength (1D).三、IBR技术理论基础-Plenoptic Funciton¾Examplez The surface light field couldbe considered asdimension-reduced versionof SPF.D. N. Wood, D. I. Azuma, K.Aldinger, B. Curless, T.Duchamp, D. H. Salesinand W. Stuetzle, “Surfacelight fields for 3Dphotography”, ComputerGraphics (SIGGRAPH’00),July 2000.三、IBR技术理论基础-Plenoptic Funcitonz5D –Plenoptic Modeling and Light field Video Plenoptic modeling, which is a 5D function•3D for the camera position, 2D for the cylindricalimage.L. McMillan and G. Bishop, “Plenoptic modeling: animage-based rendering system”, ComputerGraphics (SIGGRAPH’95), August 1995,三、IBR技术理论基础-Plenoptic Funciton¾Plenoptic Modelingz To render a novel view from the 5D representation, the close-by cylindrical projected images are warped to the viewingposition based on their epipolar relationship and some visibility tests.三、IBR技术理论基础-Plenoptic Funciton¾4D –Light field / Lumigraphz Assumption 1, 2 and 3z Trickily parameterize ray space to 2-plane representation.uf(t sv),,,三、IBR技术理论基础-Plenoptic Funciton三、IBR技术理论基础-Plenoptic Funciton¾3D –Concentric Mosaicsz Assumption 1, 2, 3 and 4Tripodz Center camera => panoramaz Off-centered cameras => motion parallax…...Top view三、IBR技术理论基础-Plenoptic Funciton¾2D –Image Mosaicingz Composes one single mosaic with multipleinput imagesz In most cases, the light rays recorded in the mosaic share the same center-of-projection(COP) -Panoramic mosaic or panorama三、IBR技术理论基础-Plenoptic Funciton ¾Examples三、IBR技术理论基础-Plenoptic Funciton¾2D –Image Mosaicingz In the more general scenario, the cameras of the input images can move in free form andthe resultant mosaic has MCOPsz Manifold Mosaic.三、IBR技术理论基础-Plenoptic Funciton ¾Examples三、IBR技术理论基础-Plenoptic Funciton¾Quicktime VRz Using environmentalmaps•Cylindrical•Cubic•sphericalz At a fixed point,sample all the raydirections.z Users can look in bothhorizontal and verticaldirections三、IBR技术理论基础-Plenoptic Funciton ¾Mars Pathfinder Panorama三、IBR技术理论基础-Plenoptic Funciton ¾IBR with Various Source Descriptions三、IBR技术理论基础-Plenoptic Funciton ¾Correspondence between Images三、IBR技术理论基础-Plenoptic Funciton ¾Example 1: View Morphing(SIGGRAPH 96) Steve Seitz etc.三、IBR技术理论基础-Plenoptic Funciton ¾Dense Depth MapViewing RegionSprite withDepthLDIEnvironment Map三、IBR技术理论基础-Plenoptic Funciton¾Layered Depth Imagesz Idea:•Handle disocclusion•Store invisible geometry in depth images三、IBR技术理论基础-Plenoptic Funciton¾Texture Map (+ Scene Geometry) z Image based modelingz View dependent texture mapping三、IBR技术理论基础-Plenoptic Funciton¾Reflection Models (+ Scene Geometry) z Image based relighting。