WebGL2 Image Processing

To draw images in WebGL we need to use textures. Similarly to the way WebGL expects clip space coordinates when rendering instead of pixels, WebGL generally ... English Deutsch 日本語 한국어 PortuguêsBrasileiro 简体中文 TableofContents WebGL2Fundamentals.org Fix,Fork,Contribute WebGL2ImageProcessing ImageprocessingiseasyinWebGL.Howeasy?Readbelow. ThisisacontinuationfromWebGL2Fundamentals. Ifyouhaven'treadthatI'dsuggestgoingtherefirst. TodrawimagesinWebGLweneedtousetextures.Similarlytotheway WebGLexpectsclipspacecoordinateswhenrenderinginsteadofpixels, WebGLgenerallyexpectstexturecoordinateswhenreadingatexture. Texturecoordinatesgofrom0.0to1.0nomatterthedimensionsofthetexture. WebGL2addstheabilitytoreadatextureusingpixelcoordinatesaswell. Whichwayisbestisuptoyou.Ifeellikeit'smorecommontouse texturecoordinatesthanpixelcoordinates. Sinceweareonlydrawingasinglerectangle(well,2triangles) weneedtotellWebGLwhichplaceinthetextureeachpointinthe rectanglecorrespondsto.We'llpassthisinformationfromthevertex shadertothefragmentshaderusingaspecialkindofvariablecalled a'varying'.It'scalledavaryingbecauseitvaries.WebGLwill interpolatethevaluesweprovideinthe vertexshaderasitdrawseachpixelusingthefragmentshader. Usingthevertexshaderfromtheendofthepreviouspost weneedtoaddanattributetopassintexturecoordinatesandthen passthoseontothefragmentshader. ... +invec2a_texCoord; ... +outvec2v_texCoord; voidmain(){ ... +//passthetexCoordtothefragmentshader +//TheGPUwillinterpolatethisvaluebetweenpoints +v_texCoord=a_texCoord; } Thenwesupplyafragmentshadertolookupcolorsfromthetexture. #version300es precisionhighpfloat; //ourtexture uniformsampler2Du_image; //thetexCoordspassedinfromthevertexshader. invec2v_texCoord; //weneedtodeclareanoutputforthefragmentshader outvec4outColor; voidmain(){ //Lookupacolorfromthetexture. outColor=texture(u_image,v_texCoord); } Finallyweneedtoloadanimage,createatextureandcopytheimage intothetexture.Becauseweareinabrowserimagesloadasynchronously sowehavetore-arrangeourcodealittletowaitforthetexturetoload. Onceitloadswe'lldrawit. +functionmain(){ +varimage=newImage(); +image.src="https://someimage/on/our/server";//MUSTBESAMEDOMAIN!!! +image.onload=function(){ +render(image); +} +} functionrender(image){ ... //lookupwherethevertexdataneedstogo. varpositionAttributeLocation=gl.getAttribLocation(program,"a_position"); +vartexCoordAttributeLocation=gl.getAttribLocation(program,"a_texCoord"); //lookupuniforms varresolutionLocation=gl.getUniformLocation(program,"u_resolution"); +varimageLocation=gl.getUniformLocation(program,"u_image"); ... +//providetexturecoordinatesfortherectangle. +vartexCoordBuffer=gl.createBuffer(); +gl.bindBuffer(gl.ARRAY_BUFFER,texCoordBuffer); +gl.bufferData(gl.ARRAY_BUFFER,newFloat32Array([ +0.0,0.0, +1.0,0.0, +0.0,1.0, +0.0,1.0, +1.0,0.0, +1.0,1.0]),gl.STATIC_DRAW); +gl.enableVertexAttribArray(texCoordAttributeLocation); +varsize=2;//2componentsperiteration +vartype=gl.FLOAT;//thedatais32bitfloats +varnormalize=false;//don'tnormalizethedata +varstride=0;//0=moveforwardsize*sizeof(type)eachiterationtogetthenextposition +varoffset=0;//startatthebeginningofthebuffer +gl.vertexAttribPointer( +texCoordAttributeLocation,size,type,normalize,stride,offset) + +//Createatexture. +vartexture=gl.createTexture(); + +//makeunit0theactivetextureunit +//(i.e,theunitallothertexturecommandswillaffect.) +gl.activeTexture(gl.TEXTURE0+0); + +//Bindtextureto'textureunit'0'2Dbindpoint +gl.bindTexture(gl.TEXTURE_2D,texture); + +//Settheparameterssowedon'tneedmipsandsowe'renotfiltering +//andwedon'trepeat +gl.texParameteri(gl.TEXTURE_2D,gl.TEXTURE_WRAP_S,gl.CLAMP_TO_EDGE); +gl.texParameteri(gl.TEXTURE_2D,gl.TEXTURE_WRAP_T,gl.CLAMP_TO_EDGE); +gl.texParameteri(gl.TEXTURE_2D,gl.TEXTURE_MIN_FILTER,gl.NEAREST); +gl.texParameteri(gl.TEXTURE_2D,gl.TEXTURE_MAG_FILTER,gl.NEAREST); + +//Uploadtheimageintothetexture. +varmipLevel=0;//thelargestmip +varinternalFormat=gl.RGBA;//formatwewantinthetexture +varsrcFormat=gl.RGBA;//formatofdatawearesupplying +varsrcType=gl.UNSIGNED_BYTE//typeofdatawearesupplying +gl.texImage2D(gl.TEXTURE_2D, +mipLevel, +internalFormat, +srcFormat, +srcType, +image); ... //Tellittouseourprogram(pairofshaders) gl.useProgram(program); //Passinthecanvasresolutionsowecanconvertfrom //pixelstoclipspaceintheshader gl.uniform2f(resolutionLocation,gl.canvas.width,gl.canvas.height); +//Telltheshadertogetthetexturefromtextureunit0 +gl.uniform1i(imageLocation,0); +//Bindthepositionbuffersogl.bufferDatathatwillbecalled +//insetRectangleputsdatainthepositionbuffer +gl.bindBuffer(gl.ARRAY_BUFFER,positionBuffer); + +//Setarectanglethesamesizeastheimage. +setRectangle(gl,0,0,image.width,image.height); } Andhere'stheimagerenderedinWebGL. clickheretoopeninaseparatewindow Nottooexcitingsolet'smanipulatethatimage.Howaboutjust swappingredandblue? ... outColor=texture(u_image,v_texCoord).bgra; ... Andnowredandblueareswapped. clickheretoopeninaseparatewindow Whatifwewanttodoimageprocessingthatactuallylooksatother pixels?SinceWebGLreferencestexturesintexturecoordinateswhich gofrom0.0to1.0thenwecancalculatehowmuchtomovefor1pixel withthesimplemathonePixel=1.0/textureSize. Here'safragmentshaderthataveragestheleftandrightpixelsof eachpixelinthetexture. #version300es //fragmentshadersdon'thaveadefaultprecisionsoweneed //topickone.highpisagooddefault.Itmeans"highprecision" precisionhighpfloat; //ourtexture uniformsampler2Du_image; //thetexCoordspassedinfromthevertexshader. invec2v_texCoord; //weneedtodeclareanoutputforthefragmentshader outvec4outColor; voidmain(){ +vec2onePixel=vec2(1)/vec2(textureSize(u_image,0)); + +//averagetheleft,middle,andrightpixels. +outColor=( +texture(u_image,v_texCoord)+ +texture(u_image,v_texCoord+vec2(onePixel.x,0.0))+ +texture(u_image,v_texCoord+vec2(-onePixel.x,0.0)))/3.0; } Comparetotheun-blurredimageabove. clickheretoopeninaseparatewindow Nowthatweknowhowtoreferenceotherpixelslet'suseaconvolutionkernel todoabunchofcommonimageprocessing.Inthiscasewe'llusea3x3kernel. Aconvolutionkernelisjusta3x3matrixwhereeachentryinthematrixrepresents howmuchtomultiplythe8pixelsaroundthepixelwearerendering.Wethen dividetheresultbytheweightofthekernel(thesumofallvaluesinthekernel) or1.0,whicheverisgreater.Here'saprettygoodarticleonit. Andhere'sanotherarticleshowingsomeactualcodeif youweretowritethisbyhandinC++. Inourcasewe'regoingtodothatworkintheshadersohere'sthenewfragmentshader. #version300es //fragmentshadersdon'thaveadefaultprecisionsoweneed //topickone.highpisagooddefault.Itmeans"highprecision" precisionhighpfloat; //ourtexture uniformsampler2Du_image; //theconvolutionkerneldata uniformfloatu_kernel[9]; uniformfloatu_kernelWeight; //thetexCoordspassedinfromthevertexshader. invec2v_texCoord; //weneedtodeclareanoutputforthefragmentshader outvec4outColor; voidmain(){ vec2onePixel=vec2(1)/vec2(textureSize(u_image,0)); vec4colorSum= texture(u_image,v_texCoord+onePixel*vec2(-1,-1))*u_kernel[0]+ texture(u_image,v_texCoord+onePixel*vec2(0,-1))*u_kernel[1]+ texture(u_image,v_texCoord+onePixel*vec2(1,-1))*u_kernel[2]+ texture(u_image,v_texCoord+onePixel*vec2(-1,0))*u_kernel[3]+ texture(u_image,v_texCoord+onePixel*vec2(0,0))*u_kernel[4]+ texture(u_image,v_texCoord+onePixel*vec2(1,0))*u_kernel[5]+ texture(u_image,v_texCoord+onePixel*vec2(-1,1))*u_kernel[6]+ texture(u_image,v_texCoord+onePixel*vec2(0,1))*u_kernel[7]+ texture(u_image,v_texCoord+onePixel*vec2(1,1))*u_kernel[8]; outColor=vec4((colorSum/u_kernelWeight).rgb,1); } InJavaScriptweneedtosupplyaconvolutionkernelanditsweight functioncomputeKernelWeight(kernel){ varweight=kernel.reduce(function(prev,curr){ returnprev+curr; }); returnweight<=0?1:weight; } ... varkernelLocation=gl.getUniformLocation(program,"u_kernel[0]"); varkernelWeightLocation=gl.getUniformLocation(program,"u_kernelWeight"); ... varedgeDetectKernel=[ -1,-1,-1, -1,8,-1, -1,-1,-1 ]; //setthekernelandit'sweight gl.uniform1fv(kernelLocation,edgeDetectKernel); gl.uniform1f(kernelWeightLocation,computeKernelWeight(edgeDetectKernel)); ... Andvoila...Usethedropdownlisttoselectdifferentkernels. clickheretoopeninaseparatewindow IhopethisarticlehasconvincedyouimageprocessinginWebGLisprettysimple.Nextup I'llgooverhowtoapplymorethanoneeffecttotheimage. Whataretextureunits? Whenyoucallgl.draw???yourshadercanreferencetextures.Texturesarebound totextureunits.Whiletheuser'smachinemightsupportmoreallWebGL2implementationsare requiredtosupportatleast16textureunits.Whichtextureuniteachsampleruniform referencesissetbylookingupthelocationofthatsampleruniformandthensettingthe indexofthetextureunityouwantittoreference. Forexample: vartextureUnitIndex=6;//usetextureunit6. varu_imageLoc=gl.getUniformLocation( program,"u_image"); gl.uniform1i(u_imageLoc,textureUnitIndex); Tosettexturesondifferentunitsyoucallgl.activeTextureandthenbindthetextureyouwantonthatunit.Example //BindsomeTexturetotextureunit6. gl.activeTexture(gl.TEXTURE6); gl.bindTexture(gl.TEXTURE_2D,someTexture); Thisworkstoo vartextureUnitIndex=6;//usetextureunit6. //BindsomeTexturetotextureunit6. gl.activeTexture(gl.TEXTURE0+textureUnitIndex); gl.bindTexture(gl.TEXTURE_2D,someTexture); What'swiththea_,u_,andv_prefixesinfrontofvariablesinGLSL? That'sjustanamingconvention.Theyarenotrequiredbutformeitmakesiteasiertoseeataglance wherethevaluesarecomingfrom.a_forattributeswhichisthedataprovidedbybuffers.u_foruniforms whichareinputstotheshaders,v_forvaryingswhicharevaluespassedfromavertexshadertoa fragmentshaderandinterpolated(orvaried)betweentheverticesforeachpixeldrawn. SeeHowitworksformoredetails. English Deutsch 日本語 한국어 PortuguêsBrasileiro 简体中文 Fundamentals HowtouseWebGL2 Fundamentals HowItWorks ShadersandGLSL WebGL2StateDiagram WebGL2vsWebGL1 What'snewinWebGL2 MovingfromWebGL1toWebGL2 DifferencesfromWebGLFundamentals.orgtoWebGL2Fundamentals.org ImageProcessing ImageProcessing ImageProcessingContinued 2Dtranslation,rotation,scale,matrixmath 2DTranslation 2DRotation 2DScale 2DMatrices 3D Orthographic3D 3DPerspective 3D-Cameras 3D-MatrixNaming Lighting DirectionalLighting PointLighting SpotLighting StructureandOrganization LessCode,MoreFun DrawingMultipleThings SceneGraphs Geometry 3DGeometry-Lathe Loading.objfiles Loading.objw.mtlfiles Textures Textures DataTextures Using2orMoreTextures CrossOriginImages PerspectiveCorrectTextureMapping PlanarandPerspectiveProjectionMapping RenderingToATexture RendertoTexture Shadows Shadows Techniques 2D 2D-DrawImage 2D-MatrixStack Sprites 3D Cubemaps Environmentmaps Skyboxes Skinning Fog Picking(clickingonstuff) Text Text-HTML Text-Canvas2D Text-UsingaTexture Text-UsingaGlyphTexture GPGPU GPGPU Tips SmallestPrograms DrawingWithoutData Shadertoy PullingVertices Optimization IndexedVertices(gl.drawElements) InstancedDrawing Misc SetupAndInstallation Boilerplate ResizingtheCanvas Animation Points,Lines,andTriangles MultipleViews,MultipleCanvases VisualizingtheCamera WebGL2andAlpha 2Dvs3Dlibraries Anti-Patterns WebGL2MatricesvsMathMatrices PrecisionIssues Takingascreenshot PreventtheCanvasBeingCleared GetKeyboardInputFromaCanvas UseWebGL2asBackgroundinHTML CrossPlatformIssues QuestionsandAnswers Reference Attributes TextureUnits Framebuffers readPixels References HelperAPIDocs TWGL,AtinyWebGLhelperlibrary github Issue/Bug?Createanissueongithub. Use

codegoeshere