WebGL Image Processing

To draw images in WebGL we need to use textures. Similarly to the way WebGL expects clip space coordinates when rendering instead of pixels, WebGL expects ... English Français 日本語 한국어 Polski Portuguese Русский 简体中文 TableofContents WebGLFundamentals.org Fix,Fork,Contribute WebGLImageProcessing ImageprocessingiseasyinWebGL.Howeasy?Readbelow. ThisisacontinuationfromWebGLFundamentals. Ifyouhaven'treadthatI'dsuggestgoingtherefirst. TodrawimagesinWebGLweneedtousetextures.SimilarlytothewayWebGL expectsclipspacecoordinateswhenrenderinginsteadofpixels,WebGLexpects texturecoordinateswhenreadingatexture.Texturecoordinatesgofrom0.0to 1.0nomatterthedimensionsofthetexture. Sinceweareonlydrawingasinglerectangle(well,2triangles)weneedto tellWebGLwhichplaceinthetextureeachpointintherectanglecorresponds to.We'llpassthisinformationfromthevertexshadertothefragmentshader usingaspecialkindofvariablecalleda'varying'.It'scalledavarying becauseitvaries.WebGLwillinterpolatethevaluesweprovideinthevertex shaderasitdrawseachpixelusingthefragmentshader. Usingthevertexshaderfromtheendofthepreviouspost weneedtoaddanattributetopassintexturecoordinatesandthenpassthose ontothefragmentshader. attributevec2a_texCoord; ... varyingvec2v_texCoord; voidmain(){ ... //passthetexCoordtothefragmentshader //TheGPUwillinterpolatethisvaluebetweenpoints v_texCoord=a_texCoord; } Thenwesupplyafragmentshadertolookupcolorsfromthetexture. precisionmediumpfloat; //ourtexture uniformsampler2Du_image; //thetexCoordspassedinfromthevertexshader. varyingvec2v_texCoord; voidmain(){ //Lookupacolorfromthetexture. gl_FragColor=texture2D(u_image,v_texCoord); } Finallyweneedtoloadanimage,createatextureandcopytheimageinto thetexture.Becauseweareinabrowserimagesloadasynchronouslysowe havetore-arrangeourcodealittletowaitforthetexturetoload.Once itloadswe'lldrawit. functionmain(){ varimage=newImage(); image.src="http://someimage/on/our/server";//MUSTBESAMEDOMAIN!!! image.onload=function(){ render(image); } } functionrender(image){ ... //allthecodewehadbefore. ... //lookupwherethetexturecoordinatesneedtogo. vartexCoordLocation=gl.getAttribLocation(program,"a_texCoord"); //providetexturecoordinatesfortherectangle. vartexCoordBuffer=gl.createBuffer(); gl.bindBuffer(gl.ARRAY_BUFFER,texCoordBuffer); gl.bufferData(gl.ARRAY_BUFFER,newFloat32Array([ 0.0,0.0, 1.0,0.0, 0.0,1.0, 0.0,1.0, 1.0,0.0, 1.0,1.0]),gl.STATIC_DRAW); gl.enableVertexAttribArray(texCoordLocation); gl.vertexAttribPointer(texCoordLocation,2,gl.FLOAT,false,0,0); //Createatexture. vartexture=gl.createTexture(); gl.bindTexture(gl.TEXTURE_2D,texture); //Settheparameterssowecanrenderanysizeimage. gl.texParameteri(gl.TEXTURE_2D,gl.TEXTURE_WRAP_S,gl.CLAMP_TO_EDGE); gl.texParameteri(gl.TEXTURE_2D,gl.TEXTURE_WRAP_T,gl.CLAMP_TO_EDGE); gl.texParameteri(gl.TEXTURE_2D,gl.TEXTURE_MIN_FILTER,gl.NEAREST); gl.texParameteri(gl.TEXTURE_2D,gl.TEXTURE_MAG_FILTER,gl.NEAREST); //Uploadtheimageintothetexture. gl.texImage2D(gl.TEXTURE_2D,0,gl.RGBA,gl.RGBA,gl.UNSIGNED_BYTE,image); ... } Andhere'stheimagerenderedinWebGL.NOTE:Ifyouarerunningthislocallyyou'll needasimplewebservertoallowWebGLtoloadtheimages.Seehereforhowtosetupone upincoupleofminutes. clickheretoopeninaseparatewindow Nottooexcitingsolet'smanipulatethatimage.Howaboutjustswappingredandblue? ... gl_FragColor=texture2D(u_image,v_texCoord).bgra; ... Andnowredandblueareswapped. clickheretoopeninaseparatewindow Whatifwewanttodoimageprocessingthatactuallylooksatotherpixels? SinceWebGLreferencestexturesintexturecoordinateswhichgofrom0.0to1.0 thenwecancalculatehowmuchtomovefor1pixelwiththesimplemath onePixel=1.0/textureSize. Here'safragmentshaderthataveragestheleftandrightpixelsofeachpixel inthetexture. precisionmediumpfloat; //ourtexture uniformsampler2Du_image; uniformvec2u_textureSize; //thetexCoordspassedinfromthevertexshader. varyingvec2v_texCoord; voidmain(){ //compute1pixelintexturecoordinates. vec2onePixel=vec2(1.0,1.0)/u_textureSize; //averagetheleft,middle,andrightpixels. gl_FragColor=( texture2D(u_image,v_texCoord)+ texture2D(u_image,v_texCoord+vec2(onePixel.x,0.0))+ texture2D(u_image,v_texCoord+vec2(-onePixel.x,0.0)))/3.0; } WethenneedtopassinthesizeofthetexturefromJavaScript. ... vartextureSizeLocation=gl.getUniformLocation(program,"u_textureSize"); ... //setthesizeoftheimage gl.uniform2f(textureSizeLocation,image.width,image.height); ... Comparetotheun-blurredimageabove. clickheretoopeninaseparatewindow Nowthatweknowhowtoreferenceotherpixelslet'suseaconvolutionkernel todoabunchofcommonimageprocessing.Inthiscasewe'llusea3x3kernel. Aconvolutionkernelisjusta3x3matrixwhereeachentryinthematrixrepresents howmuchtomultiplythe8pixelsaroundthepixelwearerendering.Wethendivide theresultbytheweightofthekernel(thesumofallvaluesinthekernel)or1.0, whicheverisgreater.Here'saprettygoodarticleonit. Andhere'sanotherarticleshowingsomeactualcodeifyouweretowritethisby handinC++. Inourcasewe'regoingtodothatworkintheshadersohere'sthenewfragment shader. precisionmediumpfloat; //ourtexture uniformsampler2Du_image; uniformvec2u_textureSize; uniformfloatu_kernel[9]; uniformfloatu_kernelWeight; //thetexCoordspassedinfromthevertexshader. varyingvec2v_texCoord; voidmain(){ vec2onePixel=vec2(1.0,1.0)/u_textureSize; vec4colorSum= texture2D(u_image,v_texCoord+onePixel*vec2(-1,-1))*u_kernel[0]+ texture2D(u_image,v_texCoord+onePixel*vec2(0,-1))*u_kernel[1]+ texture2D(u_image,v_texCoord+onePixel*vec2(1,-1))*u_kernel[2]+ texture2D(u_image,v_texCoord+onePixel*vec2(-1,0))*u_kernel[3]+ texture2D(u_image,v_texCoord+onePixel*vec2(0,0))*u_kernel[4]+ texture2D(u_image,v_texCoord+onePixel*vec2(1,0))*u_kernel[5]+ texture2D(u_image,v_texCoord+onePixel*vec2(-1,1))*u_kernel[6]+ texture2D(u_image,v_texCoord+onePixel*vec2(0,1))*u_kernel[7]+ texture2D(u_image,v_texCoord+onePixel*vec2(1,1))*u_kernel[8]; //Dividethesumbytheweightbutjustusergb //we'llsetalphato1.0 gl_FragColor=vec4((colorSum/u_kernelWeight).rgb,1.0); } InJavaScriptweneedtosupplyaconvolutionkernelanditsweight functioncomputeKernelWeight(kernel){ varweight=kernel.reduce(function(prev,curr){ returnprev+curr; }); returnweight<=0?1:weight; } ... varkernelLocation=gl.getUniformLocation(program,"u_kernel[0]"); varkernelWeightLocation=gl.getUniformLocation(program,"u_kernelWeight"); ... varedgeDetectKernel=[ -1,-1,-1, -1,8,-1, -1,-1,-1 ]; gl.uniform1fv(kernelLocation,edgeDetectKernel); gl.uniform1f(kernelWeightLocation,computeKernelWeight(edgeDetectKernel)); ... Andvoila...Usethedropdownlisttoselectdifferentkernels. clickheretoopeninaseparatewindow IhopethisarticlehasconvincedyouimageprocessinginWebGLisprettysimple. NextupI'llgooverhowtoapplymorethanoneeffecttotheimage. u_imageisneverset.Howdoesthatwork? Uniformsdefaultto0sou_imagedefaultstousingtextureunit0. Textureunit0isalsothedefaultactivetexturesocallingbindTexture willbindthetexturetotextureunit0. WebGLhasanarrayoftextureunits.Whichtextureuniteachsampleruniform referencesissetbylookingupthelocationofthatsampleruniformandthen settingtheindexofthetextureunityouwantittoreference. Forexample: vartextureUnitIndex=6;//usetextureunit6. varu_imageLoc=gl.getUniformLocation( program,"u_image"); gl.uniform1i(u_imageLoc,textureUnitIndex); Tosettexturesondifferentunitsyoucallgl.activeTextureandthen bindthetextureyouwantonthatunit.Example //BindsomeTexturetotextureunit6. gl.activeTexture(gl.TEXTURE6); gl.bindTexture(gl.TEXTURE_2D,someTexture); Thisworkstoo vartextureUnitIndex=6;//usetextureunit6. //BindsomeTexturetotextureunit6. gl.activeTexture(gl.TEXTURE0+textureUnitIndex); gl.bindTexture(gl.TEXTURE_2D,someTexture); AllWebGLimplementationsarerequiredtosupportatleast8textureunits infragmentshadersbutonly0invertexshaders.Soifyouwanttouse morethan8youshouldcheckhowmanytherearebycalling gl.getParameter(gl.MAX_TEXTURE_IMAGE_UNITS)orifyouwanttouse texturesinavertexshadercallgl.getParameter(gl.MAX_VERTEX_TEXTURE_IMAGE_UNITS) tofindouthowmanyyoucanuse.Over99%ofmachinessupportatleast4 textureunitsinvertexshaders. What'swiththea_,u_,andv_prefixesinfromofvariablesinGLSL? That'sjustanamingconvention.Theyarenotrequiredbutformeitmakesiteasiertoseeataglance wherethevaluesarecomingfrom.a_forattributeswhichisthedataprovidedbybuffers.u_foruniformswhichareinputstotheshaders,v_forvaryingswhicharevaluespassedfromavertexshadertoafragmentshaderandinterpolated(orvaried)betweentheverticesforeachpixeldrawn. SeeHowitworksformoredetails. English Français 日本語 한국어 Polski Portuguese Русский 简体中文 Fundamentals Fundamentals HowItWorks ShadersandGLSL WebGLStateDiagram ImageProcessing ImageProcessing ImageProcessingContinued 2Dtranslation,rotation,scale,matrixmath 2DTranslation 2DRotation 2DScale 2DMatrices 3D Orthographic3D 3DPerspective 3DCameras Lighting DirectionalLighting PointLighting SpotLighting StructureandOrganization LessCode,MoreFun DrawingMultipleThings SceneGraphs Geometry Geometry-Lathe Loading.objfiles Loading.objw.mtlfiles Textures Textures DataTextures Using2orMoreTextures CrossOriginImages PerspectiveCorrectTextureMapping PlanarandPerspectiveProjectionMapping RenderingToATexture RendertoTexture Shadows Shadows Techniques 2D 2D-DrawImage 2D-MatrixStack Sprites 3D Cubemaps Environmentmaps Skyboxes Skinning Fog Picking(clickingonstuff) Text Text-HTML Text-Canvas2D Text-UsingaTexture Text-UsingaGlyphTexture Textures RampTextures(ToonShading) GPGPU GPGPU Tips SmallestPrograms DrawingWithoutData Shadertoy PullingVertices Optimization IndexedVertices(gl.drawElements) InstancedDrawing Misc SetupAndInstallation Boilerplate ResizingtheCanvas Animation Points,Lines,andTriangles MultipleViews,MultipleCanvases VisualizingtheCamera WebGLandAlpha 2Dvs3Dlibraries Anti-Patterns WebGLMatricesvsMathMatrices PrecisionIssues Takingascreenshot PreventtheCanvasBeingCleared GetKeyboardInputFromaCanvas UseWebGLasBackgroundinHTML CrossPlatformIssues QuestionsandAnswers Reference Attributes TextureUnits Framebuffers readPixels References HelperAPIDocs TWGL,AtinyWebGLhelperlibrary github Questions?Askonstackoverflow. Issue/Bug?Createanissueongithub. Use

codegoeshere