javascript - Canvas getImageData returning incorrect data on certain mobile devices -

i working on canvas video player special features based on frames of video. overcome unreliable timing in video html5 tag videos using have barcode embedded in each frame indicating current frame number. using canvas getimagedata method can grab pixels , read barcode frame number. works great , have jsfiddle demonstrating works (i couldn't around cors in fiddle serve video canvas see working you'll have download example video locally upload via button. not ideal works).

on mobile devices (only android far) logic breaks. getimagedata returns incorrect values.

it works correctly on samsung galaxy s5 v6.0.1 fails on google pixel running android v7.1.2. i'll try collect more data on devices/os versions fails on.

for example, when playing on desktop first iteration of getimagedata returns:

uint8clampedarray(64) [3, 2, 3, 255, 255, 255, 255, 255, 246, 245, 247, 255, 243, 242, 243, 255, 241, 239, 241, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255] 

which correctly gets computed framenumber 1.

however on galaxy, first iteration returns:

uint8clampedarray(64) [255, 242, 217, 255, 255, 234, 209, 255, 41, 1, 1, 255, 254, 235, 210, 255, 255, 234, 209, 255, 50, 4, 0, 255, 254, 240, 215, 255, 255, 248, 224, 255, 255, 249, 225, 255, 255, 251, 232, 255, 255, 252, 233, 255, 255, 252, 233, 255, 255, 253, 234, 255, 255, 255, 237, 255, 255, 255, 237, 255, 28, 1, 1, 255]  

i read devices may being doing additional smoothing i've been playing around disabling in context via:

this.ctx.mozimagesmoothingenabled = false; this.ctx.webkitimagesmoothingenabled = false; this.ctx.msimagesmoothingenabled = false; this.ctx.imagesmoothingenabled = false;  

but didn't help.

here code being used in jsfiddle.

var framenumberdiv = document.getelementbyid('framenumber'); function load() {     var canvas = document.getelementbyid('canvas');     var ctx = canvas.getcontext('2d');     var video = document.getelementbyid('video');     canvas.width = 568;     canvas.height = 640;     video.addeventlistener('play', function() {         var = this; //cache         (function loop() {             if (!that.paused && !that.ended) {                 ctx.drawimage(that, 0, 0);                 var pixels = ctx.getimagedata(0, 320 - 1, 16, 1).data;                 getframenumber(pixels);                 settimeout(loop, 1000 / 30); // drawing @ 30fps             }         })();     }, 0); }  function getframenumber(pixels) {      let j = 0;     let thisframenumber = 0;     let str = "pixels: ";     (let = 0; < 16; i++) {         str += pixels[j] + " ";         thisframenumber += getbinary(pixels[j], i);         j += 4;     }     document.getelementbyid('framenumber').innerhtml = "framenumber: " + thisframenumber; }  function getbinary(pixel, binaryplace) {     const binary = [1, 2, 4, 8, 16, 32, 64, 128, 256,         512, 1024, 2048, 4096, 8192, 16384, 32768     ];     if (pixel > 128) return 0;     if (pixel < 128 && binary[binaryplace]) {         return binary[binaryplace]     } else {         return 0;     } }  (function localfilevideoplayer() {     'use strict';     var url = window.url || window.webkiturl;     var displaymessage = function(message, iserror) {         var element = document.queryselector('#message');         element.innerhtml = message;         element.classname = iserror ? 'error' : 'info';     }     var playselectedfile = function(event) {             console.log("playing");         var file = this.files[0];         var type = file.type;         var videonode = document.queryselector('video');         var canplay = videonode.canplaytype(type);         if (canplay === '') canplay = 'no';         var message = 'can play type "' + type + '": ' + canplay;         var iserror = canplay === 'no';         displaymessage(message, iserror);          if (iserror) {             return;         }          var fileurl = url.createobjecturl(file);         videonode.src = fileurl;         load();     }     var inputnode = document.queryselector('input')     inputnode.addeventlistener('change', playselectedfile, false) })(); 

edit

• works on nexus 6p running android v6.0
• works on samsung 6 (samsung sm g920a) running android v 5.0.2
• it does not work on samsung galaxy s7 (samsung-sm-g935a) running android v7.0

could possibly android 7 issue?

edit 2

in response question in comments:

videonode.videoheight , videowidth both 0 on google pixel entire existence same on desktop. in both of devices don't work i've encountered image of each frame painted. i'll attach screen shot google pixel. when paused consistently reads same number. in other words not jumping around whatever reading on frame of video.

edit 3: discovery

i believe i've made relevant discovery/realization should have seen earlier.

when looking @ output of getimagedata on broken device stepping through line line. hadn't (and should have) noticed video element continuing after hit break points / debugger statements. time getimagedata method executed video had moved past next frame. so, scanned barcode later frame expected.

i added console log statements , let run naturally. looking @ output can see more recognizable pattern.

here first few readings on google pixel:

uint8clampedarray(64) [255, 255, 255, 255, 246, 246, 246, 255, 243, 243, 243, 255, 240, 240, 241, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 2, 2, 2, 255]  uint8clampedarray(64) [5, 5, 5, 255, 255, 255, 255, 255, 251, 251, 251, 255, 247, 247, 248, 255, 245, 245, 245, 255, 245, 245, 245, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 6, 3, 2, 255]  uint8clampedarray(64) [235, 231, 230, 255, 17, 12, 12, 255, 252, 247, 247, 255, 255, 255, 255, 255, 255, 254, 254, 255, 255, 253, 253, 255, 255, 252, 251, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 7, 3, 1, 255]  uint8clampedarray(64) [26, 15, 14, 255, 4, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 10, 0, 0, 255] 

as may notice, results seem correct shifted 1 pixel left.

i modified jsfiddle shift getimagedata read on pixel , gives exact same response on pixel.

var pixels = ctx.getimagedata(1, 320 - 1, 16, 1).data; 

doing -1 seems have no effect.

so, reason these devices either shifting entire texture on pixel or there wrong getimagedata method.

edit 4

as experiment reconfigured code use webgl texture. same behaviour on desktop/mobile devices. allowed me use -1 x target using gl.readpixels. hoping skipping using canvas entire image stored in memory , access pixel data needed.... didn't work here data produced shows shifted using purely webgl.

uint8array(64) [0, 0, 0, 0, 255, 248, 248, 255, 25, 18, 18, 255, 254, 254, 254, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255]  uint8array(64) [0, 0, 0, 0, 255, 254, 247, 255, 255, 244, 236, 255, 48, 18, 10, 255, 254, 246, 238, 255, 255, 247, 239, 255, 255, 247, 239, 255, 255, 248, 241, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 250, 240, 255, 255, 250, 240, 255]  uint8array(64) [0, 0, 0, 0, 31, 0, 0, 255, 254, 243, 230, 255, 43, 6, 1, 255, 254, 243, 230, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 229, 255, 255, 244, 229, 255] 

using:

gl.readpixels(-1, height, 16, 1, gl.rgba, gl.unsigned_byte, pixels); 

edit 5

ok, promised more details on devices failing/not. had online qa testing done using modified jsfiddle. modified make bit more idiot proof general public work with.

the responses unfortunately mixed. hoping isolated android 7 doesn't seem case.

i have csv on google drive results of test. not these tests 100% reliable seems it's random devices......

your code fine problem here resides on how androids render barcode. implementation of barcode way small (16x1 pixels) , it's left indented.

being left indented anti alias device makes on outer pixels mess barcode , give incorrect results so, when working video render, don't want work on 1 pixel safe area.

my suggestion redo barcodes bigger size - let's 18 pixels height - use black , white (no grays), center on video , rest of line paint green: (in example it's barcode "1")

then make getimagedata of full 320x16 , rid of has rgb of 0x255x0 (and approximate) , have barcode able use framenumber.

i know answer wouldn't necessary redo video but, in case, source problem.