13 9 / 2012

Creating .webm video from getUserMedia()

There’s a ton of motivation for being able to record live video. One scenario: you’re capturing video from the webcam. You add some post-production touchups in your favorite online video editing suite. You upload the final product to YouTube and share it out to friends. Stardom proceeds.

MediaStreamRecorder is a WebRTC API for recording getUserMedia() streams (example code). It allows web apps to create a file from a live audio/video session.

MediaStreamRecorder is currently unimplemented in the Chrome. However, all is not lost thanks to Whammy.js. Whammy is a library that encodes .webm video from a list of .webp images, each represented as dataURLs.

As a proof of concept, I’ve created a demo that captures live video from the webcam and creates a .webm file from it.

LAUNCH DEMO

The demo also uses a[download] to let users download their file.

Creating webp images from <canvas>

The first step is to feed getUserMedia() data into a <video> element:

var video = document.querySelector('video');
video.autoplay = true; // Make sure we're not frozen!

// Note: not using vendor prefixes!
navigator.getUserMedia({video: true}, function(stream) {
  video.src = window.URL.createObjectURL(stream);
}, function(e) {
  console.error(e);
});

Next, draw an individual video frame into a <canvas>:

var canvas = document.querySelector('canvas');
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);

Chrome supports canvas.toDataURL("image/webp"). This allows us to read back the <canvas> as a .webp image and encode is as a dataURL, all in one swoop:

var url = canvas.toDataURL('image/webp', 1); // Second param is quality.

Since this only gives us an single frame, we need to repeat the draw/read pattern using a requestAnimationFrame() loop. That’ll give us webp frames at 60fps:

var rafId;
var frames = [];
var CANVAS_WIDTH = canvas.width;
var CANVAS_HEIGHT = canvas.height;

function drawVideoFrame(time) {
  rafId = requestAnimationFrame(drawVideoFrame);
  ctx.drawImage(video, 0, 0, CANVAS_WIDTH, CANVAS_HEIGHT);
  frames.push(canvas.toDataURL('image/webp', 1));
};

rafId = requestAnimationFrame(drawVideoFrame); // Note: not using vendor prefixes!

\m/

The last step is to bring in Whammy. The library includes a static method fromImageArray() that creates a Blob (file) from an array of dataURLs. Perfect! That’s just what we have.

Let’s package all of this goodness up in a stop() method:

function stop() {
  cancelAnimationFrame(rafId);  // Note: not using vendor prefixes!

  // 2nd param: framerate for the video file.
  var webmBlob = Whammy.fromImageArray(frames, 1000 / 60);

  var video = document.createElement('video');
  video.src = window.URL.createObjectURL(webmBlob);

  document.body.appendChild(video);
}

When stop() is called, the requestAnimationFrame() recursion is terminated and the .webm file is created.

Performance and Web Workers

Encoding webp images using canvas.toDataURL('image/webp') takes ~120ms on my MBP. When you do something crazy like this in requestAnimationFrame() callback, the framerate of the live getUserMedia() video stream noticeably drops. It’s too much for the UI thread to handle.

Having the browser encode webp in C++ is far faster than encoding the .webp image in JS.

My tests using libwebpjs in a Web Worker were horrendously slow. The idea was to each frame as a Uint8ClampedArray (raw pixel arrays), save them in an array, and postMessage() that data to the Worker. The worker was responsible for encoding each pixel array into webp. The whole process took up to 20+ seconds to encode a single second’s worth of video. Not worth it.

It’s too bad CanvasRenderingContext2D doesn’t exist in the Web Worker context. That would solved a lot of the perf issues.

08 9 / 2012

Mashups using CORS and responseType=’document’

I always forget that you can request a resource as a Document using XHR2. Combine this with CORS and things get pretty nice. No need to parse HTML strings and turn them into DOM yourself.

For html5rocks.com, we support CORS on all of our content. It’s trivial to pull down the tutorials page and query the DOM directly using querySelector()/querySelectorAll() on the XHR’s response.

Demo: http://jsbin.com/ibepag/1

https://gist.github.com/3581825

Tags:

Permalink 3 notes

06 8 / 2012

Five things you didn’t know the web could do

It’s been awhile since my last blog post. That’s because I’ve been busy writing articles all over the web! Check out my post on netmagazine, “Five things you didn’t know the web could do”.

Tags:

Permalink 2 notes

23 5 / 2012

Data Binding Using data-* Attributes

Custom data-* attributes in HTML5 are pretty rad. They’re especially handy for stashing small amounts of data and retaining minimal state on the DOM. Turns out, they can also be used for one-way data binding!

I’ve been using a nifty trick in recent projects that I thought would be worth sharing. The technique is to use a data attribute to store values (i.e. the data model) and :before/:after pseudo elements to render the values as generated content (i.e. the view). I call it "poor man’s data binding" because it’s not true data binding in the traditional sense, but the semantics are similar. Count it!

Here we go:

<style>
  input {
    vertical-align: middle;
    margin: 2em;
    font-size: 14px;
    height: 20px;
  }
  input::after {
    content: attr(data-value) '/' attr(max);
    position: relative;
    left: 135px;
    top: -20px;
  }
</style>

<input type="range" min="0" max="100" value="25">

<script>
  var input = document.querySelector('input');

  input.dataset.value = input.value; // Set an initial value.

  input.addEventListener('change', function(e) {
    this.dataset.value = this.value;
  });
</script>

image TRY IT

Notice the 25/100 updates as you move the slider, but the <input> is the only markup on the page.

The magic line is the content: attr(data-value) '/' attr(max). It uses CSS attr() to pull out the data-value and max attributes; both set using markup on the <input>. As those values change, the generated content is automatically updated. Sick data binding bro.

Really the only benefit of this technique is that we’re not including extraneous markup. For comparison, here’s the same gig, but using an extra element:

<input type="range" min="0" max="100" value="25"><span></span>

<script>
  var input = document.querySelector('input');
  input.addEventListener('change', function(e) {
    document.querySelector('span').textContent = this.value + '/' + this.max;
  });
</script>

Less elegant, but it works.

Last but not least, here’s a more complex example that uses CSS transitions to change the height of a div container when clicked. As the height changes, requestAnimationFrame() updates the data-height of the div and the pseudo element picks that up.

image TRY IT

I’m sure if HTML was conceived in the age of web apps, we’d have proper DOM/JS data binding by now. Fortunately, initiatives like MDV and Web Components are on their way. One day this stuff will be a reality and native to HTML!

Data binding is a technique for automatically synchronizing data between two sources. On the web, data binding typically manifests itself as updating DOM (UI) in response to events: XHRs, user input, or other business logic doing its thing. Take the canonical todo list for example. When I mark an item as done, the completed count increments. When it’s unchecked, the count decrements. That’s data binding!

If you want true two-way data binding, checkout one of the popular MVC frameworks like Angular, Knockout, or Ember.

Tags:

Permalink 17 notes

23 4 / 2012

idb.filesystem.js - Bringing the HTML5 Filesystem API to More Browsers

The HTML5 Filesystem API is a versatile API that addresses many of the uses cases that the other offline APIs don’t. It can remedy their shortcomings, like making it difficult to dynamically caching a page. I’m looking at you AppCache!

My ♥ for the API is deep—so much so that I wrote a book and released a library called filer.js to help promote its adoption. While filer aims to make the API more consumable, it fails to address the elephant in the room: browser support.

Introducing idb.filesystem.js

Today, I’m happy to bring the HTML5 Filesystem API to more browsers by releasing idb.filesystem.js.

idb.filesystem.js is a well tested JavaScript polyfill implementation of the Filesystem API intended for browsers that lack native support. Right now that’s everyone but Chrome. The library works by using IndexedDB as an underlying storage layer. This means any browser supporting IndexedDB, now supports the Filesystem API! All you need to do is make Filesystem API calls and the rest is magic.

Demos

I’ve thrown together two demo apps to demonstrate the library’s usage. The first is a basic example. It allows you to create empty files/folders, drag files into the app from the desktop, and navigate into a folder or preview a file by clicking its name:

imageTry the demo in Firefox 11+

Want to use filer.js’s API with idb.filesystem.js? No problem. 90% of filer.js works out of the box with idb.filesystem.js. In fact, the second demo is a slightly modified version of filer.js’s playground app, showing that the two libraries can work in harmony. \m/

What’s exciting is that both of these apps work in FF, Chrome, and presumably other browsers that implement storing binary data in IndexedDB.

I look forward to your feedback and pull requests!

27 12 / 2011

Introducing filer.js

Some 1300+ lines of code, 106 tests, and a year after I first started it, I’m happy to officially unleash filer.js (https://github.com/ebidel/filer.js); a wrapper library for the HTML5 Filesystem API.

Unlike other libraries [1, 2], filer.js takes a different approach and incorporates some lessons I learned while implementing the Google Docs Python client library. Namely, the library reuses familiar UNIX commands (cp, mv, rm) for its API. My goal was to a.) make the HTML5 API more approachable for developers that have done file I/O in other languages, and b.) make repetitive operations (renaming, moving, duplicating) easier.

So, say you wanted to list the files in a given folder. There’s an ls() for that:

var filer = new Filer();
filer.init({size: 1024 * 1024}, onInit.bind(filer), onError);

function onInit(fs) {
  filer.ls('/', function(entries) {
    // entries is an Array of file/directories in the root folder.
  }, onError);
}

function onError(e) { ... }

A majority of filer.js calls are asynchronous. That’s because the underlying HTML5 API is also asynchronous. However, the library is extremely versatile and tries to be your friend whenever possible. In most cases, callbacks are optional. filer.js is also good at accepting multiple types when working with entries. It accepts entries as string paths, filesystem: URLs, or as the FileEntry/DirectoryEntry object.

For example, ls() is happy to take your filesystem: URL or your DirectoryEntry:

// These will produce the same results.
filer.ls(filer.fs.root.toURL(), function(entries) { ... });
filer.ls(filer.fs.root, function(entries) { ... });
filer.ls('/', function(entries) { ... });

The library clocks in at 24kb (5.6kb compressed). I’ve thrown together a complete sample app to demonstrate most of filer.js's functionality:

imageTry the DEMO

Lastly, there’s room for improvement:

  1. Incorporate Chrome’s Quota Management API
  2. Make usage in Web Workers more friendly (there is a synchronous API).

I look forward to your feedback and pull requests!

28 11 / 2011

Web Audio API how-to: Playing audio based on user interaction

One thing the Web Audio API does particularly well is play sound. Of course, this is something you’d expect from an audio API :). That said, the API is complex and it’s not immediately obvious on the best way to do something simple like load a sound file and play it based on a button click. That task alone can involve a number of new platform features likes XHR2, FileReader API, and ArrayBuffers.

So…I threw together a quick example on how to load a audio file and play/stop it based on the user clicking a button:

<!DOCTYPE html>
<!-- Author: Eric Bidelman (ericbidelman@chromium.org) -->
<html>
<head>
  <meta charset="utf-8" />
  <meta http-equiv="X-UA-Compatible" content="chrome=1" />
  <title>Web Audio API: Simple load + play</title>
</head>
<body>
  <p>Example of using the Web Audio API to load a sound file
  and start playing on user-click.</p>
  <input type="file" accept="audio/*">
  <button onclick="playSound()" disabled>Start</button>
  <button onclick="stopSound()" disabled>Stop</button>
<script>
var context = new window.webkitAudioContext();
var source = null;
var audioBuffer = null;

function stopSound() {
  if (source) {
    source.noteOff(0);
  }
}

function playSound() {
  // source is global so we can call .noteOff() later.
  source = context.createBufferSource();
  source.buffer = audioBuffer;
  source.loop = false;
  source.connect(context.destination);
  source.noteOn(0); // Play immediately.
}

function initSound(arrayBuffer) {
  context.decodeAudioData(arrayBuffer, function(buffer) {
    // audioBuffer is global to reuse the decoded audio later.
    audioBuffer = buffer;
    var buttons = document.querySelectorAll('button');
    buttons[0].disabled = false;
    buttons[1].disabled = false;
  }, function(e) {
    console.log('Error decoding file', e);
  }); 
}

// User selects file, read it as an ArrayBuffer and pass to the API.
var fileInput = document.querySelector('input[type="file"]');
fileInput.addEventListener('change', function(e) {  
  var reader = new FileReader();
  reader.onload = function(e) {
    initSound(this.result);
  };
  reader.readAsArrayBuffer(this.files[0]);
}, false);

// Load file from a URL as an ArrayBuffer.
// Example: loading via xhr2: loadSoundFile('sounds/test.mp3');
function loadSoundFile(url) {
  var xhr = new XMLHttpRequest();
  xhr.open('GET', url, true);
  xhr.responseType = 'arraybuffer';
  xhr.onload = function(e) {
    initSound(this.response); // this.response is an ArrayBuffer.
  };
  xhr.send();
}
</script>
</body>
</html>

Demo

As you can see, this example includes two different ways to get an audio file into the Web Audio API: via an <input type="file"> and an XMLHttpRequest. Both methods call initSound(), which is passed an ArrayBuffer containing the audio file. That method then decodes the audio and stores the result in a global variable (so it can be reused as play/stop are pressed).

That’s it! Straightforward once you see it, right!?

Tags:

Permalink 2 notes

01 8 / 2011

Reading .mp3 ID3 tags in JavaScript

For a recent project, I needed to read an .mp3’s ID3 metadata (song title, artist, year, album) in pure JS. The idea was to have the user select a song file, and boom!, its info would display to the user. Good news…totally easy with the FileReader API and JS typed arrays.

Initially, I did a quick search to find some examples, but all of the examples I found used older techniques like manipulating a binary string. Ugh! We have better tools now for working with binary data in JS. Typed arrays to the rescue, specifically DataView.

DataView is a cousin to ArrayBufferView, which is a “view” of a portion of an ArrayBuffer. Array buffers represent chunks of bytes in memory. Multiple views can be created from a single ArrayBuffer. For example, one could create a Int8Array and a Float32Array from the same underlying data. Hence, “views”. This property makes them extremely versatile for binary data.

For my purposes, DataView turned out to be a perfect container for pulling sections of bytes as a string. However, I found the API to be a bit unintuitive in practice. Fortunately, I stumbled upon Christopher Chedeau's jDataView a while back and things started making sense. jDataView is an excellent wrapper for DataView, improving much of its jankiness and adding a few extra utility methods for things like seeking and getting at data (e.g. getString(), getChar()).

Here’s all the code that I needed:

document.querySelector('input[type="file"]').onchange = function(e) {
  var reader = new FileReader();

  reader.onload = function(e) {
    var dv = new jDataView(this.result);

    // "TAG" starts at byte -128 from EOF.
    // See http://en.wikipedia.org/wiki/ID3
    if (dv.getString(3, dv.byteLength - 128) == 'TAG') {
      var title = dv.getString(30, dv.tell());
      var artist = dv.getString(30, dv.tell());
      var album = dv.getString(30, dv.tell());
      var year = dv.getString(4, dv.tell());
    } else {
      // no ID3v1 data found.
    }
  };

  reader.readAsArrayBuffer(this.files[0]);
};

Pretty slick.

DataView is implemented in Chrome 9+ and Webkit nightlies. However, jDataView provides the DataView API for all the browsers using the best available option between Strings, JS typed arrays, and DataView.

28 7 / 2011

My book is finally out: Using the HTML5 Filesystem API, &#8220;A True Filesystem for the Browser&#8221; 
Until now, web applications have been unable to organize binary data into a hierarchy of folders. That has changed with the advent of HTML5. With this book, you&#8217;ll learn how to provide your applications with a true file system that enables them to create, read, and write files ands folders in a sandboxed section of the user&#8217;s local filesystem. Author Eric Bidelman provides several techniques and complete code examples for working with the HTML5 Filesystem API.

My book is finally out: Using the HTML5 Filesystem API, “A True Filesystem for the Browser”

Until now, web applications have been unable to organize binary data into a hierarchy of folders. That has changed with the advent of HTML5. With this book, you’ll learn how to provide your applications with a true file system that enables them to create, read, and write files ands folders in a sandboxed section of the user’s local filesystem. Author Eric Bidelman provides several techniques and complete code examples for working with the HTML5 Filesystem API.