Ghostlab 2 is here

We’re excited to announce that Ghostlab 2 is now available! Let us give you a quick tour of what’s new, and what’s the same.

As in version one, Ghostlab 2 focuses on helping you making responsive and multi-device testing much faster and more convenient. It lets you browse any site you’re interested in, keeping any number of connected clients in sync by propagating scrolling, clicking, and any other site interactions it discovers. This works for local browsers and mobile devices on your network. It lets you (validly!) fill out forms automatically within a fraction of a second, lets you keep track of all connected clients, and offers you the ability to identify problems and fix them on any of them.

At the same time, we’ve introduced many changes in version 2 – some of them significant, some of them more on the minor-but-useful side. We’d like to present three of them that we’re particularly happy to share with you.

ghostlab interface

Client inspection

In Ghostlab 1, we’ve included Weinre, an awesome project that allowed us to remotely inspect the DOM on any connected device. However, we wanted to do more than that, and decided we’d have to rethink how we provide remote inspection tools. In Ghostlab 2, we are including the developer tools you know from Google Chrome – with some tweaks of our own. You are still able to inspect any connected client – but in addition, you can now inspect all of them synchronously.

That means that when you remotely “fix” that CSS property that destroys your page in Chrome or Firefox, you’ll immediately also see what the consequences are for any other connected browser or device.

On top of that, we decided that inspecting CSS is often not enough – especially after some bad experiences finding JavaScript bugs on mobile phones. So we’ve come up with a solution that lets you remotely debug JavaScript on any client that is connected to Ghostlab – be it a browser, a tablet, a mobile phone or even a TV set. We’ve even been told it works on IE8!

ghostlab inspect debug Javascript

Preprocessors

To allow Ghostlab to further integrate into the workflow, we’ve also added the ability to compile several languages. Initially, we’re shipping support for Sass / Less / Stylus, Jade / Haml, and CoffeeScript / Typescript. When you’re using Ghostlab with your local site, it will now be able to compile all these source files into browser-ready HTML/CSS/JS and then automatically refresh the page on your devices.

We’re not including binary compiler packages in our distribution, rather, Ghostlab will attempt to install them them should they not be present on your system. We’re using npm and gem for that purpose. This makes it easy to offer a wide range of preprocessors in the future while not making the Ghostlab distribution too heavy. While the initial set of precompilers may seem limited, we’ll constantly be on the lookout for interesting candidates to include.

ghostlab compile scss compile less

Redirection service

We’ve been told by several customers that when working in a team, pointing mobile devices to the Ghostlab server can be cumbersome. Of course, in order to avoid having to enter the URL on every one of them each time (which is slow even using the provided QR Code), they had bookmarked the Ghostlab server URL on every device. but: which URL? Several developers means several computers means several IP addresses – so each developer has a different (and potentially changing) IP address and the device bookmark will ever so often point to the wrong URL.

The idea of the redirection service is simple. A redirection bookmark is just a URL, looking something like http://302.pm/ABC677. It redirects to an IP address in your local network, like http://192.168.2.3:8005. The redirection bookmark has an access token, and using that access token, Ghostlab is able to update the redirection bookmark to the current IP and port where Ghostlab runs. When on a device, you access the bookmark, you will always be redirected to the up-to-date Ghostlab address. This way, you can either share redirection bookmarks within a team (by sharing the access token), or simply create your individual and permanent Ghostlab URL for your devices to use.

Enough said. Give it a spin!

So much from us. You can have a look at the Ghostlab website, where you can download a full-featured and completely free 7-day trial!

IndexedDB fundamentals – Plus a indexeddb example – tutorial

IndexedDB is a relatively fresh NOSQL database implemented in modern browsers. In this blog post I’m going to show you how to perform some basic operations with it. The tutorial has been optimised for use with Google’s latest Chrome browser.

In Chrome’s developer tools, IndexedDB can be found in the Resources tab. For the moment, any changes to your IndexedDB databases aren’t automatically visible in the developer tools – you have to either refresh the page or right-click on “IndexedDB” in the Resources tab and choose “Refresh IndexedDB”.

After two short general remarks, we’ll dive right into the code!

Limits
IndexedDB has no hard storage limits on it’s own. However, browser vendors have soft limits. Firefox will ask for permission to store blobs bigger than 50 MB. Google Chrome has various limits for different use cases, for more information about Chrome limits see https://developers.google.com/chrome/whitepapers/storage

Fallback
For older browsers that don’t support IndexedDB, you can use IndexedDBShim. It allows WebSQL (which is no longer developed) to handle IndexedDB API calls when they are not present in the browser. Available at https://github.com/axemclion/IndexedDBShim.

Creating, dropping and retrieving databases

Firefox does not prefix IndexedDB, Chrome had prefixed it for a while but now no longer does it, and Internet Explorer uses an MS vendor prefix. To achieve compatibility across browsers, you might want to use the declaration solution below.

[javascript]
var indexedDB = window.indexedDB || window.webkitIndexedDB || window.msIndexedDB;
var IDBKeyRange = window.IDBKeyRange || window.webkitIDBKeyRange;
var openCopy = indexedDB && indexedDB.open;

var IDBTransaction = window.IDBTransaction || window.webkitIDBTransaction;

if (IDBTransaction)
{
IDBTransaction.READ_WRITE = IDBTransaction.READ_WRITE || ‘readwrite’;
IDBTransaction.READ_ONLY = IDBTransaction.READ_ONLY || ‘readonly’;
}
[/javascript]

Remember IndexedDB is a NoSQL database, so it doesn’t contain tables, but object stores. Object stores contain data in the form <key, value> – where the value can be a simple string or a complex object. To create your first database, simply attempt to open it. Since it doesn’t exist yet, the upgradeneeded event will be triggered – which you handle in order to create your object store(s). Notice that whenever you want to modify your databases object store, you need to do this by increasing the version number of your database – the version upgrade will again trigger the upgradeneeded event, which you can handle to make the necessary modifications. To open a database indicating that you want to create a new version, simply add the version number as the second parameters to your indexedDb.open() call. Dropping a database is straightforward.

[javascript]
/***
* Create database snippet
* */
var request = indexedDB.open(‘todos’);

request.onupgradeneeded = function(e)
{
// e is an instance of IDBVersionChangeEvent
var idb = e.target.result;

if (idb.objectStoreNames.contains(‘todo’))
{
idb.deleteObjectStore(‘todo’);
}

var store = idb.createObjectStore(‘todo’, {keyPath: ‘text’, autoIncrement: true});
// createIndex operations possible to be pefromed on store.createIndex
store.createIndex(‘by_todo’, ‘todo’, {unique: true, multiEntry: false});
};

request.onsuccess = function(e) { /* add, update, delete, … */ };
request.onerror = function(e) { /* handle error */ };

/***
* Remove database snippet
* */
var dropDatabase = function(name)
{
var request = indexedDB.deleteDatabase(name);
request.onsuccess = function() { /* drop succeeded */ };
request.onerror = function() { /* drop failed */ };
};
[/javascript]

If you want to programatically find out which databases are available, you are currently limited to webkit (other browsers don’t offer such a method). Note that IndexedDB follows a same-origin policy – so you won’t be able to read databases from domains other than your own.

[javascript]
indexedDB.webkitGetDatabaseNames().onsuccess = function(e)
{
var databaseNames = [];
for (var i = 0, l = e.target.result.length; i < l; i++)
databaseNames.push(e.target.result[i]);
};
[/javascript]

Creating records

Adding a record to object store requires creating a “readwrite” transaction, as demonstrated below. If your browser supports IndexedDB, you can populate a database with a todo object store and some example entries by clicking the button below the code sample. Make sure you have your dev tools open so that you can see what is going on on the database.

[javascript]
var request = indexedDB.open(‘todos’);

request.onsuccess = function(e)
{
var idb = e.target.result;
var trans = idb.transaction(‘todo’, IDBTransaction.READ_WRITE);
var store = trans.objectStore(‘todo’);

// add
var requestAdd = store.add({text: ‘Go to Coop’, todo: ‘Groceries’});

requestAdd.onsuccess = function(e) {
// do something
};

requestAdd.onfailure = function(e) {
// failed
};
};
[/javascript]

Retrieving records

You can retrieve records in a read-only transaction. For iterating over all entries in an object store, simply use the openCursor function. If you have previously chosen to add the six predefined todo entries to the demo store, you can retrieve them using the button below the code.

[javascript]
var request = indexedDB.open(‘todos’);
request.onsuccess = function(e)
{
idb = e.target.result;
var transaction = idb.transaction(‘todo’, IDBTransaction.READ_ONLY);
var objectStore = transaction.objectStore(‘todo’);

objectStore.openCursor().onsuccess = function(event)
{
var cursor = event.target.result;
if (cursor)
{
console.log(‘Cursor data’, cursor.value);
cursor.continue();
}
else
{
console.log(‘Entries all displayed.’);
}
};
};
[/javascript]

When opening a cursor to the object store, you can specify various parameters for limiting the scope of the cursor. The following examples demonstrate several of these constraints. Again, if you have chosen to use the build-in demo via the buttons, you can continue to do so for every code example.

The interface we are using to apply constraints to our cursor is IDBKeyRange. When specifying bounds, you can indicate if you want them to be open (excluding the endpoint values, pass true) or closed (including the endpoint values, pass false – this is the default). For in-depth information refer to the documentation on the Mozilla Developer Network.

[javascript]

// Retrieve all records after and including ‘Groceries’ key
var cursor = IDBKeyRange.lowerBound(‘Groceries’, false);
objectStore.openCursor(cursor).onsuccess = function(event)

[/javascript]

[javascript]

// Retrieve all records up to and including ‘Groceries’ key
var cursor = IDBKeyRange.upperBound(‘Groceries’, false);
objectStore.openCursor(cursor).onsuccess = function(event)

[/javascript]

[javascript]

// Retrieve all records between ‘Bills’ and ‘Flat’ keys (inclusive)
var cursor = IDBKeyRange.bound(‘Bills’, ‘Flat’, false, false);
objectStore.openCursor(cursor).onsuccess = function(event)

[/javascript]

[javascript]

// Retrieve only the record matching the ‘FeedTheDog’ key
var cursor = IDBKeyRange.only(‘FeedTheDog’);
objectStore.openCursor(cursor).onsuccess = function(event)

[/javascript]

 

Editing data

To edit or delete records on a database, you will need to create a read-write transaction. The following examples show you how you can edit a record (basically, retrieve it and update it via the object store’s put method) and delete a record (calling delete on the object store and passing the key). If you have populated the demo database using the buttons before, you can now edit and delete these records using the buttons below the code.

[javascript]
// Editing a record
var editRecord = function(key, newValue) {
var request = indexedDB.open(‘todos’);
request.onsuccess = function(e)
{
var idb = e.target.result;
var objectStore = idb.transaction(‘todo’, IDBTransaction.READ_WRITE).objectStore(‘todo’);
var request = objectStore.get(key);

request.onsuccess = function(ev)
{
var data = ev.target.result;
var editDivEl = document.querySelector(‘#editRecordDiv’);

if (data === undefined)
{
editDivEl.innerHTML = ‘Key doesnt exist or has been previously’ +
‘removed’;
return;
}

data.text = newValue;
var result = objectStore.put(data);

result.onsuccess = function(ev)
{
var todoName = ev.target.result;
editDivEl.innerHTML = ‘Successfully edited key <b>’ +
todoName + ‘</b>’;
};

result.onerror = function(ev)
{
console.log(‘Error occured’, ev.srcElement.error.message);
};
};

request.onerror = function(ev)
{
console.log(‘Error occured’, ev.srcElement.error.message);
};
};
};

var reminderDate = new Date();
editRecord(‘Bills’, ‘Reminder ‘ + reminderDate.getHours() + ‘:’ + reminderDate.getMinutes() +
‘:’ + reminderDate.getSeconds());
[/javascript]

[javascript]
// Deleting a record
var request = indexedDB.open(‘todos’);
request.onsuccess = function(e)
{
var idb = e.target.result;
var objectStore = idb.transaction(‘todo’, IDBTransaction.READ_WRITE).objectStore(‘todo’);
var request = objectStore.delete(‘Bills’);

request.onsuccess = function(ev)
{
console.log(ev);
};

request.onerror = function(ev)
{
console.log(‘Error occured’, ev.srcElement.error.message);
};
};
[/javascript]

Limiting queries

As a final example in this tutorial, I want to show you a way of limiting the number of results returned by a query. IndexedDB doesn’t provide a straightforward method for limiting and grouping database results. Therefore, I have created a simple helper function that helps us limiting the number of records retrieved by a query. Note that in such a context, the IDBCursor.advance method might come in handy, as it lets you skip retrieved records. In the example below, we abort the transaction once the indicated limit has been reached.

[javascript]
/***
* IndexedDB limit output syntax similar to LIMIT 10, 5;
* */
var limitRecords = function(pageSize, skipCount)
{
var request = indexedDB.open(‘todos’);

request.onsuccess = function(e)
{
idb = e.target.result;
var transaction = idb.transaction(‘todo’, IDBTransaction.READ_ONLY);
var objectStore = transaction.objectStore(‘todo’);
var idx = 0;

objectStore.openCursor().onsuccess = function(e)
{
var cursor = e.target.result;
if (cursor)
{
if (skipCount <= idx && idx < pageSize + skipCount)
console.log(‘Cursor is in the range ‘, cursor);

idx++;

if (idx >= pageSize + skipCount + 1)
{
// we have all data we requested
// abort the transaction
transaction.abort();
} else {
// continue iteration
cursor.continue();
}
}

};
};
};
[/javascript]

Done.

We hope you’ve enjoyed this little tutorial! Of course, there is much more to IndexedDB than what we have covered here – this was just to get you started!








How to Proxy Requests in node.js

Proxy Request in node.js

Recently we received quite a few requests from people wanting to use Ghostlab for their website and webapp testing from behind corporate proxies. Unfortunately, as Ghostlab’s server component is built on node.js and the HTTP client in node.js assumes that a direct connection to the Web can be made (i.e., node.js ignores the system’s proxy settings), we had to build our own support to proxy requests in node.

There are some other blog posts on how to do a request through a proxy for HTTP, so adding that was easy. But not for HTTPS, so that’s what this little write-up is mainly about.

HTTP

Let’s start with the easy part. Assume you want to request the content of https://www.vanamco.com/ghostlab. Requesting it through a proxy, we would do something like:

  • Connect to the proxy server (e.g. with telnet 192.168.5.8 3128 if 192.168.5.8 is the IP of your proxy server running on port 3128),
  • do the GET request using the full URL as the path:

[code]
GET https://www.vanamco.com/ghostlab/ HTTP/1.1
[/code]

Or with node.js:

[javascript]
var Http = require(‘http’);

var req = Http.request({
host: ‘192.168.5.8’,
// proxy IP
port: 3128,
// proxy port
method: ‘GET’,
path: ‘https://www.vanamco.com/ghostlab/’ // full URL as path
}, function (res) {
res.on(‘data’, function (data) {
console.log(data.toString());
});
});

req.end();
[/javascript]

HTTPS

While the above code is the correct way to do a HTTP GET request through a proxy, it won’t work if you try to use HTTPS. For instance, if you replace the path by https://www.twitter.com, your proxy server might say something like
“Unsupported Request Method and Protocol. Squid does not support all request methods for all access protocols.”

In fact, you actually wouldn’t want the proxy to be able to do this; you wouldn’t want it to decrypt and send you back your unencrypted data. Instead, proxies support the CONNECT request, which will establish a tunnel to the remote server, so the data will remain encrypted.

When you open a telnet session (telnet 192.168.5.8 3128) and do a CONNECT request

[code]
CONNECT twitter.com:443
[/code]

the proxy will answer with

[code]
HTTP/1.0 200 Connection established
[/code]

This means, that the proxy set up the tunnel to the host. Now you can communicate with the host through the tunnel. I.e., you’ll have to start by sending your encrypted request (which is a bit hard to do with telnet, so the next code listing shows how to do it with node.js).

[javascript]
var Http = require(‘http’);
var Tls = require(‘tls’);

var req = Http.request({
host: ‘192.168.5.8’,
port: 3128,
method: ‘CONNECT’,
path: ‘twitter.com:443’,
});

req.on(‘connect’, function (res, socket, head) {
var cts = Tls.connect({
host: ‘twitter.com’,
socket: socket
}, function () {
cts.write(‘GET / HTTP/1.1rnHost: twitter.comrnrn’);
});

cts.on(‘data’, function (data) {
console.log(data.toString());
});
});

req.end();
[/javascript]

The node.js program does a HTTP CONNECT request to the proxy and listens for the connect event. The event handler is passed an instance of net.Socket which we can use as if we were directly communicating with the remote server. I.e., we need to start by sending the encrypted request and we’ll receive back the encrypted content.

To do the encryption and decryption, we use node.js’s TLS module. The tls.connect method accepts an options argument which contains the socket we received from the connect event handler. Make sure to also include the host in the options, even though the node.js docs say the host will be ignored when you’re passing a socket. If you don’t pass the host, an error will be thrown, “Hostname/IP doesn’t match certificate’s altnames”. As a second argument, tls.connect accepts a callback function (a listener to the secureConnect event, really), in which we do our HTTP request (manually, in this snippet) writing to the tls.ClearTextStream object returned by tls.connect.

A HTTPS Proxy Agent

To make life a little easier and also benefit from node.js’s HTTPS client implementation, we can wrap the CONNECT request and dealing with TLS into a https.Agent so we only have to do this when doing an HTTPS request via a proxy:

[javascript]
var Https = require(‘https’);
var agent = new HttpsProxyAgent({
proxyHost: ‘192.168.5.8’,
proxyPort: 3128
});
Https.request({
// like you’d do it usually…
host: ‘twitter.com’,
port: 443,
method: ‘GET’,
path: ‘/’,

// … just add the special agent:
agent: agent
}, function (res) {
res.on(‘data’, function (data) {
console.log(data.toString());
});
}).end();
[/javascript]

In summary: in addition to a regular HTTPS request, you only need to instantiate an HttpsProxyAgent and pass the instance in the agent property of the request options.


 

To implement the HttpsProxyAgent, we can reuse most of node.js’s implementation of the http.Agent and https.Agent. But we need to overwrite two methods, addRequest and createSocket to handle the asynchronous nature of how we receive the socket (the original implementation assumes that createConnection returns a socket), and so we can emit an error event on the request object when the connection to the proxy fails.

Here we go (Gist):
Update (March 23, 2015): The code below was updated to work with node v0.12.0. See Gist for more information.

[javascript]
var Util = require(‘util’);
var Https = require(‘https’);
var Tls = require(‘tls’);

function HttpsProxyAgent(options) {
Https.Agent.call(this, options);

this.proxyHost = options.proxyHost;
this.proxyPort = options.proxyPort;

this.createConnection = function (opts, callback) {
// do a CONNECT request
var req = Http.request({
host: options.proxyHost,
port: options.proxyPort,
method: ‘CONNECT’,
path: opts.host + ‘:’ + opts.port,
headers: {
host: opts.host
}
});

req.on(‘connect’, function (res, socket, head) {
var cts = Tls.connect({
host: opts.host,
socket: socket
}, function () {
callback(false, cts);
});
});

req.on(‘error’, function (err) {
callback(err, null);
});

req.end();
}
}

Util.inherits(HttpsProxyAgent, Https.Agent);

// Almost verbatim copy of http.Agent.addRequest
HttpsProxyAgent.prototype.addRequest = function (req, options) {
var name = options.host + ‘:’ + options.port;
if (options.path) name += ‘:’ + options.path;

if (!this.sockets[name]) this.sockets[name] = [];

if (this.sockets[name].length < this.maxSockets) {
// if we are under maxSockets create a new one.
this.createSocket(name, options.host, options.port, options.path, req, function (socket) {
req.onSocket(socket);
});
} else {
// we are over limit so we’ll add it to the queue.
if (!this.requests[name])
this.requests[name] = [];
this.requests[name].push(req);
}
};

// Almost verbatim copy of http.Agent.createSocket
HttpsProxyAgent.prototype.createSocket = function (name, host, port, localAddress, req, callback) {
var self = this;
var options = Util._extend({}, self.options);
options.port = port;
options.host = host;
options.localAddress = localAddress;

options.servername = host;
if (req) {
var hostHeader = req.getHeader(‘host’);
if (hostHeader)
options.servername = hostHeader.replace(/:.*$/, ”);
}

self.createConnection(options, function (err, s) {
if (err) {
err.message += ‘ while connecting to HTTP(S) proxy server ‘ + self.proxyHost + ‘:’ + self.proxyPort;

if (req)
req.emit(‘error’, err);
else
throw err;

return;
}

if (!self.sockets[name]) self.sockets[name] = [];

self.sockets[name].push(s);

var onFree = function () {
self.emit(‘free’, s, host, port, localAddress);
};

var onClose = function (err) {
// this is the only place where sockets get removed from the Agent.
// if you want to remove a socket from the pool, just close it.
// all socket errors end in a close event anyway.
self.removeSocket(s, name, host, port, localAddress);
};

var onRemove = function () {
// we need this function for cases like HTTP ‘upgrade’
// (defined by WebSockets) where we need to remove a socket from the pool
// because it’ll be locked up indefinitely
self.removeSocket(s, name, host, port, localAddress);
s.removeListener(‘close’, onClose);
s.removeListener(‘free’, onFree);
s.removeListener(‘agentRemove’, onRemove);
};

s.on(‘free’, onFree);
s.on(‘close’, onClose);
s.on(‘agentRemove’, onRemove);

callback(s);
});
};
[/javascript]

Image credit: hiroze