Streaming Systems history, part 3

Finally, here we are on the third part of the streaming system history. I'm sorry it took so long, but I had many, many things to do for work and I was very busy with real life stuff.
In this part I'm going to introduce systems from 2011. If you find any mistake or anything incomplete feel free to let me know by commenting the post! Not all the 2011 systems are shown in this post, I will complete it with the final, fourth part.

In this part I'm going to quickly talk about Storm (2011) and WebRTC (2011).

Storm is a distributed computational environment, free and open source (check out the link!) originally developed by Twitter. The building blocks, where information can be manipulated and created are called "spouts" (blocks that produce data) and "bolts" (blocks that receive streamed data). With them, developers can build a pipeline; they are indeed very similar to MapReduce jobs, with the only difference that they can (theoretically) run forever. Every spout and bolt can be run by one or more worker, where the core of the Storm computation is executed.

Given spouts and bolts as basic building blocks for a pipeline, developers can build Direct Acyclic Graph pipelines. The programming language is Java, thus the programming representation is imperative. Storm pipelines can be deployed on the cloud: that is they can run on local machine and have spouts or bolts somewhere on the cloud. This implementation detail became very important as time passed by. The pipeline is dynamic at node level, that is a Storm spout or bolt can alter the number of workers at runtime. Faults are dealt by reconfiguring the pipeline at runtime thanks to some controller threads that constantly run checking for errors. I call these kind of controllers "dynamic adaptive", like the ones seen in DryadLINQ and S4.

Web Real-Time Communication (WebRTC) is an API currently drafted by the World Wide Web Consortium (W3C). It enables browser-to-browser connection for applications like video, chat or voIP. Many examples of this API have been implemented since its initial release, for example peer-to-peer file sharing or video conferencing applications. This API is not a real streaming system per se, but brings a whole new connectivity for what concerns browser applications, opening path for future browser streaming applications.

Given the fact that WebRTC is only an API, we can build any kind of pipeline (arbitrary pipelines). It can only be programmed in JavaScript, thus likewise Storm, its programming representation is imperative while the deployment, for now, is only the web browser. The pipeline flexibility is dynamic at topology level, as nodes (browsers) can connect and disconnect from the pipeline at runtime. Of course there is no fault tolerance or load balancing, thus once the pipeline is disrupted, it can't be reconstructed again. Ideally, we would need a software on top of it to do so.

This was a small part, in respect to the previous two. As I mentioned, I'm very busy lately thus I only found time to write this. The next part will hopefully be the last one (this one was supposed to be, but it turned out to be very long).
See ya!

DYHV7EDQ28UB

Read More

Install a specific version of Node.JS

Quick post (personal reminder above all) about how to install a specific version of Node.js. I'm working on Ubuntu 12.04. The idea is to clone the whole repository and then checkout only the interested version. Then install it. Here are the commands.

git clone https://github.com/joyent/node.git
cd node
git checkout "v0.8.18"
export JOBS=2
./configure
make
make install

And you are good to go!

Read More

I'm still alive - !false and !undefined in JavaScript

Hi all, I've been very busy lately, and then I went for a holiday, so I didn't update the blog. Sorry!
I recently came back from the seaside and started to work on a bug I had since forever. I thought it was a deeply rooted bug that was spread in different methods. It turns out it was not like that. And that's what I discovered:

First of all, I had this array that kept count of how many distributed processes answered a pull request. Each process has an ID, and it corresponded with the index of the array. The array first is filled with false values. As an example, imagine process ID 0 that is polled. At first the value in the array at index 0 is false, but when the process answers, I set the value in the array at index 0 to true.

If the process don't reply after a certain threshold time, I execute something. I used to check this by going through the array in this way:

for(var i = 0; i < processes.length; i++)
    if(!array[i])
        //do something

In other words, if array[i] is false, it means the process did not answer.
This could look correct if only I would take into account the fact that, concurrently, some processes may be spawned, thus increasing the processes array. Since I didn't polled the newborn processes, I don't want them to be checked. Of course, with the shown code, this was happening. Luckily, JavaScript fills with undefined the indexes of an array which have not be initialised; but on the other hand the evaluation of !undefined is the same as the evaluation of !false. This clearly lead to a bug which always executed something, even if it was not the case. Again, luckily with JavaScript I could correct this very easily:

for(var i = 0; i < processes.length; i++)
    if(array[i] === false)
        //do something

And that's it!

Read More

Read a file line by line in node.js

Hi all. Today I decided to add a small functionality to my project: when I start my framework, I run stuff one after the other with a list of commands; I decided to add a functionality that reads a particular file (which I want to be a .k file) and extract the commands from that list.
To do so I need some things:

1. Read a file
2. Make sure it's a .k file
3. Split it line by line
4. Execute each line

Reading a file in node.js is very simple. We just need to import the fs module and call the function readFile:

fs.readFile(cmd[1], 'utf8', function(err, data) {
    if (err) throw err;
    //do something with the file
});

This will read the file specified in cmd[1] (which is the input variable I give). Next, I would like to add some more checks (for example, that it needs to be a .k file):

if(cmd[1].indexOf(".k") === cmd[1].length - 2){

    fs.readFile(cmd[1], 'utf8', function(err, data) {
 if (err) throw err;
 //do something with the file    
    });
}
else{
    console.log("Not a .k file!");
}

Finally, let's read the file line by line:

if(cmd[1].indexOf(".k") === cmd[1].length - 2){

    fs.readFile(cmd[1], 'utf8', function(err, data) {
 if (err) throw err;
 
 var commands = data.split('\n');
 for(var i = 0; i < commands.length; i++){
     //commands[i] contains lines of the input
 }    
    });
}
else{
    console.log("Not a .k file!");
}

Notice the .split('\n'); function, that splits the string into an array at every '\n' occurrence. Hope it helps!

Read More

Closures...

I'm sorry for the long lack of updates. Recently I've been very busy. I travelled a lot and I had a lot of work to do, so not really much time to work on my project and discover new interesting things.
Today I want to post something that bothered me for a while. I was calling an asynchronous function, m_cli.get(), inside a for loop and wanted to keep the index variable as it was going to be used in the callback.
The first approach was the following, and of course was not working:

for(var i = 0; i < list.length; i++){
    mc_cli.get(list[i], function(err, response) {
        do_something(i);
    });
}

With this approach, the callback would always execute using the last value of i. So, for example, in the list was long 10, it would always call do_something(9). To fix the problem I tried with a closure:

do_something((function(x){return x})(i)) 

The idea is to "keep" somehow the variable so that it could call the function only when the callback returns and keeps the right index. Unfortunately, also this approach wont work. Also by creating the closure outside the for-loop and calling it in the callback would not lead to a satisfying result.
Later on, I managed somehow to fix the problem like this:

for(var i = 0; i < list.length; i++){
    (function(i){
        mc_cli.get( parsed_result.list[i], function(err, response) {
            do_something(i);
        });
    }(i));
}

The idea is that the parameter now is the input parameter of an anonymous function which will then call the asynchronous function and when the callback fires, call the do_something(i) which the local input parameter, which is the correct index (and not the last element of the array). Basically it will treat the index as an input parameter, thus "remembering" it through the execution of the asynchronous function.
Hope I helped somebody!

Read More

Passing parameters to setInterval & setTimeout

Sometimes we need to play with timed events. JavaScript comes to the rescue giving access to setTimeout and setInterval. The first one fires only once after a certain amount of time passes, while the second one fires every time a given interval passes. For example if we give 1000 as one of the input variables to setInterval, it will fire every second.

One problem that may arise using these functions is when we want to call a function defined somewhere else and pass a parameter to it. If we try to do the following:

var myFun = function(param) {
    //do something
}

var myParam = 'parameter';

setTimeout(myFun(myParam), 1000);

Instead of executing the function after 1000ms, it will execute it immediately. This is because the compiler first interprets the function calls, like myFun(myParam), and then passes the result as the callback to the setTimeout function. To avoid this, the usual way is to pass function as a variable, thus omitting the parentheses. This of course leads to one problem: how to pass parameters to the callback function? It is done in the following way:

var myFun = function(param) {
    //do something
}

var myParam = 'parameter';

setTimeout(myFun, 1000, myParam, ...);

The three dots there indicates that each following input parameter will be treated as an input parameter to the callback function. The same concept applies for the setTimeout function.

Read More

Dynamically resizing Google Chrome Extensions

Lately I've been working on an extension for Google Chrome to read the last entry of the famous xkcd comic. At a certain point I stopped because I didn't understand how to dynamically resize the popup window of the Chrome extension. This required quite some research to understand what was going wrong, here is how I created the extension, which problem I had and how I solved it.

How I did it

The extension is very easy, it executes a GET request to the xkcd website, gets the content of the index page and returns it. What I used to do, for a matter of simplicity, was getting the content, paste it in an hidden div, get the image of the comic of the day (through a getElementById call) and put it in a visible div.

The Problem

What one would expect is that the Google Chrome popup page of the extension notice the change of content and resize itself to fit the new content. But this doesn't happen. The popup page remains as small as it can get without styles, or of a fixed with/height if specified in the css file. I didn't wanted to have a fixed width/height as comics usually have not a fixed size (sometimes they are as small as a strip, sometimes they are quite big).
Since the popup extension doesn't resize, I though about resizing it using JavaScript:

var width =  document.getElementById("comic").offsetWidth + 10;
var height = document.getElementById("comic").offsetHeight + 10;

//[...]

document.body.style.width = ""+width;
document.body.style.height = ""+height;
document.getElementsByTagName("html")[0].style.width = ""+width;
document.getElementsByTagName("html")[0].style.height = ""+height;

(notice that I get the width and the height of the image and add a small offset to make it visually finer). The fact is that this approach wasn't working either. When opening the extension, one would see it flicker: first a frame of the correct size, and then a resize to the default 10 by 10 pixel frame.

The Solution

The solution was quite simple, but without somebody mentioning it on StackOverflow I would be still stuck. The fact is that the popup of the Chrome Extensions don't resize if there is a hidden element in the HTML tree. That is, basically the whole page I was hiding was preventing me to resize the popup frame. So the solution is:

document.getElementById("hiddenDiv").innerHTML = "";

And that solved every problem. The frame now is the correct size and I can read every two/three days the new xkcd comic.

Read More

Removing an event listener from an element

Today I was working while I was asked how to remove an event listener from an element which had an
  click event. At first I didn't really remember how to do it, so I Googled a bit and I found this way:

document.getElementById("your_element_id").addEventListener('click', function(){
    //do something
    this.removeEventListener('click',arguments.callee,false); 
}, false);

As you can see, all the work is done by addEventListener and removeEventListener functions. The first one takes as parameter the event (click in this case) and a function that instruct the event listener on what it has to do with that event. The third argument specifies whether the EventListener being removed was registered as a capturing listener or not (set false by default).

The function removeEventListener takes basically the same input parameters. In the element that has to remove, it specifies which element (click), which function (in this case I use arguments.callee as callee is a property of the arguments object and it can be used to refer to the currently executing function), and the third argument that specifies if it was registered as a capturing listener or not.

Of course there are simpler way to do this. It turned out that who asked me how to remove an event listener from an element was using jQuery. In that case the code looks even simpler:

$('#your_element_id').unbind();
//or
$('#foo').unbind('click');
//or
$('#foo').unbind('click', function() {
   //do something
});
//or
$('#foo').unbind('click', function() {
   //do something
}, false);

As the previous examples, also in this case there are three parameters. jQuery is smart enough to take all the three possibilities as fine. The first one will remove every listener binded. The second one every function binded to that listener, the third one will remove that particular function binded to that particular event, while the fourth one just specifies the third argument that by default is false.
And that's it. Pretty easy.

Read More

setInterval/setTimeout and this

I think everybody is familiar with the setInterval and the setTimeout functions. They create a timed event which will occur after a given timeout. setInterval creates an interval: after the given time it will repeat the same action over and over, while the setTimeout function will only execute the action once. Let's see a small example on how they work:

var timed_function = function(){
    console.log("hello!");
}

var timeout = setInterval(timed_function, 1000);

The shown example will execute every second (1000 milliseconds) the function timed_function.

Now, I recently worked with a timed event and I had to access a variable within this. The naive approach was the following:

//previously set variable
this.message = "hello!";

var timed_function = function(){
    console.log(this.message);
}

var timeout = setInterval(timed_function, 1000);

But this printed undefined. I then discovered that when setInterval or setTimeout are executed, the scope of this apparently refers to the window object, thus it's completely loosing the scope of the object in which the function is executed. The idea to solve this problem is to wrap the call inside another object which has to be called to initialise the timeout:

//previously set variable
this.message = "hello!";

var wrapper = function(that) {
    var timed_function = function(){
        console.log(that.message);
    }

    var timeout = setInterval(timed_function, 1000);
}

//later
wrapper(this);

In this way I basically instantiate the timeout passing through the wrapper function which will keep a reference to this thanks to the input variable of the function. The input variable then will be called inside the timed_function. Hope this helped somebody. I had a couple of issues trying to solve this because I was confused by the scope of the variables. Some folks helped me on StackOverflow luckily, even though some of them where short-sighted and arrogant as always.

Read More

Measuring performance of a processor with Node.js

Some time ago I needed to measure the performance of the cores I was using on a machine while a Node.js server was running. I came up with different ideas, including writing a bash script that would measure it the hardcore way. I then realised I could use the os module that comes with Node and use the data it returns to measure it myself.

First of all, take a look at the specification of the method os.cpus() here. It shows how to call it and what is the return value. To measure the performance of the system, it is sufficient to take consecutive snapshots, compute a delta, sum all the data and compute the usage percentage like so:

//include the module
var os = require('os');

//data collected on the previous tick
var previous_cpu_usage;

//[...] collect previous_cpu_usage

//get the cpu data
var cpu_usage = os.cpus();

//iterate through it
for(var index = 0, len = cpu_usage.length; index < len; index++) {
    var cpu = cpu_usage[index], total = 0;
    var prev_cpu = previous_cpu_usage[index];
           
 //for each type compute delta
 for(type in cpu.times){
     total += cpu.times[type] - prev_cpu.times[type];
 }
           
 //print it for now
 for(type in cpu.times){
     if(type === "user" || type === "idle")
         console.log("\t", type, Math.round(100 * (cpu.times[type] - prev_cpu.times[type])/ total));
}

The idea is very simple. Here I'm not showing how to collect data consecutively. This can be done by the means of two timeouts: one that collect one data, then the second collect the second one and then in the callback of the second timeout data gets aggregated. I'm not showing the code exactly as it looks in my solution as it involves message passing through a (big) topology.

In the example shown, I'm printing the data that corresponds to user and idle time, since it's the one most people may be interested in. Anyways, with this code any percentage out of the returned data can be computed.

Read More

Bitwise operations

Some time ago I was talking with a second-year student after a C exam. He was pretty satisfied about the job done, but asked me a question about the ~ operator. At first I didn't quite remember, so I told him that was probably the bitwise not. Later I discovered I was right, so I took the opportunity to refresh my knowledge of the bitwise operations in JavaScript. The purpose of this article is to show which bitwise operations JavaScript offers.

First of all, we can only apply bitwise operations on numeric operands that have integer values. These integer are represented in the operation as a 32-bit integer representation instead of the equivalent floating-point representation. Some of these operations perform Boolean algebra on the single bits, while the others are used to shift bits.

Bitwise NOT ( ~ )

This operator simply flips all the bits of the given integer. In numeric terms, it changes the sign of the number and subtract 1. For example ~0x0000000f will become 0xfffffff0 after the evaluation.

Bitwise AND ( & )

Performs a bitwise Boolean AND operation and set in the result the bit only if it is set in both operands. For example 0xfff666fff & 0x000fff000 will evaluate to 0x000666000.

Bitwise OR ( | )

Likewise the bitwise AND, the bitwise OR behaves in the same way, but a bit is set in the result only if it is set in one or both of the operands.

Bitwise XOR ( ^ )

The bitwise exclusive OR behaves like a normal XOR. The bit in the result is set only if the first operand is true and the second is false, or viceversa. If both are true or both are false, the bit in the result is set to 0.

Shift Right ( >> )

This operator moves all the bits in the first operand to the right by the number of bits specified in the second operand (up to 31). The bits that are shifter off the right are lost, and the filling is given by the initial sign of the first operand (if the operand is positive, it will be zero-filled, otherwise f-filled).

Shift Right with zero Fill ( >>> )

Behaves exactly like the shift right operator, but the filling on the left is always 0, regardless of the sign of the first operand at the beginning of the operation.

Shift Left ( << )

This operator moves all the bits in the first operand to the left by the number of bits specified in the second operand (again, up to 31). The rightmost part of the number is filled with 0s.

These operations are not really used by JavaScript programmers, but it's important to know they exist.

Read More

instanceof & typeof Operators

Sometimes, even if it is considered bad habit, we need to check the class of an object in JavaScript. Classes may be defined by the programmer, but I will cover this topic later, for now let's just focus on classes defined by the interpreter. There are many, for example:

• Object
• String
• Number
• Date
• Function
• ...

To check if an object is an instance of a specific class, JavaScript offers the instanceof method, which may be used in the following way:

var mynmbr = new Number(1);
var mystrng = new String("disasterjs!");
var myfunc = function(){
    console.log("hello world!");
}
mynmbr instanceof Number //returns true
mynmbr instanceof Object //returns true
mystrng instanceof String //returns true
mystrng instanceof Object //returns true
mynmbr instanceof String //returns false
myfunc instanceof Function //returns true

Be careful when using instanceof about what you are checking. For example numbers and strings not wrapped in the constructor are treated as primitive value, thus a call for instanceof in that case would produce an unexpected result:

var mystrng = "disasterjs!";
var mynmbr = 1;

mynmbr instanceof Number //returns false
mynmbr instanceof Object //returns false
mystrng instanceof String //returns false
mystrng instanceof Object //returns false

What about the typeof operator? This operator returns the class as a string. The following table summarises its possible return values:

Type	Result
Undefined	"undefined"
Null	"object"
Boolean	"boolean"
Number	"number"
String	"string"
Host object (provided by the JS environment)	Implementation-dependent
Function	"function"
E4X XML object	"xml"
Any other object	"object"

Programmer may be careful here when checking a Null object as the result of a typeof will return "object". Here are a couple of examples on how to use typeof.

typeof 1 === "number" //true
typeof Number(1) === "number" //true
typeof 1 === "object" //false
typeof undefined_variable === "undefined" //true
typeof true === "boolean" //true

And that's it.

Read More

JavaScript, passing-by-reference and slice()

Some time ago I had a problem with arrays and assignments. I was working with a distributed system in JavaScript; the root process sent some "work" to do to some subprocesses (workers) and those returned an array as result, which was modified inside the callback of the dispatching function.

The issue was that when I modified the array, I then assigned it to a variable and passed it to a function. The problem was that while I was working in the callback with this array, it got modified somehow. I later understood that the problem was due to the fact that JavaScript passes values by reference. That is basically the array I assigned to the variable before passing it to the function was simply saving it's reference on the variable, but not really duplicating it.

The following example will show the problem.

var arr = [1, 2, 3]

var arr2 = arr;

arr2[0] = 0;

console.log(arr[0]);

//prints 0

As shown, this prints zero. To solve the issue I used a custom procedure to copy every single value of the array into a new one. This is pretty painful and later I discovered I could simply use the slice() function. The following snippet based on the previous example will show the black magic:

var arr = [1, 2, 3]

var arr2 = arr.slice();

arr2[0] = 0;

console.log(arr[0]);

//prints 1

Basically that's it. It's a very naive problem, and a skilled programmer may quickly find the issue, but knowing the existence of the slice() function may save quite some time.

Read More

Hello World!

console.log("I'm opening this blog to talk about JavaScript and things I come around with it.");

console.log("Updates are not going to be very frequent.");

console.log(":-)");

Read More