martes, 24 de marzo de 2015

console.timing() - measure multiple calls of a code block

The following is an utility, similar to console.time / console.timeEnd to measure how long a code block takes to execute but it will take in account sevaral times and report the total calling time and average. Very useful wile profiling code blocks execution time.


console.timing = function(key)
{
 this.timings = this.timings || {}; 
 this.timings[key] = this.timings[key] || []; 
 this.timings[key].push(performance.now()); 
};

console.timingEnd = function(key)
{
 var t0 = this.timings[key][this.timings[key].length-1]; 
 this.timings[key][this.timings[key].length-1] = performance.now() - t0;
}; 

console.timingReport = function(key)
{
 var times = this.timings[key]; 
 var sum = 0;
 for (var i = 0; i < times.length; i++) 
 {
  sum += times[i]; 
 }
 var average = sum / times.length;
 console.log(key + ': times: ' + times.length + ', sum: ' + sum + ', average: ' + average);  
};

console.timingReset = function(key)
{
 this.timings[key] = []; 
}; 

domingo, 22 de febrero de 2015

Writing unit tests for command line tools using jasmine and shelljs

So this is the situation, I have a project using gulp and the build tasks are increasingly getting complex and hard to maintain. These command line tools are written by me in JavaScript and I want a way of testing their command line API, similarly of how I write other source code unit test.

Before proceed, all the source code explained in this document is  maintained here https://github.com/cancerberoSgx/javascript-sample-projects/tree/master/gulp-test-setup-shelljs-jasmine

In particular, I want to write jasmine code with specs that call the command line tools and then assert if folders/files are created, etc. So I will use jasmine as the unit test framework and shelljs for easy filesystem operations & assertations.

So suppose you have a project with build system written with gulp or grunt, npm (or whatever) and you want to write a jasmine unit test that runs these tasks and make assertations. Then users can launch unit tests calling npm test on your app.

First we install jasmine and shelljs in your project:

npm install jasmine --save-dev
npm install shelljs --save-dev
Now let's generate the jasmine unit tests with the following command. This will generate the folder spec in which we will store our tests:

node node_modules/jasmine/bin/jasmine.js init
Now let's create the file specs/buildSpec.js with the following content.

require('shelljs/global');

describe("test the build", function() 
{
 it("npm install", function() 
 {
  rm('-rf', 'node_modules'); 
  expect(test('-d', 'node_modules')).toBe(false); 
  expect(exec('npm install', {silent:true}).code).toBe(0); 
  expect(test('-d', 'node_modules')).toBe(true); 
 }); 

 it("gulp sass", function() 
 {
  rm('-rf', 'dist'); 
  expect(test('-d', 'dist')).toBe(false); 
  expect(exec('gulp sass', {silent:true}).code).toBe(0); 
  expect(test('-f', 'dist/main.css')).toBe(true); 
 }); 

 it("gulp src", function() 
 {
  rm('-rf', 'dist'); 
  expect(test('-d', 'dist')).toBe(false); 
  expect(exec('gulp src', {silent:true}).code).toBe(0); 
  expect(test('-f', 'dist/all.js')).toBe(true); 
 }); 
});
This test only make sense in my application, you should write your own. I will use it only as an example.

As you can see it is a jasmine spec with three it(), the first checks if the command 'npm install' works, the second if the task 'gulp sass' works and the third if 'gulp src' works.

npm install: here the test first removes the folder node_module, then executes the command 'npm install' and then checks that the folder node_modules exists.

gulp sass: here the test removes the 'dist' folder, then executes the command 'gulp sass' and then makes sure the file dist/main.css was generated. gulp src is very similar.

Appendix: asserting if a server is turned on.

In my application, there is a command 'gulp connect' that will start a local server hosting the application. For testing this I use the following code.

It basically perform the following checks:

  1. check that the port 8080 is free 
  2. run the command 'gulp connect' asynchronously.
  3. wait for two seconds (time to start),
  4. check that the http port 8080 is used, 
  5. kill the server 6) and then checks that the port 8080 is free. Before killing a handler optionally given by the user is called so he can do some assertations over served resources (in my case using curl())
I tried to define a reusable and jasmine-agnostic function for testing all this automatically and assert something before killing the server. Unfortunately there are some hardcoded timeouts and have a asynchronous syntax so it might be hard to understand.

Notice how I have handcrafted an utility assertCommandOpenPort that performs all these work and a curl() function to assert on url resources. 


require('shelljs/global');

describe('gulp connect', function() 
{
 it('gulp connect should serve index at 8080', function(done) 
 {
  console.log('gulp connect should serve index at 8080'); 
  assertCommandOpenPort({
   cmd: 'gulp connect'
  , port:8080
  , timeout:2000
  , predicate: function(val, msg)
   { 
    if(!val)
    {
     expect('gulp connect fail: '+msg).toBe(false); 
    }
   }
  , testBeforeKill: function(done)
   {
    curl({
     host: 'localhost'
    , port: 8080
    , path: '/'
    , dataHandler: function(data, res)
     {
      expect(data.indexOf('</html>') !== -1).toBe(true); 
      done();
     }
    , errorHandler: function()
     {
      expect('html served').toBe(true); 
      done();
     }
    });     
   }
  , done: done
  }); 
 });  
});


// general utility for testing that a executing a command open a certain port. What it does: 
// 0) assert port is free 1) assert port is unused 2) run the command 3) assert port is used 4) kill the command 5) assert the port is free.
// I promise I tried to write the async part the easier I could....
function assertCommandOpenPort(config)
{
 var port = config.port
 , predicate = config.predicate
 , timeout = config.timeout || 2000; 
 isPortTaken(port, function(isUsed)
 {
  predicate(!isUsed, 'step 1 port taken initially'); 
 }); 
 var childProcess = exec(config.cmd, {silent: true, async: true});
 setTimeout(function()
 {
  isPortTaken(port, function(isUsed)
  {
   predicate(isUsed, 'step 2 port not taken'); 
   config.testBeforeKill(function()
   {
    childProcess.kill();
    setTimeout(function() // hack: give some time to the killing
    {
     isPortTaken(port, function(isUsed)
     {
      predicate(!isUsed, 'step 3 port not free');
      config.done();
     });
    }, 50); 
   });
      
  }); 
 }, timeout); // ugly hack: wait until server is up
}

// Utility used by assertCommandOpenPort to know if a port is currently used. Used by assertCommandOpenPort. 
// Usage: isPortTaken(8080, function(isUsed){})
function isPortTaken(port, fn) 
{
 var net = require('net')
 var tester = net.createServer()
 .once('error', function (err) 
 {
  if (err.code != 'EADDRINUSE') 
  {
   return fn(false); 
  }
  fn(true);  
 })
 .once('listening', function() 
 {
  tester.once('close', function() { fn(false);  })
  .close()
 })
 .listen(port)
}

function curl(config)
{
 var http = require('http');

 var options = {
     host: config.host
 ,   path: config.path || '/'
 , port: config.port || 80
 }; 
 var request = http.request(options, function (res) 
 {
  var data = '';
  res.on('data', function (chunk) 
  {
   data += chunk;
  });
  res.on('end', function () 
  {
   config.dataHandler && config.dataHandler(data, res);
  });
 });
 request.on('error', function (e) 
 {
  config.errorHandler && config.errorHandler(e);
 });
 request.end();
}


viernes, 9 de enero de 2015

Node watch task problems in linux&mac

Some developer tools I'm working with lately, like node watch, listen to changes in a filesystem folder using native system APIs for this. On unix systems I had some trouble with those, but good trouble, let me explain.

In unix systems there is a limit on the number of files being watched at a time b a process. And the kernel won't allow a process to watch too many files at the same time.

This is why, we can have trouble while executing development environments based in node watch (grunt watch, gulp watch, etc) in unix (linux and mac). The name of the error has the code : ENOSPC. It mostly fails or enters in a infinite loop, for example:

grunt watch
Warning: watch ENOSPC

Solution

The solution in linux is the following:
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

viernes, 2 de enero de 2015

my nodejs tool can't use itself

So I have this nodejs application short-jsdoc for documenting JavaScript code. The strange thing is that If I want to use it in itself, this is, installing it in its own project, nodejs refuses to do so:
cd short-jsdoc
npm install short-jsdoc --save-dev
npm WARN install Refusing to install short-jsdoc as a dependency of itself
At first I was surprised and I still am because nodejs always install dependencies locally inside the project and so this situation could be managed easily, but on the other side, the user from irc.freenode.org#Node.js give me crucial tips:
1) Why would I want foo depend on foo since in foo I already have all the code of foo?
2) If you are publishing a npm module then you should expose a main .js file and you can always require that file using require('./') syntax. That will always work. Thanks !

viernes, 3 de octubre de 2014

Bash script to measure http request times


The other day I created a nice bash script for measuring web site load time. Basically you pass an url you
want to test and the number of times you want to hit it and it returns the total time and promedy.

I use curl to make the http request and it returns me the response time in seconds (like "0.123").

The hard part was to be able to do the maths in bash. First I thought I needed to parse the numbers somehow,
but there is no such thing. I found the command bc and based the script on it to do the math.


Well, here is the code:

# Usage example: 
# sh run-times.sh 40 http://wordpress.com/

times=$1

i=0
sum=0
while [  $i -lt $times ]; do

 # make the request output will be the response time in seconds. 
 output=`curl -w "@curl-format-only-total.txt" -o /dev/null -s $2`

 #multiply by 1000
 ms=`echo "$output * 1000" | bc` 

 sum=`echo "$sum + $ms" | bc`

 #round float to integer
 sum=`printf "%.0f\n" "$sum"` 

 let i=i+1 

done

mean=`echo "$sum / $times" | bc` 

echo "count: $times, total_time: $sum, mean: $mean"


domingo, 13 de julio de 2014

How to update a branch with master

I always forget how to update a branch with all my changes in the master branch. For example in github, I want to update my branch gh-pages with my latest changes in master - so the documentation and examples of my projects are updated. This is how to do it:
git checkout gh-pages
git rebase master
git push

git checkout master

miércoles, 28 de mayo de 2014

How to identify in IRC

Sometimes I need to identify myself on irc chats, mostly freenode.org for programming questions and I often not remember how to. Well this is it:
/nick cancerbero_sgx
/msg NickServ identify my_secret