What is a real life project - one can ask. The real life - in terms of software development - is built of limitations. Your time is limited, the budget is limited, your abilities are limited and even knowledge of what you should exactly do is limited. Only your client's expectations and need to maintain the legacy code are unlimited. Funny - but let's puts jokes aside as this article should have some limit too.

Most of the times you can do nothing - or nearly nothing - about limited time or the budget. But the limited knowledge about the problem you have to solve is another kettle of ​fish. Requirements are liquid, definitions blur - and a software developer has to somehow find himself in this ever-changing environment. There are some tools and approaches, I will try to describe in this humble article, that can successfully diminish the impact of changing problem description on your code.

So here is the story (based on facts, but modified and simplified to the needs of the article): there are several sensors distributed around the neighbourhood, we want to collect their measurements and expose them to our system. Here is what we know about the devices:

  • each device will connect to the tcp server we will provide,
  • each device has to be polled to acquire a measurement,
  • each device will communicate with us in accordance with provided protocol

And the protocol is:

  • after the connection with the sensor is established server should send CONFIRMATION frame,
  • each time we want to gather fresh measurement server should send STATUS frame - the sensor will reply with DATA frame,
  • the sensor will reply the server with NACK frame in answer to any wrong frame received

And the frames are:

  • CONFIRMATION frame:
  • STATUS frame:
  • DATA frame:
  • NACK frame:

So what's the catch?

The catch is that the described sensors don't exist yet. We are waiting for the producer to finish them. And we are provided with some initial information about how will they probably work. The good point is that we can influence the final shape of the device. For example it will use tcp protocol instead of virtual serial port to communicate with us. The bad point is that we have no sample to experiment with.

Unfortunately we don't know much about the communication protocol. We only know the expected shapes of the frames but don't know exact values of fields like stx or etx etc. We don't know how should we calculate crc value and how to interpret data_low and data_high fields. Should we send a frame periodically to keep the connection alive, or the sensor device will deal with it itself? Again - don't know.

Oh, I have forgotten to add - the sensors will be ready after a week time, and we will have only couple of days to play with them before we will have to send them to the production. Reality hits you hard, bro...

Black box

Each piece of a software can be treated as a black box. But what does that really mean? It means that specified action will cause a certain reaction. How will it happen? No one knows or cares - that's why black.

How can we use it? We have to define inputs and outputs of our box, write end to end tests checking if expected reactions are results of specified actions and then put something in the box what make tests pass. Of course, we don't have to create everything at once - nice feature of the black box is that it can hold another black boxes inside. I will try to present this approach by example.

What are the profits of writing end to end tests in the beginning? We have nice documentation describing exactly what our software is supposed to do. We can monitor a state of our application during the development - if we break something, we will notice. And finally - we can perform drastic refactoring (like for example removing everything and starting from scratch) and can be sure that we didn't forget about any feature we should implement.

Abstraction hunting - first attempt

What do we know about the task? Let's sum up. We have N number of devices and we need to poll them to send acquired data to the rest of our system. We need to be flexible in the way we handle the sensors as we don't have any sure knowledge about how to integrate. The only certain thing is how we should handle with the rest of our system. External components communication always carries more risk than internal one. I can see following entities in our ecosystem: sensor and sensorAgent. Sensor will represent mocked device and sensorAgent stands for the component we are trying to create. It will aggregate data form sensors and expose it conveniently. The process will be something like this: acquire frames in a kind of dataSource, convert it a little bit with dataConverter and throw it inside a dataSink so it will be available to the rest of the system. But we shouldn't go too deeply into these considerations. One thing at a time! Let's do some coding...

Walking skeleton

Ok, again I have forgotten to mention - this app will be written in Node Js. By the way, that was my first app written with Node Js technology - so please, forgive me (but don't hesitate to indicate in comments) the glitches you will probably find in my code. My first app in Node Js folks! Reality hits you hard...

Integration with the system - step 1 - test

Let's prepare basic parts of our application:

package.json

{
  "name": "sensor-agent",
  "version": "1.0.0",
  "description": "",
  "main": "app.js",
  "scripts": {
    "start": "node app.js",
    "test": "mocha test"
  },
  "author": "Damian Krychowski",
  "license": "MIT",
  "dependencies": {
    "bluebird": "^3.3.5",
    "express": "^4.13.4",
    "mocha": "^2.4.5",
    "net": "^1.0.2",
    "request-promise": "^3.0.0",
    "should": "^8.3.1",
    "winston": "^2.2.0"
  }
}

The starting point of our application will be app.js, and all tests will be located within test folder.

app.js

var logger = require("winston");
var sensorAgentFactory = require("./sensorAgent");
var config = require("./config")

var sensorAgent = sensorAgentFactory.create(config);

sensorAgent
    .startAsync()
    .catch(err => logger.error(err.toString()));

App.js file is simple now - and I would like it to stay simple forever. This is the entry point of our application - it creates the sensorAgent object with configuration and launches it. We probably won't use app.js for quite a long time so forget about it for now.

sensorAgent.js

var webSinkFactory = require("./webSink");

module.exports.create = function(config){
    var sensorAgent = {
        startAsync: startAsync,
        stopAsync: stopAsync,

        exposeData: exposeData,        
    }

    var webSink = webSinkFactory.create(config);

    function startAsync(){
        return webSink.startAsync();
    }

    function stopAsync(){
        return webSink.stopAsync();
    }

    function exposeData(newData){
        webSink.exposeData(newData);
    }

    return sensorAgent;
}

SensorAgent will be the core component of the app. It may be started, stopped or new sensors data can be exposed with it. It is created with the configuration object, which is currently empty:

config.js

module.exports.config = {
}

WebSink required in sensorAgent will be used to expose sensors measurements via http server. It will be implemented later.

webSink.js

module.exports.create = function (config) {
    var webSink = {
        startAsync: startAsync,
        stopAsync: stopAsync,
        exposeData: exposeData
    };

    function startAsync() {
    }

    function stopAsync() {
    }

    function exposeData(newValue) {
    }

    return webSink;
}

Now the most important part - first end-to-end test checking if exposed sensors data are available externally.

test/endToEnd.js

var logger = require("winston");
var should = require("should");
var sensorAgentFactory = require("../sensorAgent");
var webSinkRequestFactory = require("./webSinkRequest");

describe("Full end to end tests", function () {

    logger.remove(logger.transports.Console);
    logger.add(logger.transports.Console, { "timestamp": true, "colorize": true });
    logger.level = "fatal";

    const config = {
        webSinkPort: 6001
    };

    const webSinkUrl = "http://localhost:" + config.webSinkPort + "/sensors";

    var sensorAgent;
    var webSinkRequest = webSinkRequestFactory.create(webSinkUrl);

    beforeEach(function () {
        sensorAgent = sensorAgentFactory.create(config);
    });

    afterEach(function () {
        return sensorAgent.stopAsync();
    });

    it("sensorAgent should expose data acquired from sensors externally", function () {
        var sensorData = {
            sensorId: 1,
            sensorValue: 1000
        };

        return sensorAgent
            .startAsync()
            .then(() => sensorAgent.exposeData(sensorData))
            .then(() => webSinkRequest.executeAsync())
            .then(acquiredData => acquiredData.should.be.eql(sensorData));
    });

});

As we can see, our first test is ready. It is quite straightforward, but let's explain it step by step. The endToEnd.js file will be a container for all our end-to-end tests. We will be adding them incrementally alongside with our code base growth. Remember - one thing at a time! At beginning of tests we setting logger configuration so the console after we run test script is not dominated with logs. If we find that our tests are not passing we can change it to follow and understand the process and find the origination of the errors.

There are beforeEach and afterEach sections responsible accordingly for setting up and cleaning our core component - sensorAgent.

First test checks if data exposed with sensorAgent are available externally. To make sure they are, there is http request performed.

test/webSinkRequest.js

var request = require("request-promise");

module.exports.create = function(webSinkUrl) {
    var webSinkRequest = {
        executeAsync: executeAsync
    };

    function executeAsync(subzoneId, availableBays) {
        var options = {
            method: "GET",
            uri: webSinkUrl,
            json: true
        }

        return request(options);
    }

    return webSinkRequest;
}

As you have already noticed I am going to base the sensor application on Promises with the Bluebird library. As this is my first Node Js application (that was not a joke!) I have done research about callbacks vs promises approach. I have asked my more experienced colleagues but the opinions were divided. This page helped to finally make my mind up.

Let's run the test with "npm test" command

first test

As it was expected - test failed.

Integration with the system - step 2 - implementation

Implementation of the webSink component is an easy one. It shouldn't be changed much during the development process as our project team have already decided on the format and the way sensor data should be exposed to the rest of the system. So we are covering easier part of the application first - to warm up. Another benefit of doing it in the beginning of the app development is that we can deploy it and our colleagues can consume mocked data somewhere else - so no one is blocked with his job.

webSink.js

var Promise = require("bluebird");
var express = require("express");
var logger = require("winston");
var config = require("./config");

module.exports.create = function (config) {
    var webSink = {
        startAsync: startAsync,
        stopAsync: stopAsync,
        exposeData: exposeData
    };

    var server = express();
    var currentState;
    var sensorHttpServer;

    server.get("/sensors", function (req, res) {
        res.send(currentState);
    });

    function startAsync() {
        return new Promise(function (resolve, reject) {
            sensorHttpServer = server.listen(config.webSinkPort, function (err) {
                if (err) { reject(err); }
                else { resolve(); }
            });
        }).then(() => logger.info("Sensor web sink is started on port " + config.webSinkPort));
    }

    function stopAsync() {
        return new Promise(function (resolve, reject) {
            sensorHttpServer.close(function (err) {
                if (err) { reject(err); }
                else { resolve(); }
            });

        }).then(() => logger.info("Sensor web sink on " + config.webSinkPort + " port is stopped."));
    }

    function exposeData(newValue) {
        currentState = newValue;
    }

    return webSink;
}

The implementation of webSink component is achieved with express library. The library allows to start http server up easily. Function responsible for exposing data changes the internal state of the component which is returned as a result of http request.

Now the test is passing! So we can move on to more interesting features.

second test

Integration with the sensors - step 1 - test

The webSink is ready but the only data it can expose comes from sensorAgent.exposData() function calls. The aim is to remove .exposeData() function from the sensorAgent and replace it with internal mechanisms translating received TCP frames into convienient sensor measurements data. We will need kind of a connectionManager component responsible for handling many sensorConnections. But before we get to it let's create at least one test which help us develop necessary components under controlled conditions.

I have created very simple test checking if sensorAgent accept new sensor connection and reply accordingly to the protocol.

test/endToEnd.js

it("sensorAgent should accept connection from sensor and reply with CONFIRMATION frame", function () {
    var receivedFrame;

    var sensor = sensorFactry
        .create({serverPort: 6002, serverHost: "localhost"}, frame => receivedFrame = frame);

    return sensorAgent
        .startAsync()
        .then(() => sensor.connectAsync())
        .delay(50)
        .then(() => receivedFrame.should.be.eql(new Buffer([0x01, 0x02, 0x03, 0x04])));
});

Sensor is trying to connect to the started sensorAgent, and after 50 milliseconds left to the system to settle, received frame is checked. Presented code is rather raw and not too easy to understand, but it will be refactored later. As we don't know values of the specific frame fields I used random ones. We need this test only to start the implementation of new features in the sensorAgent component. But firstly, let's implement the sensorFactory so we can run the test.

test/sensor.js

var Promise = require("bluebird");
var net = require("net");
var logger = require("winston");

module.exports.create = function(config, frameReceivedHandler){
    var sensor = {
        connectAsync: connectAsync
    }

    var client = new net.Socket();
    var identity;

    client.on("data", function(data) {
        frameReceivedHandler(data);
    });

     function connectAsync() {
        return new Promise(function(resolve, reject) {
            client.connect(config.serverPort, config.serverHost, function(err) {
                if (err) {
                    logger.error("Client error " + err.toString());
                    reject(err);
                }
                else {
                    identity = client.localAddress + ":" + client.localPort;
                    resolve();
                }
            });
        });
    }

    return sensor;
}

Sensor component code is simple - it allows to create connection with specified host and port, and calls the handler if new data is received. It should be enough for now - we will add more functions after we implement basic functionality described by added test in the sensorAgent.

And of course:

third test

The test fails.

Integration with the sensors - step 2 - implementation

I was trying to implement only as much as we need to make second test green. This is the outcome:

sensorConnectionManager.js

var Promise = require("bluebird");
var net = require("net");
var logger = require("winston");

module.exports.create = function (config) {
    var sensorConnectionManager = {
        startAsync: startAsync,
        stopAsync: stopAsync
    }

    var server;
    var connections = [];

    function startAsync() {
        return new Promise(function (resolve, reject) {
            server = net.createServer(createConnection);

            server.listen(config.sensorServerPort, config.sensorServerHost, function (err) {
                if (err) { reject(err); }
                else { resolve(); }
            });
        }).then(() => logger.info("Sensor server started on " + config.sensorServerPort + " port."));
    }

    function stopAsync() {
        return new Promise(function (resolve, reject) {
            closeConnections();

            server.close(function (err) {
                if (err) { reject(err); }
                else { resolve(); }
            });
        }).then(() => logger.info("Sensor server is stopped"));
    }

    function createConnection(newConnection) {
        connections.push(newConnection);
        newConnection.write(new Buffer([0x01, 0x02, 0x03, 0x04]));
    }

    function closeConnections(){
        connections.forEach(conn => conn.end());
    }

    return sensorConnectionManager;
}

The manager can only be started or stopped. It uses config object to acquire information about server port and host. The config object will be mocked during tests so we can control the environment of the application.

I had to modify sensorAgent component and add sensorServerPort and sensorServerHost to config object in endToEnd.js file.

sensorAgent.js

var webSinkFactory = require("./webSink");
var sensorConnectionManagerFactory = require("./sensorConnectionManager");

module.exports.create = function(config){
    var sensorAgent = {
        startAsync: startAsync,
        stopAsync: stopAsync,

        exposeData: exposeData,        
    }

    var webSink = webSinkFactory.create(config);
    var sensorConnectionManager = sensorConnectionManagerFactory.create(config);

    function startAsync(){
        return webSink.startAsync()
            .then(() => sensorConnectionManager.startAsync());
    }

    function stopAsync(){
        return webSink.stopAsync()
            .then(() => sensorConnectionManager.stopAsync());
    }

    function exposeData(newData){
        webSink.exposeData(newData);
    }

    return sensorAgent;
}

test/endToEnd.js

const config = {
    webSinkPort: 6001,
    sensorServerPort: 6002,
    sensorServerHost: "localhost"
};

Start and stop functions were modified. After these simple modifications our tests are green.

fourth test

Abstraction hunting - second attempt

In my opinion we can officially close the 'walking skeleton' chapter. We have prepared very basic functionality touching each endpoint of our system. We have webSink prototype for exposing converted sensor data externally and sensorConnectionManager to deal with incoming sensor tcp connections. We have already achieve much, but still there is a long way to go. Let's pause for a minute and try to reconsider abstractions in our system. The sensor and sensorAgent are doing their job. Each time I am trying to discover more abstractions in the system I am concentrating on the ugliest parts of my code. Here are the candidates:

test/endToEnd.js

.then(() => receivedFrame.should.be.eql(new Buffer([0x01, 0x02, 0x03, 0x04])));

Creating a buffer with some bytes inside the end-to-end test is not making it easier to understand. I suppose we should create something like frame to encapsulate all knowledge (which we don't have at the moment) about sensor communication protocol. This step will provide us with a few benefits

  • our end-to-end tests will be easier to understand,
  • out end-to-end tests will describe the flow of the application and not the implementation details,
  • we will be free to change the implementation of the frame at any moment - so we can provide all stx, etx etc. fields with proper values when we will know them.

sensorConnectionManager.js

var connections = [];

function createConnection(newConnection) {
    connections.push(newConnection);
    newConnection.write(new Buffer([0x01, 0x02, 0x03, 0x04]));
}

function closeConnections(){
    connections.forEach(conn => conn.end());
}

Another candidate for the ugliest part is located in the sensorConnectionManager. As we can see there are methods dealing with single connections. 'Manager' would suggest dealing with collection, not with single element. SensorConnection seems to be legit to be introduced.

Frame - step 1 - tests

There are four types of frame: confirmation, status, data and nack. The goal is to encapsulate all frame-related knowledge so it can be easily modified in one place. We will introduce unit tests to be sure that our encapsulation is correct. Let's prepare some tests:

test/frameTests.js

var logger = require("winston");
var should = require("should");
var frameFactory = require("../frame");

describe("Creating frames from bytes", function () {
    it("Confirmation frame should be created from bytes array", function () {
        //                                                     STX,   FC,  CRC,  ETX 
        frame = frameFactory.createFrameFromBytes(new Buffer([0x01, 0x01, 0x00, 0x02]));

        frame.isConfirmation().should.be.true();

        frame.isStatus().should.not.be.true();
        frame.isData().should.not.be.true();
        frame.isNack().should.not.be.true();
    });

    it("Status frame should be created from bytes array", function () {
        //                                                     STX,   FC,  CRC,  ETX 
        frame = frameFactory.createFrameFromBytes(new Buffer([0x01, 0x02, 0x00, 0x02]));

        frame.isStatus().should.be.true();

        frame.isConfirmation().should.not.be.true();
        frame.isData().should.not.be.true();
        frame.isNack().should.not.be.true();
    });

    it("Data frame should be created from bytes array", function () {
        //                                                     STX,   FC,   ID,   DL,   DH,  CRC,  ETX
        frame = frameFactory.createFrameFromBytes(new Buffer([0x01, 0x03, 0x01, 0x02, 0x03, 0x00, 0x02]));

        frame.isData().should.be.true();
        frame.getSensorId().should.be.eql(1);
        frame.getSensorValue().should.be.eql(5);

        frame.isConfirmation().should.not.be.true();
        frame.isStatus().should.not.be.true();
        frame.isNack().should.not.be.true();
    });

    it("Nack frame should be created from bytes array", function () {
        //                                                     STX,   FC,  ERR,  CRC,  ETX
        frame = frameFactory.createFrameFromBytes(new Buffer([0x01, 0x04, 0x01, 0x00, 0x02]));

        frame.isNack().should.be.true();
        frame.getErrorCode().should.be.eql(1);

        frame.isConfirmation().should.not.be.true();
        frame.isStatus().should.not.be.true();
        frame.isData().should.not.be.true();
    });

    it("Should throw on wrong crc value", function (done) {
        try {
            //                                                     STX,   FC,  CRC,  ETX 
            frame = frameFactory.createFrameFromBytes(new Buffer([0x01, 0x01, 0xFF, 0x02]));
        } catch (error) {
            done();
        }
    });

    it("Should throw on unkown frame type", function (done) {
        try {
            //                                                     STX,   FC,  CRC,  ETX 
            frame = frameFactory.createFrameFromBytes(new Buffer([0x01, 0xFF, 0x00, 0x02]));
        } catch (error) {
            done();
        }
    });

    it("Should throw on wrong frame format", function (done) {
        try {                                                 
            frame = frameFactory.createFrameFromBytes(new Buffer([0xFF, 0xFF]));
        } catch (error) {
            done();
        }
    });
});

First portion of tests checks if the frameFactory, which we are going to create, allows us to build different types of frames directly from bytes array. There are also tests checking if error is thrown when wrong crc, wrong frame code or unexpected bytes array are provided. As the frame is a delicate part of our code (we don't have full knowledge about how it will behave with real sensors) we should be particularly meticulous while creating different test cases.

Currently, we have covered function parsing bytes into the frame. It will be useful during accepting received frames from connected sensors. However, we will be forced to send some frames back, and don't forget about our end-to-end tests, where we will be creating frames too. CreateFrameFromBytes function doesn't provide encapsulation we have expected at all. Let's get it a step further.

test/frameTests.js

describe("Creating frames with convinient functions", function () {
    it("Confirmation frame should be created", function () {
        frame = frameFactory.createConfirmationFrame();

        frame.isConfirmation().should.be.true();

        frame.isStatus().should.not.be.true();
        frame.isData().should.not.be.true();
        frame.isNack().should.not.be.true();
    });

    it("Status frame should be created", function () {
        frame = frameFactory.createStatusFrame();

        frame.isStatus().should.be.true();

        frame.isConfirmation().should.not.be.true();
        frame.isData().should.not.be.true();
        frame.isNack().should.not.be.true();
    });

    it("Data frame should be created", function () {
        frame = frameFactory.createDataFrame(1, 5);

        frame.isData().should.be.true();
        frame.getSensorId().should.be.eql(1);
        frame.getSensorValue().should.be.eql(5);

        frame.isConfirmation().should.not.be.true();
        frame.isStatus().should.not.be.true();
        frame.isNack().should.not.be.true();
    });

    it("Nack frame should be created", function () {
        frame = frameFactory.createNackFrame(1)

        frame.isNack().should.be.true();
        frame.getErrorCode().should.be.eql(1);

        frame.isConfirmation().should.not.be.true();
        frame.isStatus().should.not.be.true();
        frame.isData().should.not.be.true();
    });    
});

OK, now we have tools we can work with. Let's run the tests.

sensor-tests

Frame - step 2 - implementation

As always, we should try to make prepared tests green. Let's start from .createFrameFromBytes function.

frame.js

const STX = 0x01;
const STX_INDEX = 0x00;

const ETX = 0x02;

const FRAME_CODE_INDEX = 0x01;

const CONFIRMATION_FRAME_CODE = 0x01;
const CONFIRMATION_CRC_INDEX = 0x02;
const CONFIRMATION_ETX_INDEX = 0x03;

const STATUS_FRAME_CODE = 0x02;
const STATUS_CRC_INDEX = 0x02;
const STATUS_ETX_INDEX = 0x03;

const DATA_FRAME_CODE = 0x03;
const DATA_SENSOR_ID_INDEX = 0x02
const DATA_LOW_INDEX = 0x03;
const DATA_HIGH_INDEX = 0x04;
const DATA_CRC_INDEX = 0x05;
const DATA_ETX_INDEX = 0x06;

const NACK_FRAME_CODE = 0x04;
const NACK_ERROR_CODE_INDEX = 0x02;
const NACK_CRC_INDEX = 0x03;
const NACK_ETX_INDEX = 0x04;

function createFrameFromBytes(bytes) {
    var frame = {
        getBytes: getBytes,
        isConfirmation: isConfirmation,
        isStatus: isStatus,
        isData: isData,
        isNack: isNack,
        getSensorId: getSensorId,
        getSensorValue: getSensorValue,
        getErrorCode: getErrorCode
    }

    var frameType;
    verify();

    function getBytes() {
        return bytes;
    }

    function isStatus() {
        return frameType === "status";
    }

    function isConfirmation() {
        return frameType === "confirmation";
    }

    function isData() {
        return frameType === "data";
    }

    function isNack() {
        return frameType === "nack";
    }

    function getSensorId() {
        if (isData()) {
            return bytes[DATA_SENSOR_ID_INDEX];
        }
    }

    function getSensorValue() {
        if (isData()) {
            //todo modify due to specification
            return bytes[DATA_HIGH_INDEX] + bytes[DATA_LOW_INDEX];
        }
    }

    function getErrorCode() {
        if (isNack()) {
            return bytes[NACK_ERROR_CODE_INDEX];
        }
    }

    function verify() {
        byteShouldBeEqualTo(bytes, STX_INDEX, STX);
        verifyFrameCode();
    }

    function verifyFrameCode() {
        if (bytes[FRAME_CODE_INDEX] === CONFIRMATION_FRAME_CODE) {
            verifyConfirmationFrame();
        }
        else if (bytes[FRAME_CODE_INDEX] === STATUS_FRAME_CODE) {
            verifyStatusFrame();
        }
        else if (bytes[FRAME_CODE_INDEX] === DATA_FRAME_CODE) {
            verifyDataFrame();
        }
        else if (bytes[FRAME_CODE_INDEX] === NACK_FRAME_CODE) {
            verifyNackFrame();
        }
        else {
            throw new Error("Unkown frame code.")
        }
    }

    function verifyConfirmationFrame() {
        frameType = "confirmation";

        verifyCrc(bytes[CONFIRMATION_CRC_INDEX]);
        byteShouldBeEqualTo(bytes, CONFIRMATION_ETX_INDEX, ETX);
    }

    function verifyStatusFrame() {
        frameType = "status";

        verifyCrc(bytes[STATUS_CRC_INDEX]);
        byteShouldBeEqualTo(bytes, STATUS_ETX_INDEX, ETX);
    }

    function verifyDataFrame() {
        frameType = "data";

        //todo add data high and data low verification

        verifyCrc(bytes[DATA_CRC_INDEX]);
        byteShouldBeEqualTo(bytes, DATA_ETX_INDEX, ETX);
    }

    function verifyNackFrame() {
        frameType = "nack";

        //todo add error code verification        

        verifyCrc(bytes[NACK_CRC_INDEX]);
        byteShouldBeEqualTo(bytes, NACK_ETX_INDEX, ETX);
    }

    function verifyCrc(actualCrc) {
        //todo modify due to specification
        if (actualCrc != 0x00) {
            throw new Error("Wrong crc value. Actual: " + actualCrc + ", expected: " + 0x00);
        }
    }

    function byteShouldBeEqualTo(buffer, index, expected) {
        if (buffer[index] != expected) {
            throw new Error("Wrong byte value (" + buffer[index] + ") on " + index + " position. Expected " + expected);
        }
    }

    return frame;
}

module.exports.createFrameFromBytes = createFrameFromBytes;

In the beginning I have defined constants representing values directly connected with the protocol. I really don't like using magic numbers my in code - they are difficult to understand, and what is worse, to replace. Then, I have prepared each functions used in tests. In the end I have added verify functions. And the result is:

createFrameFromBytes tests

Now we should implement convenient functions. To achieve this we can use .createFrameFromBytes function.

frame.js

module.exports.createConfirmationFrame = function () {
    var frameBytes = [STX, CONFIRMATION_FRAME_CODE];
    var crc = calculateCRC(frameBytes);

    frameBytes.push(crc);
    frameBytes.push(ETX);

    return createFrameFromBytes(new Buffer(frameBytes));
}

module.exports.createDataFrame = function (sensorId, sensorValue) {
    var frameBytes = [STX, DATA_FRAME_CODE, sensorId, 0, sensorValue];
    var crc = calculateCRC(frameBytes);

    frameBytes.push(crc);
    frameBytes.push(ETX);

    return createFrameFromBytes(new Buffer(frameBytes));
}

module.exports.createStatusFrame = function () {
    var frameBytes = [STX, STATUS_FRAME_CODE];
    var crc = calculateCRC(frameBytes);

    frameBytes.push(crc);
    frameBytes.push(ETX);

    return createFrameFromBytes(new Buffer(frameBytes));
}

module.exports.createNackFrame = function (errorCode) {
    var frameBytes = [STX, NACK_FRAME_CODE, errorCode];
    var crc = calculateCRC(frameBytes);

    frameBytes.push(crc);
    frameBytes.push(ETX);

    return createFrameFromBytes(new Buffer(frameBytes));
}

function calculateCRC(frameBytes){
    return 0x00;
}

As we don't know how to calculate crc value yet, I have chosen the simplest assumption, that it will always be zero.

Finally, all tests are green.

frame tests green

SensorConnection - step 1 - tests

The frame component is tested and ready. Now we can move on to sensorConnection component. Actually, the sensorConnection is only a tool to achieve greater goal. The goal is to handle different scenarios of sensors behavior. For example:

  • more than one sensor is trying to connect with the agent,
  • connection with a sensor is broken, the sensor is trying to reconnect,
  • agent is trying to get sensors measurements,
  • sensor reply with data frame,
  • sensor reply with nack frame,
  • sensor reply with wrong frame.

    Let's add those scenarios to our end-to-end tests. But firstly, we should refactor current tests.

End-to-end tests refactoring

Currently our test looks like this:

test/end-to-end.js

it("sensorAgent should accept connection from sensor and replay with CONFIRMATION frame", function () {
    var receivedFrame;

    var sensor = sensorFactory
        .create({serverPort: 6002, serverHost: "localhost"}, frame => receivedFrame = frame);

    return sensorAgent
        .startAsync()
        .then(() => sensor.connectAsync())
        .delay(50)
        .then(() => receivedFrame.should.be.eql(new Buffer([0x01, 0x02, 0x03, 0x04])));
});

Let's replace the buffer with freshly created frame component.

test/end-to-end.js

it("sensorAgent should accept connection from sensor and reply with CONFIRMATION frame", function () {
    var receivedFrame;

    var sensor = sensorFactory
        .create({serverPort: 6002, serverHost: "localhost"}, frame => receivedFrame = frame);

    return sensorAgent
        .startAsync()
        .then(() => sensor.connectAsync())
        .delay(50)
        .then(() => receivedFrame.should.be.eql(
            frameFactory.createConfirmationFrame().getBytes()
        ));
});

It looks good now, but can be even better...

test/end-to-end.js

it("sensorAgent should accept connection from sensor and reply with CONFIRMATION frame", function () {
    var receivedFrame;

    var sensor = sensorFactory
        .create({ serverPort: 6002, serverHost: "localhost" }, frame => receivedFrame = frame);

    return sensorAgent
        .startAsync()
        .then(() => sensor.connectAsync())
        .delay(50)
        .then(() => receivedFrame.should.be.eql(confirmationFrame()));
});

function confirmationFrame() {
    return frameFactory
        .createConfirmationFrame()
        .getBytes()
}

first refactoring

The test failed because we forgot to amend sensorConnectionManager component. Let's do it now.

sensorConnectionManager.js

function createConnection(newConnection) {
    connections.push(newConnection);
    newConnection.write(frameFactory.createConfirmationFrame().getBytes());
}

And as a result we have...

first refactoring green

... tests green again!

Let's look at this fragment of code:

test/end-to-end.js

var receivedFrame;

var sensor = sensorFactory
    .create({ serverPort: 6002, serverHost: "localhost" }, frame => receivedFrame = frame);

It is quite ugly. We have a lambda, we have raw variable - it is quite technical. Good end-to-end test should not be technical at all. It should be like best-selling novels - easy to understand and should not surprise you too much. That test should look like this:

test/end-to-end.js

it("sensorAgent should accept connection from sensor and reply with CONFIRMATION frame", function () {
    var sensor = sensorFactory.create()
        .withServerHost("localhost")
        .withServerPort(6002);

    return sensorAgent
        .startAsync()
        .then(() => sensor.connectAsync())
        .delay(50)
        .then(() => sensor.firstReceivedFrame.should.be.eql(confirmationFrame()));
});

To achieve this we need to work on sensor component.

test/sensor.js

module.exports.create = function(){
    var sensor = {
        withServerHost: withServerHost,
        withServerPort: withServerPort,
        connectAsync: connectAsync
    }

    var serverHost, serverPort;
    var client = new net.Socket();
    var identity;
    var receivedData = [];
    var receivedDataCounter = 0;

    function withServerHost(host){
        serverHost = host;
        return sensor;
    }

    function withServerPort(port){
        serverPort = port;
        return sensor;
    }

    client.on("data", function(data) {
        receivedData.push(data);
        receivedDataCounter++;

        if(receivedDataCounter === 1){
            sensor.firstReceivedFrame = data;
        }
    });

     function connectAsync() {
        return new Promise(function(resolve, reject) {
            client.connect(serverPort, serverHost, function(err) {
                if (err) {
                    logger.error("Client error " + err.toString());
                    reject(err);
                }
                else {
                    identity = client.localAddress + ":" + client.localPort;
                    resolve();
                }
            });
        });
    }

    return sensor;
}

sensor refactored

Everything is green so we consider end-to-end tests refactoring finished. All technical stuff is hidden inside sensor component and our novel-like test is easy to follow. Now we should concentrate on describing different scenarios of communication.

End-to-end tests for different sensor communication scenarios

SensorAgent should be able to accept connection from more than one sensor. As protocol describes, each fresh connection should be replied with confirmation frame.

test/end-to-end.js

it("sensorAgent should accept connections from more than one sensor", function () {
    var firstSensor = sensorFactory.create()
        .withServerHost("localhost")
        .withServerPort(6002);

    var secondSensor = sensorFactory.create()
        .withServerHost("localhost")
        .withServerPort(6002);

    return sensorAgent
        .startAsync()
        .then(() => firstSensor.connectAsync())
        .then(() => secondSensor.connectAsync())
        .delay(50)
        .then(() => firstSensor.firstReceivedFrame.should.be.eql(confirmationFrame()))
        .then(() => secondSensor.firstReceivedFrame.should.be.eql(confirmationFrame()));
});

If something will happen to the connection between agent and a sensor, the sensor will try to reconnect. SensorAgent should be able to accept this second connection attempt.

test/end-to-end.js

it("sensorAgent should accept second connection from the sensor if the first connection was lost", function () {
    var sensor = sensorFactory.create()
        .withServerHost("localhost")
        .withServerPort(6002);

    return sensorAgent
        .startAsync()
        .then(() => sensor.connectAsync())
        .delay(25)
        .then(() => sensor.destroyConnection())
        .delay(25)
        .then(() => sensor.connectAsync())
        .delay(50)
        .then(() => sensor.firstReceivedFrame.should.be.eql(confirmationFrame()))
        .then(() => sensor.secondReceivedFrame.should.be.eql(confirmationFrame()))
});

To acquire fresh sensor's measurements, sensorAgent should send status frame to the desired sensor. I have introduced .collectDataFromSensorsAsync() function in sensorAgent component. The function will trigger process of acquiring data from sensors. It is not yet implemented, so we should take care of it in the implementation phase.

test/end-to-end.js

it("sensorAgent should send STATUS frame to acquire data from the sensor", function () {
    var sensor = sensorFactory.create()
        .withServerHost("localhost")
        .withServerPort(6002);

    return sensorAgent
        .startAsync()
        .then(() => sensor.connectAsync())
        .delay(50)
        .then(() => sensorAgent.collectDataFromSensorsAsync())
        .delay(50)
        .then(() => sensor.firstReceivedFrame.should.be.eql(confirmationFrame()))
        .then(() => sensor.secondReceivedFrame.should.be.eql(statusFrame()))
});

function statusFrame(){
    return frameFactory
        .createStatusFrame()
        .getBytes();
}

There is the nack frame described in the protocol. At present, we don't know much about it. We know that the nack frame will be a response to an error connected with the frame we previously sent to the sensor. As our knowledge is limited, I have decided that we should close the connection with such a sensor, and of course, log this behaviour. Maybe later, when we will be provided with more accurate information about sensors we will come back to change this.

test/end-to-end.js

it("sensorAgent should close connection with the sensor on receiving NACK frame", function () {
    var sensor = sensorFactory.create()
        .withServerHost("localhost")
        .withServerPort(6002)
        .withFramesToRespondWith([
            nackFrame({ errorCode: 0x01 })
        ]);

    return sensorAgent
        .startAsync()
        .then(() => sensor.connectAsync())
        .delay(50)
        .then(() => sensorAgent.collectDataFromSensorsAsync())
        .delay(50)
        .then(() => sensor.isConnected().should.not.be.true());
});

function nackFrame(nackData) {
    return frameFactory
        .createNackFrame(nackData.errorCode)
        .getBytes();
}

The same behavior in response to unknown frame received from the sensor.

test/end-to-end.js

it("sensorAgent should close connection with the sensor on receiving unknown frame", function () {
    var sensor = sensorFactory.create()
        .withServerHost("localhost")
        .withServerPort(6002)
        .withFramesToRespondWith([
            new Buffer([0xFF, 0xFF])
        ]);

    return sensorAgent
        .startAsync()
        .then(() => sensor.connectAsync())
        .delay(50)
        .then(() => sensorAgent.collectDataFromSensorsAsync())
        .delay(50)
        .then(() => sensor.isConnected().should.not.be.true());
});

And finally, we are testing system as a whole. We are creating two sensors which will connect the sensorAgent. Then, sensorAgent gathers their measurements and exposes them with the WebSink.

test/end-to-end.js

it("sensorAgent should expose data acquired from the sensors", function () {
    var firstSensor = sensorFactory.create()
        .withServerHost("localhost")
        .withServerPort(6002)
        .withFramesToRespondWith([
            dataFrame({sensorId: 1, sensorValue: 25})
        ]);

    var secondSensor = sensorFactory.create()
        .withServerHost("localhost")
        .withServerPort(6002)
        .withFramesToRespondWith([
            dataFrame({sensorId: 2, sensorValue: 10})
        ]);

    return sensorAgent
        .startAsync()
        .then(() => firstSensor.connectAsync())
        .then(() => secondSensor.connectAsync())
        .delay(50)
        .then(() => sensorAgent.collectDataFromSensorsAsync())
        .delay(50)
        .then(() => webSinkRequest.executeAsync())
        .then(acquiredData => acquiredData.should.be.eql([
            { sensorId: 1, value: 25 },
            { sensorId: 2, value: 10 }
        ]));
});

function dataFrame(data) {
    return frameFactory
        .createDataFrame(data.sensorId, data.sensorValue)
        .getBytes();
}

During preparation of the tests presented above, I have introduced a lot of new functions to the sensor component. I really like this freedom dynamic language grant, which allows me to use functions which don't exist yet. Writing tests in this fashion is very comfortable. There is no compiler yelling at me and pointing out 'mistakes' I have deliberately made. The bad point is, that there is no compiler yelling at me and pointing out my real mistakes, which I don't know about. Quid pro quo...

test/sensor.js

var Promise = require("bluebird");
var net = require("net");
var logger = require("winston");
var frameFactory = require("../frame");

module.exports.create = function () {
    var sensor = {
        withServerHost: withServerHost,
        withServerPort: withServerPort,
        withFramesToRespondWith: withFramesToRespondWith,
        connectAsync: connectAsync,
        destroyConnection: destroyConnection,
        isConnected: isConnected
    }

    var serverHost, serverPort;
    var client = new net.Socket();
    var identity;
    var receivedDataCounter = 0;
    var framesToRespondWith;
    var framesToRespondWithIndex = 0;
    var sensorId;
    var sensorValue;
    var wasClosed = false;

    client.on("data", function (data) {
        receivedDataCounter++;

        if (receivedDataCounter === 1) {
            sensor.firstReceivedFrame = data;
        }

        if (receivedDataCounter === 2) {
            sensor.secondReceivedFrame = data;
        }

        if (framesToRespondWith && framesToRespondWith[framesToRespondWithIndex]) {
            client.write(framesToRespondWith[framesToRespondWithIndex]);
            framesToRespondWithIndex++;
        }
    });

    client.on("close", function () {
        wasClosed = true;
    });

    function withServerHost(host) {
        serverHost = host;
        return sensor;
    }

    function withServerPort(port) {
        serverPort = port;
        return sensor;
    }

    function withFramesToRespondWith(frames) {
        framesToRespondWith = frames.slice();
        return sensor;
    }

    function isConnected() {
        return !wasClosed;
    }

    function connectAsync() {
        return new Promise(function (resolve, reject) {
            client.connect(serverPort, serverHost, function (err) {
                if (err) {
                    logger.error("Client error " + err.toString());
                    reject(err);
                }
                else {
                    identity = client.localAddress + ":" + client.localPort;
                    resolve();
                }
            });
        });
    }

    function destroyConnection() {
        client.destroy();
    }

    return sensor;
}

OK, the sensor is ready, so let's try to run our tests.

more end-to-end tests

What is interesting, is that accepting more than one connection and reconnecting after first connection is broken is working with the sensorAgent we already have. Looking at the code of the sensorAgent we can see, that this is actually possible. There is just an array keeping all connections, even those closed one. Not good. We should remember to remove all closed connections from the agent. And that fact should be definitely described by a test. Maybe creating that test should be an exercise for a reader of this article?

SensorConnection - step 1 - implementation

As we have added a lot of new tests, which are describing a lot of new features, I will try to introduce them gradually. One thing at a time! And the gradation is as follows: the sensorAgent should accept connection from a sensor; it should send them status frame; received data frame should be verified; correct frames should be sent to the webSink.

Connecting with sensors

At the beginning I am going to cover only basics like handling end, close and error events of the socket, closing the connection and sending confirmation frame which is necessary to pass first tests.

sensorConnection.js

logger = require("winston");
frameFactory = require("./frame");

module.exports.create = function (socket, connectionClosedHandler) {
    var sensorConnection = {
        close: close,
        sendConfirmationFrameAsync: sendConfirmationFrameAsync,
        getIdentity: getIdentity
    };

    var identity = socket.remoteAddress + ":" + socket.remotePort;

    socket.on("end", function () {
        logger.info("Connection end: " + identity);
        socket.end();
    });

    socket.on("close", function (data) {
        logger.info("Connection closed: " + identity);

        connectionClosedHandler(sensorConnection);
    });

    socket.on("error", function (err) {
        logger.error("Connection error " + identity + ": " + JSON.stringify(err));
    });

    function getIdentity() {
        return identity;
    }

    function close() {
        socket.end();
    }

    function sendConfirmationFrameAsync() {
        return new Promise(function (resolve, reject) {
            var confirmationFrame = frameFactory
                .createConfirmationFrame();

            socket.write(confirmationFrame.getBytes(), function (err) {
                if (err) { reject(err); }
                else { resolve(); }
            });
        });
    }

    return sensorConnection;
}

The sensorConnectionManager should also be modified to consume new features provided by the sensorConnection. Here is the changed fragment:

sensorConnectionManager.js

var connections = [];

function createConnection(socket) {
    var sensorConnection = sensorConnectionFactory.create(
        socket,
        connectionClosed
    );

    connections.push(sensorConnection);

    sensorConnection
        .sendConfirmationFrameAsync()
        .then(() => logger.info("Confirmation frame sent to sensor - " + sensorConnection.getIdentity()));
}

function closeConnections() {
    connections.forEach(
        sensorConnection => sensorConnection.close());
}

function connectionClosed(sensorConnection) {
    var index = connections.indexOf(sensorConnection);
    connections.splice(index, 1);
}

Now, the connections array holds sensorConnection objects instead of sockets. There is also function responsible for removing the connection after it is closed.

sensorConnection

Expected tests are green, so we can move on.

Sending STATUS frame to sensors

Our next task is to send status frame to sensors, so the sensorAgent could obtain fresh measurements. We need to add following function to the sensorConnection component and expose it.

sensorConnection.js

function sendStatusFrameAsync() {
    return new Promise(function (resolve, reject) {
        var statusFrame = frameFactory
            .createStatusFrame();

        socket.write(statusFrame.getBytes(), function (err) {
            if (err) { reject(err); }
            else { resolve(); }
        });
    });
}

Now, I will modify the sensorAgent and sensorConnectionManager. I will add .collectDataFromSensorsAsync() function, which was mentioned before.

sensorAgent.js

var webSinkFactory = require("./webSink");
var sensorConnectionManagerFactory = require("./sensorConnectionManager");

module.exports.create = function(config){
    var sensorAgent = {
        startAsync: startAsync,
        stopAsync: stopAsync,

        collectDataFromSensorsAsync:collectDataFromSensorsAsync,
        exposeData: exposeData   
    }

    var webSink = webSinkFactory.create(config);
    var sensorConnectionManager = sensorConnectionManagerFactory.create(config);

    function startAsync(){
        return webSink.startAsync()
            .then(() => sensorConnectionManager.startAsync());
    }

    function stopAsync(){
        return webSink.stopAsync()
            .then(() => sensorConnectionManager.stopAsync());
    }

    function collectDataFromSensorsAsync(){
        return sensorConnectionManager
            .collectDataFromSensorsAsync();
    }

    function exposeData(newData){
        webSink.exposeData(newData);
    }

    return sensorAgent;
}

sensorConnectionManager.js

function collectDataFromSensorsAsync() {
    var sendStatusPromises = [];

    connections.forEach(sensorConnection =>{
        sendStatusPromises.push(
            sensorConnection.sendStatusFrameAsync()); 
    });

    return Promise.all(sendStatusPromises);
}  

The function is quite simple. It creates promises sending status frames to each connected sensor, and waits until all of them are finished. Receiving data from sensors will be handled in the next subsection.

sensorAgent sends status

Handling wrong sensor frames

The status frame is sent, now we have to deal with a frame we will receive from the sensor in a response. Let's concentrate on wrong frames firstly. After the wrong frame is received, we would like the connection to be closed. Here is the code we need to add to the sensorConnection component.

sensorConnection.js

socket.on("data", function (data) {
    try {
        logger.debug(JSON.stringify(data));
        var receivedFrame = frameFactory.createFrameFromBytes(data);
        processFrame(receivedFrame);
    } catch (err) {
        logger.error(identity + ": " + err.toString());
        close();
    }
});

function processFrame(frame) {
    if(frame.isNack()){
        throw new Error("Sensor " + identity + " responded with NACK frame")
    }
}

The code represents a handler for data socket event. Received bytes are parsed to a frame, and then the frame is analyzed. In case of the nack frame, connection will be closed as well as in case of unknown frame (frameFactory will throw an exception if not able to parse a frame to any known type).

handling wrong sensor frames

There is only one more test ahead.

Collecting and exposing data from sensors

To collect received data we have to detect them in the sensorConnection and pass them somehow to the sensorAgent so it could expose them with the webSink. I have added dataReceivedHandler and expand a little the processFrame function.

sensorConnection.js

logger = require("winston");
frameFactory = require("./frame");

module.exports.create = function (socket, connectionClosedHandler, dataReceivedHandler) {
    var sensorConnection = {
        close: close,
        sendConfirmationFrameAsync: sendConfirmationFrameAsync,
        sendStatusFrameAsync: sendStatusFrameAsync,
        getIdentity: getIdentity
    };

    var identity = socket.remoteAddress + ":" + socket.remotePort;

    socket.on("data", function (data) {
        try {
            logger.debug(JSON.stringify(data));
            var receivedFrame = frameFactory.createFrameFromBytes(data);
            processFrame(receivedFrame);
        } catch (err) {
            logger.error(identity + ": " + err.toString());
            close();
        }
    });

    function processFrame(frame) {
        if (frame.isNack()) {
            throw new Error("Sensor " + identity + " responded with NACK frame")
        }
        else if (frame.isData()) {
            dataReceivedHandler({
                sensorId: frame.getSensorId(),
                value: frame.getSensorValue()
            });
        }
    }

    socket.on("end", function () {
        logger.info("Connection end: " + identity);
        socket.end();
    });

    socket.on("close", function (data) {
        logger.info("Connection closed: " + identity);

        connectionClosedHandler(sensorConnection);
    });

    socket.on("error", function (err) {
        logger.error("Connection error " + identity + ": " + JSON.stringify(err));
    });

    function getIdentity() {
        return identity;
    }

    function close() {
        socket.end();
    }

    function sendConfirmationFrameAsync() {
        return new Promise(function (resolve, reject) {
            var confirmationFrame = frameFactory
                .createConfirmationFrame();

            socket.write(confirmationFrame.getBytes(), function (err) {
                if (err) { reject(err); }
                else { resolve(); }
            });
        });
    }

    function sendStatusFrameAsync() {
        return new Promise(function (resolve, reject) {
            var statusFrame = frameFactory
                .createStatusFrame();

            socket.write(statusFrame.getBytes(), function (err) {
                if (err) { reject(err); }
                else { resolve(); }
            });
        });
    }

    return sensorConnection;
}

There are also some changes in the sensorConnectionManager component. There were dateReceivedHandler added too. Function dataReceived is being passed to each new sensorConnection and is responsbile for gathering current sensors state. Each time the state is changed we want to pass it as a whole, not separately for each sensor. To store the state there is the sensorsData array introduced.

sensorConnectionManager.js

var Promise = require("bluebird");
var net = require("net");
var logger = require("winston");
var frameFactory = require("./frame");
var sensorConnectionFactory = require("./sensorConnection");

module.exports.create = function (config, dateReceivedHandler) {
    var sensorConnectionManager = {
        startAsync: startAsync,
        stopAsync: stopAsync,
        collectDataFromSensorsAsync: collectDataFromSensorsAsync
    }

    var server;

    function startAsync() {
        return new Promise(function (resolve, reject) {
            server = net.createServer(createConnection);

            server.listen(config.sensorServerPort, config.sensorServerHost, function (err) {
                if (err) { reject(err); }
                else { resolve(); }
            });
        }).then(() => logger.info("Sensor server is started " + config.sensorServerHost + ":" + config.sensorServerPort));
    }

    function stopAsync() {
        return new Promise(function (resolve, reject) {
            closeConnections();

            server.close(function (err) {
                if (err) { reject(err); }
                else { resolve(); }
            });
        }).then(() => logger.info("Sensor server is stopped"));
    }

    function collectDataFromSensorsAsync() {
        var sendStatusPromises = [];

        connections.forEach(sensorConnection => {
            sendStatusPromises.push(
                sensorConnection.sendStatusFrameAsync());
        });

        return Promise.all(sendStatusPromises);
    }

    var connections = [];

    function createConnection(socket) {
        var sensorConnection = sensorConnectionFactory.create(
            socket,
            connectionClosed,
            dataReceived
        );

        connections.push(sensorConnection);

        sensorConnection
            .sendConfirmationFrameAsync()
            .then(() => logger.info("Confirmation frame sent to sensor - " + sensorConnection.getIdentity()));
    }

    function closeConnections() {
        connections.forEach(
            sensorConnection => sensorConnection.close());
    }

    function connectionClosed(sensorConnection) {
        var index = connections.indexOf(sensorConnection);
        connections.splice(index, 1);
    }

    var sensorsData = [];

    function dataReceived(sensorData) {
        sensorsData = sensorsData
            .filter(data => data.sensorId != sensorData.sensorId);

        sensorsData.push(sensorData);

        dateReceivedHandler(sensorsData);
    }

    return sensorConnectionManager;
}

The sensorAgent was slightly changed too. Currently we are passing the exposeData function, responsible for sending data to the webSink, to the sensorConnectionManager.

sensorAgent.js

var sensorConnectionManager = sensorConnectionManagerFactory.create(config, exposeData);

Finally all the tests are green.

collecting and exposing data

Cleaning up

At the very beginning of the development we have created test for the webSink itself. We needed exposeData function to be public. Now we can remove it.

exposeData is not public

The test of the webSink can be removed as well. This part of the system is tested with "sensorAgent should expose data acquired from the sensors" test.

test/end-to-end.js

it("sensorAgent should expose data acquired form sensors externally", function () {
    var sensorData = {
        sensorId: 1,
        sensorValue: 1000
    };

    return sensorAgent
        .startAsync()
        .then(() => sensorAgent.exposeData(sensorData))
        .then(() => webSinkRequest.executeAsync())
        .then(acquiredData => acquiredData.should.be.eql(sensorData));
});

When redundant public function and test are removed, system is still working correctly.

redundant test removed

Abstraction hunting - third attempt

We have written a lot of code until now. We should do quick code scan and check if there are some candidates for an encapsulation. We also shouldn't forget that our sensorAgent has to somehow collect fresh measurements periodically. So, the first thought is to introduce something like heartbeat - it will be responsible for calling collectDataFromSensorsAsync function with interval of - let's say - thirty seconds. The sensor's documentation we presently have doesn't specify the interval length, so we have to make an assumption.

I have found this fragment in the sensorConnectionManager. It is responsible for collecting fresh sensor's measurements, updating the array and sending new results to the sensorAgent.

sensorConnectionManager.js

var sensorsData = [];

function dataReceived(sensorData) {
    sensorsData = sensorsData
        .filter(data => data.sensorId != sensorData.sensorId);

    sensorsData.push(sensorData);

    dateReceivedHandler(sensorsData);
}

Quite a lot of things to do for one function. Let's introduce sensorDataCache to handle the measurements storing.

SensorDataCache - step 1 - tests

SensorDataCache will be rather small and simple, nevertheless we should start with unit tests. Let's prepare empty sensorDataCache component so we can require it inside test file.

sensorDataCache.js

module.exports.create = function(){
    var sensorDataCache = {};

    return sensorDataCache;
}

We have three simple test cases. First one assures that created cache is empty. Second one checks if new sensor's data can be added to the cache for the first time. Checking if sensor's measurements can be replaced with fresh one is a responsibility of the third test case.

test/sensorDataCacheTests.js

var should = require("should");
var sensorDataCacheFactory = require("../sensorDataCache");

describe("Caching sensors data", function () {
    it("Cache should be initialized as empty", function () {
        var sensorDataCache = sensorDataCacheFactory.create();
        sensorDataCache.getData().should.be.eql([]);
    });

    it("Cache should accept new sensors data", function () {
        var sensorDataCache = sensorDataCacheFactory.create();

        sensorDataCache.update({
            sensorId: 1,
            value: 10
        });

        sensorDataCache.update({
            sensorId: 2,
            value: 15
        });

        sensorDataCache.getData().should.be.eql([
            {
                sensorId: 1,
                value: 10
            },
            {
                sensorId: 2,
                value: 15
            }
        ]);
    });

    it("Cache should update stale sensor data with fresh one", function () {
        var sensorDataCache = sensorDataCacheFactory.create();

        sensorDataCache.update({
            sensorId: 1,
            value: 10
        });

        sensorDataCache.update({
            sensorId: 1,
            value: 15
        });

        sensorDataCache.getData().should.be.eql([
            {
                sensorId: 1,
                value: 15
            }
        ]);
    });
});

caching sensors data

SensorDataCache - step 2 - implementation

The implementation is quite simple. We have an array to store data, and each time the update function is called, stale data object is replaced with fresh one.

sensorDataCache.js

module.exports.create = function(){
    var sensorDataCache = {
        getData: getData,
        update: update
    };

    var data = [];

    function getData(){
        return data.slice();
    }

    function update(sensorData){
        data = data.filter(data => data.sensorId != sensorData.sensorId);
        data.push(sensorData);
    }

    return sensorDataCache;
}

And of course, we want to use new sensorDataCache in the sensorConnectionManager component. Now the presented code fragment is clean and easy to follow.

sensorConnectionManager.js

var sensorsDataCache = sensorDataCacheFactory.create();

function dataReceived(sensorData) {
    sensorsDataCache.update(sensorData);
    dateReceivedHandler(sensorsDataCache.getData());
}

Happily, introduction of the cache didn't spoil anything.

sensor data cache green

Heartbeat - step 1 - tests

It will be tricky to introduce the heartbeat mechanism to our system. It is always difficult to test components which behaviour depends on time flow. I will try to do as much as I can. Firstly, I have prepared empty heartbeat component.

heartbeat.js

var logger = require("winston");

module.exports.create = function () {
    var heartbeat = {};

    return heartbeat;
}

The idea how should I test the heartbeat is simple. If the interval of the heartbeat is set to 100 milliseconds I can presume that in the period of 500 milliseconds it will call the function five times. Of course, I need some error margin, because called function executes longer than 0 miliseconds.

`test/heartbeatTests.js'

var Promise = require("bluebird");
var should = require("should");
var hearbeatFactory = require("../heartbeat");

describe("Heartbeating", function () {
    it("Promise should be called exact number of times during specified period", function () {
        var counter = 0;

        var promise = function () {
            return new Promise(function (resolve, reject) {
                counter++;
                resolve();
            });
        };

        var heartbeat = hearbeatFactory.create();
        heartbeat.start(promise, 100);

        return Promise.delay(550)
            .then(() => heartbeat.stopAsync())
            .then(() => counter.should.be.eql(5));
    });
});

Is that test reliable? I am not sure, but I suppose it is still better than nothing.

heartbeat

Heartbeat - step 2 - implementation

The implementation is based on the snippet found here. I have modified it a little, so it could be used with promises easily.

heartbeat.js

var Promise = require("bluebird");
var logger = require("winston");

module.exports.create = function () {
    var heartbeat = {
        start: start,
        stopAsync: stopAsync
    }

    var shouldExecute = false;
    var task;

    function start(promise, delay) {
        shouldExecute = true;
        interval(promise, delay);
    }

    function stopAsync() {
        return new Promise(function (resolve, reject) {
            shouldExecute = false;
            clearTimeout(task);
            resolve();
        });
    }

    function interval(promise, wait) {
        var interv = function (w) {
            return function () {
                if (shouldExecute) {
                    logger.debug("Heartbeat will be executed.");
                    promise.call(null)
                        .then(() => logger.debug("Heartbeat was executed."))
                        .then(() => task = setTimeout(interv, w));
                }
            };
        } (wait);

        task = setTimeout(interv, wait);
    };

    return heartbeat;
}

To add the heartbeat to the sensorAgent I have simply created it and started with the collectDataFromSensorsAsync as the handler and thirty seconds as interval period. And of course, stopped in stopAsync function.

sensorAgent.js

var webSinkFactory = require("./webSink");
var sensorConnectionManagerFactory = require("./sensorConnectionManager");
var heartbeatFactory = require("./heartbeat");

module.exports.create = function (config) {
    const THIRTY_SECONDS = 30 * 1000;

    var sensorAgent = {
        startAsync: startAsync,
        stopAsync: stopAsync,

        collectDataFromSensorsAsync: collectDataFromSensorsAsync
    }

    var webSink = webSinkFactory.create(config);
    var sensorConnectionManager = sensorConnectionManagerFactory.create(config, exposeData);
    var heartbeat = heartbeatFactory.create();

    function startAsync() {
        return webSink.startAsync()
            .then(() => sensorConnectionManager.startAsync())
            .then(() => heartbeat.start(collectDataFromSensorsAsync, THIRTY_SECONDS))
    }

    function stopAsync() {
        return heartbeat.stopAsync()                 
            .then(() => sensorConnectionManager.stopAsync())
            .then(() => webSink.stopAsync());
    }

    function collectDataFromSensorsAsync() {
        return sensorConnectionManager
            .collectDataFromSensorsAsync();
    }

    function exposeData(newData) {
        webSink.exposeData(newData);
    }

    return sensorAgent;
}

hearbeat green

All the tests are green, but can we trust them now? The introduction of the heartbeat doesn't affected them because is starts working after thirty seconds of delay - all tests are finished by then. Could it affect the tests if started sooner? Probably yes - our sensor mocks are not prepared to handle more received frames than exactly expected.

I suppose we should be aware that no tests can guarantee us with 100% safety. Maybe we should introduce a separate test which will be running for a couple of minutes and checks if everything is OK? I am not sure.

The another issue I haven't mentioned yet is that collectDataFromSensorsAsync function of the sensorAgent is still exposed publicly. It is not used in the app.js as the sensorAgent is independent component and needs only to be started to work. But I'm not sure if I should remove the function from the public ones. It allows the tests to stay easy. I didn't have to introduce any heartbeatMock to trigger the sensorAgent with sensors communication process in more hidden and roundabout way. Moreover, the heartbeatMock wouldn't allow me to test time flow dependencies anyway. But I suppose the opinions about that case may be divided - I will gladly read about different (maybe more correct) approaches in the comments.

Summary

This article was the first part, supposed to present an approach to software development which can be time efficient and ready to face changing environment of real life project. The next one will cover different scenarios which may occur after sensors are finally delivered. This is of course my private way to deal with programmer's everyday life problems - and it is still evolving. I invite to discuss the topic in the comments.

The code of the presented application can be found here.

About the author:

Damian Krychowski

Damian is a software developer at BT Skyrise. He specializes in .NET technology, yet he likes challenges and willingly discovers other languages and platforms. Damian is a passionate engineer able to work on different levels of abstraction - ranging from embedded systems prototyping to cloud computing and architecture. Open source enthusiast.

Next Post Previous Post

blog comments powered by Disqus