Importance Of Using $elemMatch Operator, While Working With Arrays In MongoDB.

In MongoDB, its a common scenario, where we need to filter out the documents, based on the sub document/arrays, but there are couple of ways in which we can query the documents with arrays. It purely depends on the developer use case.

1. Using the dot operator. (like:  comments.rating).
2. Using the $elemMatch operator.

The main difference is that, they both behave differently when using with multiple conditions in filter operation. This blog is mainly to showcase that difference, and how they work when using it with array.

Lets understand this by example:

Suppose, we have a restaurant collection, with a document as follows: It has a comments array, with customer rating, and customer category (Bronze/Gold).

{
    "name": "Fly High",
    "restaurant_id" : "30075444"
    "street" : "Baner",
    "zipcode" : "10462"
    "comments" : [
        {
            "rating" : "3",
            "message":"Nice Ambience Only",
            "customer":  "Gold"
        },
        {
            "rating" : "3.5",
            "message":"Ok Food",
            "customer":"Bronze"
        },
        {
            "rating" : "5", 
            "message":"Nice Ambience & Food",
            "customer":  "Bronze"
        }, 
        {
            "rating" : "2",
            "message":"Bad Service",
            "customer":  "Gold"
        }
    ]
}

This is just the single restaurant, suppose if we have information for multiple restaurants (documents), and if we need to find out, all those restaurants, who have been rated greater than 4, by Gold customer.

Lets evaluate both the supported ways, for querying the document with arrays.

1. Using the (.) dot operator :

Query Statement  :  Find all restaurants, who have been rated greater than 4, by "gold" customer.

Query :  db.restaurant.find({"comments.rating":{$gt:4}, "comments.customer":"gold"})

Suppose, if we only have the above specified document in the collection, and we executed the above query, the result count should be 0, as we don;t have any document, where the gold customer has given rating greater than 4. But the inverse of this, we will get the above document in result.

As, we can see in the above document, only rating greater than 4 we have, is from "Bronze" customer, still we get the current document in the result.

So, how "." dot operator works in multiple condition is, it will check, whether any combination of these condition matches in the comments array, if it has found the match, it will return that document.

In the current query case :

1. It will check, whether we do have any rating greater that 4 ? (Yes, we do have from the bronze customer).
2. It will check, whether we have any "gold" customer comment ? (Yes, we do have multiple comments).

i.e combination of conditions are matched in the array, lets return that document.

Note: It is good, when we are doing "OR" matching, but for "AND" matching, it can return some unexpected results.

The alternate for this, using the "$elemMatch" operator, which will behave as expected.


2. Using the ($elemMatch) operator : 

Lets rewrite the same above query using the "$elemMatch" operator and validate the results.

Query Statement  :  Find all restaurants, who have been rated greater than 4, by "gold" customer.

Query :

db.restaurant.find({"comments":{$elemMatch:{"rating":{$gt:4},"customer":"gold"}}})

If we only have the above document in collection, and we execute the elemMatch query, it will return the result count as 0. As, no document exist, where "gold" customer has given the rating greater than 5, and that was the expected result.

Hope it will help you, in understanding the difference between, in both the operators. The dot operator still can be used, when we do have only single condition in query.

How To Define Custom Static & Dynamic Roles In LoopBack 3.

There are many use-cases or scenario's where we felt like, we need some custom roles in our application, apart from the provided built in ones ($authenticated/$owner/$everyone).

LoopBack 3 provides a flexibility,  where we can define custom roles, based on our application requirements. It has majorly divides the custom roles into two categories, (Static and Dynamic one's).

Generally a beginner, who have just started looking into what loopback is, struggles in finding, what will be the scenario, where we should use static role or dynamic one, how they are different, and from where should he start.

In this article, i will explain each of them, with common scenarios/examples.

Static Roles : Suppose you owe a restaurant, and it has both managers and waiter's, manager's role is mostly related to administration, and waiter's role is mainly to deal with the customer orders.

Manager role is also to decide, what should be the rate of any specific dish, at any particular season, and also to decide the menu list, based on the season and the availability. No waiter can modify the rates of it.

So, now we have a predefined role and its responsibilities, that, only a person with role manager can do the above mentioned things. So, this is pretty straight forward. We know that the manager is a user, but with some administration responsibilities. 

So, how will we assign the user with this manager responsibilities in loopback :

1.  We create a user in the user model.
2. We define a new role called "Manager" (Role Model).
3. We mapped the newly created user with the newly created role. (Role Mapping Model).

All of the above are predefined/built in models, we just need to create a entry in it. Now, we have defined the role, but not have defined its responsibilities. 

Suppose, we have our custom model's called (Rate List) and (Menu List), we can define acl's on it, that, only a principal type "Role", with a principal id "Manager" can perform update/create operations on it, and all other's can only read/view it.

Sample, acl's for it.

    {
      "accessType": "*",
      "principalType": "ROLE",
      "principalId": "$everyone",
      "permission": "DENY"
    },
    {
      "accessType": "READ",
      "principalType": "ROLE",
      "principalId": "$everyone",
      "permission": "ALLOW"
    },
    {
      "accessType": "EXECUTE",
      "principalType": "ROLE",
      "principalId": "manager",
      "permission": "ALLOW",
      "property": "create"
    },
    {
      "accessType": "WRITE",
      "principalType": "ROLE",
      "principalId": "manager",
      "permission": "ALLOW"
    }


So, static role is something, where you do not need to query multiple custom model's for checking, whether we should allow the request or not. It is pretty straight forward, you have the user and its mapping done.

It is generally used, for implementing a restriction at the high level.What i mean by that is, suppose in the IT park, where lot of organization have offices, now a person is only allowed in the campus, if he has the company id card. So, we will only do a high level checking here :

a. First that company should have a office in the campus.
b. Should be a valid employee of that company.

Now, what if he swipes a card on the entry gate of any restricted area. We need to check, if his card is still valid or expired, do he has the enough permission or not. To check this we will need to query our models, if this card id is still valid or not, whether he/she access to that gate/room or not. So, here comes the role of dynamic roles, you cannot directly check if it belongs to that role, and let him go. You need to take decision here, based on other responses, and then only allow/reject the request.

Lets discuss this in detail, with different example.

Dynamic Roles : Suppose you have a leave management system, where every employee of your company can submit his request, and only a manager of that employee/or either HR manager can approve/reject the request.

Following ?

So, here we have two models till now :

A. User model, consist of all employees, having a assigned designated manager.
B. Time-Off model, consist of all time offs.

Roles : HR Manager (static role), employee (static role) can create time off request. 

So, suppose employee "B" is a manager of  "C" & "D", and employee "A" is a manager of  "B".  i.e. C & D reports to B, and B reports to A, and "H" is the HR manager.

So, suppose, employee "C" has applied for a leave, and now we want only his manager or HR can approve his time off.

Static role, won't work in this case, as just by checking the "manager" role, we will not be able to identify whether he is actually his manager or not. We need to query user table, with the current user id, to find, whether he is actually the manager of the "C" or not, then only we can allow him to update/approve the request.

In overall, i mean to say that, every time we need to take decision dynamically, based on the requested context( C or B, or D) find out whether current user is his manager or not, or he/she is the HR manager, and then only allow the request to be executed. So, we can do this with dynamic roles.

Dynamic roles, can be defined by writing the role resolver, and then register it in the boot script.

Example : role-resolver.js (inside boot directory) sample code.

module.exports = function(app) {
    var Role = app.models.Role;
  
    Role.registerResolver('approver', function(role, context, cb) {
      if (context.modelName !== 'Timeoff') {
        return cb(null, false);
      }
  
      var managerId = context.accessToken.userId;
      if (!managerId) {
        return cb(null, false);
      }
  
      context.model.findById(context.modelId, function(err, timeoff) {
        if(err) return cb(err);
        if(!timeoff) return cb(new Error("Timeoff not found"));
  
        let User = app.models.User;
        User.count({
          userId: timeoff.empId,
          managerId: managerId
        }, function(err, count) {
          if (err) return cb(err);
  
          if(count > 0){
            return cb(null, true);
          }
  
          else{
            return cb(null, false);
          }
        });
      });
    });
  };

And then on the Timeoff model, we can apply the acl for the same.

 {
      "accessType": "WRITE",
      "principalType": "ROLE",
      "principalId": ["approver"],
      "permission": "ALLOW",
      "property": "approve"
    }

I have applied the acl on the remote method "approve". So, if we have directly applied on all the methods, of write, then only manager can update the item(change date/reason), not even the employee who has created it. So, that is the reason custom remote endpoint is created.

Also, we can create a relation of Timeoff model with the User (belongsTo) , then we can add one more acl to the Timeoff model. $owner for rest of the write methods, so only the user/employee who have created the request can only modify it. No, other employee should modify each other timeoff's request accidentally.

    {
      "accessType": "WRITE",
      "principalType": "ROLE",
      "principalId": "$owner",
      "permission": "ALLOW"
    }


The whole idea of this blog, to give understanding of static/dynamic roles in loopback 3, and, how we can use it in our application. Hope this helps!.

Adding Code Coverage To Your NodeJS App Using Istanbul.

Writing unit test cases is not enough, for maintaining the quality of the application. 

How will we make sure that each scenario has been covered ? Every branches and conditions have been covered in the unit testing ? What if new features or functionality have been added, are they covered in unit test cases ?

Code/Test coverage is the answer to all the questions.

It not only helps in maintaining the quality of the application, but also helps developer in getting deeper understanding of the code. Most of the times, we skip certain conditions and scenario’s assuming that our code will execute properly as intended, but it may happen that, based on the certain inputs it can behave differently.

What can we do, to avoid these conditions and make sure that our code is thoroughly tested. There are certain code coverage tools, that we can integrate with test script.

So, when test cases will execute, they will record the execution of our code files and will check, what code/function/blocks have not been executed during unit testing and will generate the coverage report.

We can also set the threshold/limits for the code coverage tool, that how much percentage of functions/lines/statements needs to be covered while unit test case has been executed, and below that limits we will fail the build.

Lets start by adding coverage to our library, we will be using “Istanbul”.

We will add it as a dev dependency to our library:

npm install -D istanbul

After adding this, we will add/modify the test script in package.json, to add coverage.

"scripts": {

          "test" : "istanbul cover -x '*.test.js' node_modules/mocha/bin/_mocha -- -R spec src/api.test.js"

     }

In the above test script, we have added a “istanbul cover -x ‘*.test.js’”, which specifies that to record code coverage on all files, except the files with the test.js extensions. After that we have provided the path to the mocha executable and specified that the reporter is “spec” which is the most commonly used reporter.

Mocha available reporters.

Till now we have added the code coverage recording to our library/project.

Lets run the test script: npm run test

We will get the output similar to the below one, which will show the coverage report:

Also, a new directory has been created, with the name coverage, and it will contain the icov report and the coverage.json file.

The Percentage can vary based on the test cases return, in my example I have covered all the scenarios under unit testing.

Note : Do add the coverage directory to the .gitignore file to avoid committing it to the source control.

Viewing the Coverage Report In Browser:

The generated coverage report can be viewed in the browser, navigate to the Icov-report directory under the coverage directory, Open the index.html file in the browser.

In the left panel of the report, it will show the no of times the function or statement has been executed. Also, it will show a red cross, before the function/statement that has not been executed.

Setting the threshold/limit for the code coverage:

The above output shows us the standard coverage report, But how can we validate that, whenever a new feature added or existing functionality modified, the unit testing must be done for them, to avoid any breaking of existing functionality.

We can set the threshold/limits for our coverage reporting and before pushing it to source control, we can validate whether the set threshold have been met or not, if not met we can restrict the code to be pushed into the source control.

Istanbul has an inbuilt module called check-coverage, which we can use for setting the limits for the different specs.

Let’s add another command in the script part of package.json

"scripts": {

"check-coverage":"istanbul check-coverage --statements -100 --branches - 100 --functions 100 --lines 100",

"test":"istanbul cover -x '*.test.js' node_modules/mocha/bin/_mocha -- -R spec src/api.test.js"

}

We have added a check-coverage command in the existing script part of package.json and have specified some limits percentage, that we expect. We can set the threshold limits as required.

Lets run the command, to test whether our coverage report is matching the threshold criteria.

npm run check-coverage

if everything works fine then the output will be similar to the above one.

Now, lets try by adding some dummy function in our code and re-generate the code coverage report.

Example function:

function dummy(){

console.log(“not in use”);

}

1. Re-generate coverage report: npm run test

We can see that, the percentage have been dropped from 100%.

2. Check coverage: npm run coverage

We can see the coverage validations failed as it didn’t meet the set threshold, that simply states that the newly added code has not been covered in unit testing. We can add the test case for the dummy function, and re run the coverage.

Now, we can add it to the git hooks to avoid committing the code, until the threshold has been met.

Adding Git Hooks:

We will use the ghooks npm module for adding the git hooks to our library.We will install it as a dev dependency.

npm install -D ghooks

Add the config script in the package.json file, and under config add a “ghooks” node, with the 

"pre-commit" sub node:

"config": {

     "ghooks": {

                    "pre-commit": "npm run test && npm run check-coverage"

     }

}

Once, it is done and now whenever we try to commit the code, it will initially run the test script, which in turn generates the coverage report and secondly it will check, whether our coverage thresholds has been met or not.

In this way, we can assure that, whatever code has been pushed is unit tested.

In the next article, we will integrate the  code coverage reporting service codecov.io , which will take reporting to the next level.

Note: Do add the coverage directory to the gitignore file to avoid committing it to the source control.

Link to my GitHub Repository