Sunday, 16 July 2017

Debugging a Kubernetes Pod (Node.js Application)

Debugging a node.js application is very easy, if it is running locally. But when it is deployed on kubernetes, it requires a lot of effort.

Every time you have found some bug, you re-build your image , re-deploy your pod and again start debugging.

In this approach, we will attached a debugger to a running pod (node.js instance) in the kubernetes, and using chrome-dev tools, we will debug our application.

We have updated our instance image with the bash script, which will check whether to run application in a debug mode or a normal mode. The bash script will check for environment variable "DEBUG_MODE", whether it is defined or not, if not it will run the application in a normal mode.We will pass that environment variable with the deployment yaml/json file.

The main advantage of using a bash script is, if you have completed your debugging and now want to start a pod in a normal mode, you just remove the environment variable from the yaml and restart the pod, it will run in a normal manner. Which reduces our time of updating code and re-building image.

Let's start with the implementation :

1. Bash Script
2. Update Dockerfile.
3. Create Pod with the newly created image.
4. Port-Forward the pod.

1. Creating Bash Script : 

I am using node:alpine as a base image, it is pretty light weight. So, the terminal will be a /bin/ash instead of /bin/bash. So, do change the first line based on the base image you are using.

In this script, i am using a optional "DEBUG_FILE" variable, which allow us to provide a file path while debugging.

The script is pretty simple, i am just checking initially whether the "DEBUG_MODE" is defined or not (not checking any value), if it is defined then attaching a chrome-dev tools to it (node --debug-brk --inspect app.js).

Note: Update your startup file name, in place of app.js in the bash script.

#!/bin/ash
echo "
check-mode.sh script checks whether debugging is ON or not, while initiating a container.
It accepts two environment variables :
a. DEBUG_MODE (mandatory for debugging)
b. DEBUG_FILE (optional file path for debugging)
Example : docker run-it -e DEBUG_MODE=debug -e DEBUG_FILE=app.js 'imagename' /bin/ash
Example : kubectl --namespace=app-debug port-forward backend-0 9229:9229"

if [ -z "$DEBUG_MODE" ]
then
echo "DEBUG_MODE is not defined, initiating without debugging.."
node app.js
else
echo
echo "---- 1. Environemt Variable DEBUG_MODE is Defined -----"
echo "---- 2. Checking Environment Variable DEBUG_FILE is defined or not,
and also does the file exist at that path ? ----"

if [ ! -z "$DEBUG_FILE" ] && [ -f "$DEBUG_FILE" ]
then
echo "---- 3. Environment Variable DEBUG_FILE is defined and also File Exist ----"
echo
node --debug-brk --inspect $DEBUG_FILE
else
echo "----- 3. DEBUG_FILE or File Path doesn't exist ----"
echo "----- 4. Debugging the default entry point app.js ----"
echo
node --debug-brk --inspect app.js
fi
fi


2. Update a docker file : 

FROM node:6.10.3-alpine

ENV NODE_ENV=development app="/home/app"

RUN mkdir "/home/app"

WORKDIR "$app"

RUN npm install --production

COPY "app.js" "$app"

COPY "check-mode.sh" "$app"

EXPOSE 3000

RUN chmod +x $app/check-mode.sh

ENTRYPOINT  $app/check-mode.sh


3. Create Pod with the newly created image:

After the new image is successfully built using the above docker file, we can create a new pod on kubernetes with the newly create image. Also make sure to pass the "DEBUG_MODE" environement variable in the pod yaml/json. The value right now doesn't matter for the env variable as, in the script we are just checking whether it is defined or not.

After the pod is created, in the logs you can see that, it will log that the debugger is listening on some port, generally the default port is 9229, but it can varies also.

Here is the docker run output:

docker run -it -e DEBUG_MODE=debug -e DEBUG_FILE=app.js 30657b10fb02 /bin/ash
externally, i have passed the environment variable using the -e.

Here is the kuberentes pod output:

pod logs ouptut, shows that the debugger is running at 9229 port.



environment variable declared in the pod yaml/json.

Now, in the final step we will port-forward it to local using the kubectl command line, and will attach it to the chrome://inspect.


4. Port-Forward the pod :

To attached the running debugger to the local chrome://inspect we will require to port-forward it to local.

Using kubectl we can port-forward the running pod to the local.

Command : kubectl --namespace="your namespace name" port-forward "pod name" "debugger running port in a pod"

Example : kubectl --namespace=default port-forward testenv-0 9229:9229

Here is the output you will get after port-forwarding :


After successfully port-forwarding, we can open the chrome-dev tools, to start debugging :

a. Type chrome://inspect in your browser new tab.
b. In the remote target, you will see the startup file of your pod.



Now, after your debugging is completed, we can just remove the environment variable from the pod yaml/json and restart the pod. It will work as normal instance. 

This is only a one time investment, anytime you think of associating a debugger to a pod, just update the environment variable. You don't need to rebuild your code image and re-deploy.

Note : if you are ever facing issue for copying the bash script file while building docker image, just open the bash script file in sublime text editor and then go to view->line endings->unix and save your file again.