Enabling bufferbot to do parallel deployments
At Buffer we do a lot of our deployments via Slack still. We are in the process of migrating to continuous deployments where merging a branch to master on Github (merging Pull requests) will trigger a deployment to production. But for some existing services like the main buffer monolith service itself, we still use a slackbot command:
@bufferbot deploy web to push code to our Elasticbeanstalk environment.
We have a few Elasticbeanstalk environments and one of them is for our API. We are also in the process of moving the API layer to Kubernetes where it will take advantage of our newer deployment process. Till it fully moves over however, I need to enable our bufferbot to deploy to kubernetes and Elasticbeanstalk in parallel when someone runs:
@bufferbot deploy api
Normally a microservice deployment via slack happens using:
@bufferbot servicedeploy <name-of-service>.
In this case the final result of this exercise will be as if someone typed in both commands at once.
heroku currently to host bufferbot's code since we use
hubot. Hubot is useful but we will be migrating away soon to
botkit. In the meantime though I have had to break out coffeescript again to do the changes which is simple enough. In pseudocode it looks like this:
//list for bufferbot deploy .... string //store params in stringParams targetedDeployment = stringParams //this could be web, api, or any of our other environments doStandardDeploymentStuff(targetedDeployment) if targetedDeployment == 'api' //code to call intermediate service responsible for doing the kubernetes build and deploy steps
July 19th 2018
I'm now running almost all my development on a windows machine. I use the
WSL layer to run things I would normally do on Ubuntu. So far it's been pretty solid. But today I'm running into troubles. Installing the heroku CLI is proving to be quite the challenge to the point that I might just use the provided windows installer to manage things for me. I'll also need to install git for windows.
12:36 While I download git, I'll just highlight the issue that I'm having installing heroku CLI in the Ubuntu layer in windows (henceforth referred to as WSL layer).
- Heroku's instructions for Ubuntu states to use
snapto install. You can check out everthing about snapcraft if you want to know more about that.
- But turns out that
snapneeds some services running in the background that need to be started by systemd which apparently does not sit well with WSL. Turns out that's quite a dealbreaker for me.
- So I'm downloading heroku now but I'm also going to download the binaries and add them to
/usr/local/bin and/usr/local/lib` to see if I can get things started.
12:47 Downloaded and installed on WSL using the binary download option. Time to login and get things setup.
- Done and dusted. So far so good. Now to setup my local repo to be connected to the heroku remote as well and preferably I don't bust the deployment over there.
12:53 Annd the git remote is ready. Now I need to actually test my code to make sure I'm not uploading busted code. The only thing is I'm not entirely sure how to test the code in this case. The problem being that I want to see if it actually reaches the production stuff correctly. So I might have to take a chance and do a deployment and test in production. YOLO.
12:59 I do love vs code. Not that vim couldn't do this but the ease of finding and installing a plugin to do coffeescript code checks and then actually catching a problem before deploying a test branch is awesome.
13:05 Deploying. Please work.
13:07 So far so good.
13:08 So I'm going to first deploy the
master branch to the version of the API running on kubernetes. I need to make sure that process is still working as intended. I'll be using
bufferbot servicedeploy for that. Any subsequent builds will still be based off the same git hash for now (unless someone merges something in). We have a nifty thing in the deployment pipeline where if something has been built already, it skips the building of the docker image step and jumps straight ahead to the step which does the deployment in kubernetes. More on that in another blog post I guess.
- Build was a success. On to testing the parallel version.
13:18 Noooo. Something is wrong. For whatever reason, the request is not going to the deployment.
13:42 After a short break we are back to it. I've added some logging.
- Oki. Found the error in the logs. Hubot has multiple ways to make http requests. You can pass the object that contains the message in it. That has a method called
message.http. Or you can pass the
robotobject in. The
robotobject is responsible for listening. That too has a
robot.httpmethod in it. When writing the code under the
ifcondition, I copy pasted code from the
servicedeploycommand stuff. That uses
robotto make the
httprequests. But I wasn't passing
robotinto the method.
The code looks like this:
module.exports = (robot) -> robot.respond /deploy\s*([.\_a-zA-Z0-9-/']+\s)(to\s+)?(.+)/i, (msg) -> branch = msg.match or null target = msg.match or null branch = branch.replace /^\s+|\s+$/g, "" startBuild(robot, msg, target, branch) //previously this line wasn't passing in robot robot.respond /deploy\s*(.+)/i, (msg) -> target = msg.match or null if !/.+\s.+/i.test(target) startBuild(robot, msg, target) //previously this line wasn't passing in robot
- These kinds of oddities are why we want to switch away to
13:48 One more missing reference error to fix. This wasn't somehow picked up by the linter :(
13:53 And the parallel deployments are working! But somehow deployments to Kubernetes aren't working. Scratches head. Something weird is going on. Will dig in.
- Taking another break here for a bit
14:20 Back and digging ingot why the deployment to Kubernetes isn't working
14:34 After looking up the status of the deployment using
helm status <name-of-release> I discovered that the load balancers aren't getting allocated. Looks like we may have hit our load balancer limit for the region. Darn.
14:36 Annnd we've done it! After deleting some test deployments that were sitting around using up LoadBalancers, the deployment went through succesfully. Need to test if subsequent updates will work as well.
- It does! We are done here! :D
Posted on November 19 2018 by Adnan Issadeen