Ship and Scale – The Empowering Fooda Tech Stack

Throughout my tenure at Fooda, we have built a rich and dynamic technology team. We’re about 30 people right now and, among other interesting things about our team, we do not have a dedicated DevOps engineer to manage our hardware. Don’t mistake our team structure as ignorance or lack of findings. Rather, we’ve built a stable DevOps foundation with a handful of battle-tested tools that allows us to better focus our energy. At the top of our toolchain is Kubernetes – a DevOps platform built with developers in mind. It leverages abstractions and an API that can easily be expanded on and it’s great at managing, deploying and scaling software applications. Now, the entire Fooda technology team can share the responsibilities of our environments. Our engineers don’t just build software, they build features; they build applications; they add intrinsic value to the technology stack and the company. This is the story of the system that runs our applications and empowers our team to build.

In the beginning, there was rails new. We had a single application run across a production and staging environment. We pushed to Heroku. We adapted GitHub Flow and a culture of deploying code frequently. We were a small technology team but, even from the beginning, we tried to structure the organization so that the team could write, test, and deploy code quickly. Shipped code was (and still is) one of biggest performance indicators.

As the company grew, our technology grew with it, and we wanted more flexibility than Heroku could offer at the time. The executive decision was made to move into the world of AWS.

We dockerized our application services. Containers were baked on vanilla EC2 machines with an AWS AMI, orchestrated via Ansible and deployed through Asgard. Fooda technology ran and scaled on this platform for years. It was big, robust, complicated and expensive. It took 2 hours to go from a git tag to a production release and it was hard to introduce new engineers to our hardware stack. We knew it would have to change as the team continued to grow.

Then we saw the light. Fooda’s Principal Engineer kicked open the door with tales of Kubernetes and the new world you could build with it. We started, as all projects now start at Fooda, with a spike. We converted one of our staging environment to a Kubernetes environment and the team was sold quickly after that. An executive decision was made and we moved our production cluster, all of our staging environments, and our new tooling services into Kube. It took about 6 months as a part time project for a handful of engineers and we have be elated ever since.

Let’s talk about the unicorn called Kubernetes

If you’re like us, you started your project as a Monolithic MVC application. Hopefully, you built an API to supplement the original features and maybe you split your application into a cornucopia of microservices, web clients and iphone apps. Honestly, it doesn’t really matter how mature your technology stack is because Kubernetes can handle it all with the readiness to expand.

In the simplest abstraction, Kubernetes will take the docker container you just built (they call them pods), and run it. Lots of DevOps frameworks will do that, so let’s keep going.

Let’s say you want to run an API service like us. You’ve got a load balancer and a cluster of web servers. Kubernetes can do that out of the box. It can scale your application and keep the containers running if one happens to fail.

You’ll probably want to deploy that web service with zero down time, and migrate databases and Kubernetes has a solution for that too. The Kubernetes configuration details how the cluster should handle deployment strategy and the Kubernetes Job feature is the perfect tool for one off tasks like migrations or reindexes. It’s all written in a few simple config files and can be stored in the same repository as your application to keep things tidy.

Running containers and data manipulations is the foundation of every DevOps structure though. When I reflect on the best benefits that Kubernetes has provided Fooda, scale stands out as the dominant force.

For Kubernetes, this comes in several flavors. There’s the ease of scaling a pod’s resources vertically and horizontal. You can give a process more memory* or up the parallelism with a config change.

(*If you’re managing your cluster resources, you need to manage the node resources too. That is a blog post in itself and may sound daunting but it’s really not once you understand it. GCP runs kubernetes clusters for you and can remove a lot of this complexity.)

There’s the ease of duplicating or scaling an application within your cluster. For example, each of our applications has its own Kubernetes deployment, replica set, and pods that are managed through the same Kubernetes interface. One application may run a handful of RabbitMQ workers while another set of applications may act as an API. For our API specifically, we decided to duplicate the application to ensure different SLAs for different clients. One Kubernetes deployment is for public consumption, one is for back-office administration, and the other is for our point of sale. The applications are running the same source code and start the same web servers, but are named differently so Kubernetes can identify them. Because they are independent, we can monitor and scale our services independently too. Just copy your configs, tweak them for the new service, and deploy to your cluster.

At a very meta level of scale, you can duplicate entire technology stacks within your Kubernetes cluster. As an example, we run about 10 staging environments in the same Kubernetes cluster with the Kubernetes namespace feature.

A namespace is a way to group resources like your pods and deployments within a cluster. Out of the box, you’ll probably get the default, kube-system, and kube-public namespaces. The kube-system runs services like the DNS, auto-scaler and the dashboard while you’re default namespace will hold all of your applications. If you’re running a Kubernetes cluster that doesn’t need to be virtually split into many clusters, like Fooda’s production cluster, the default namespace is a great place for your containers. However, to run multiple staging environments on a single Kubernetes cluster, we create a new namespace (that’s usually named after someone’s favorite food like ‘burger’) and duplicate the deployments, replica sets, and pods under the new namespace. Each of our staging namespaces has about 10 deployments within it. The have separate configs and environment variables, and the staging Postgres instance has multiple databases within it. Because you still have the default namespace, you’re not forced into duplicate every service. For example, our logging service, FluentD, does not need to be duplicated for every staging environment, so we keep that container in the default namespace for all the stagings to use.

Above being easy, this keeps our dev/prod parity tight and our deploy confidence high.

Scale comes in many flavors with Kubernetes and because each level of scale is a fundamental part of the framework, you can leverage the Kubernetes tooling throughout every level of scale.

Enough of this hand waving – Show me how you do this magic

A bunch of software engineers at Fooda write ruby code and some of that code runs Rails. As I mentioned, we wrap our code in a simple Docker container with a Dockerfile that looks kind of like this:

Our Dockerfiles are very generic. Instead of using them to run the specific services (like a RabbitMQ worker), we use them to setup the environments. The final command for every Dockerfile we have is always a call to a Foreman Procfile that decrypts secrets, starts a logging service, and a simple web server. This way we know that every service in our cluster, regardless of it’s responsibility, has these basic tools.

That Procfile that gets called usually looks something like this:

And that’s your environment. Keeping these services simple and their responsibilities separate is key to having a solid foundation. This way you know where to add things when you expand, and when there’s a bug, it’s simpler to diagnose.

Now that we have a common interface for our Kubernetes services to connect to, let’s talk about how we define each service.

As an example, a RabbitMQ staging environment service gets broken down with two files. One to define the service and one to deploy it. Here’s the service config:

From that service, there are several deployments because we want to run several RabbitMQ workers with their own tasks. A deployment to match looks like this:

This deployment config defines the horizontal and vertical scale of the pods and also defines another Procfile that runs when the container starts. This Procfile is where you get specific about the service you want running. It usually looks similar to the Procfile that we started earlier. For this RabbitMQ worker example, it looks like this:

With that, you’re service is defined. You can deploy with the kubectl command line tool to start your new service.

With services defined like this, the complexities of shipping code has been reduced to a few simple keyboard commands at Fooda. Anyone on the team (including people that don’t write code) can go from a git SHA, to a built docker container, to a deployed environment in 15 minutes or less.

Kubernetes has even helped us speed up our deployment pipeline! We were initially using docker.io to build our containers, and we noticed that it could take upwards of 20 minutes to build the container before we shipped it to an environment. To streamline the entire process, our Principal Engineer built a deploybot application in Kubernetes. This simple rails app lives in our tooling cluster and builds the container from source code with a cache before deploying the new container to an environment of our choosing. Like most things in our world, it is controlled with an API and gets run with a simple CLI script but it also hooks into a Slack room for ease. Similar to GitHub’s hubot, it removes the complexities of deployment for people focused on building applications and speeds up our development pipeline where we need it.

With all this power at our fingertips, we don’t talk deploys anymore; we talk about writing better code.

Conclusion

The story of our hardware is one filled with change and strife. But today, we are more confident than ever in our frameworks and tooling. We can see how it will scale to meet the demand of the business and the expansion of our technology stack. We have built our own tools around the shortcomings that we’ve seen. And most importantly, we understand how to leverage these tools to empower our team to ship code.

Author

David Bremner is the Lead Software Engineer at Fooda. As one of the first software engineers, he helped start the Fooda technology stack and still loves to hack on it today. Before that, his entrepreneurial spirit guided him through Chicago’s startup scene and the halls of CS@UIUC. He enjoys cookies.