Let’s take a look at how Torque Framework changes the nature of our relationship with the local development environment—a critical link in the software supply chain.
The Current State of Affairs With Local Development Environments
In my experience, the challenges of maintaining local development environments are obscure, if not non-existent, for everyone in the organization except developers themselves. Even engineering managers react with skepticism to requests for improvement. Negotiations usually devolve into determining the minimum effort required to fix only the acute problem. As “non-roadmap” work, improving things like local environments is always a hard sell.
Then, at a certain point in an organization's growth—around 100-200 engineers—the organization will realize they need to form an Internal Development Platform Team. It may go by different names like Developer Acceleration Team, Horizontal Team, or Developer Tooling Team, but the goal is always the same: make the developer experience better and their operations smoother.
The KPIs for these teams focus on developer velocity, measured by stats like coding to production time, the number of production bugs or outages, the average lifetime of a code branch, or the time developers spend fixing the development environment. Spending on internal developer tool development can reach 10% of engineering budget.
Why does internal tool development go from ignored in small organizations to a huge cost center in large ones? The short answer is the fragmented nature of scripts and configuration files that developers use to operate all their deployments, including their local environments.
For the long answer, we’ll break it down into four main buckets.
New Developer Onboarding: Long and Error-Prone
Sitting down to work on a system for the first time is a slow process. We have to set up the local environment on our laptop before actual work can begin. Usually, we are greeted by long README files for each service and each codebase. The longer the README, the more chances there are for the setup to fail since all the tools have to be installed on the host OS. There are ways to simplify the process with virtual environments, but many times, these virtual environments are global, so the requirements might conflict with other projects.
If you’ve been there, you know. It’s a mess. And maybe no one on the team even knows how to get it right on your new laptop with that particular OS. "Yeah, you’re gonna have to Google it!" is actually a legitimate solution to the problem.
Making Changes To The Local Environment
Unless the system is super simple (one application service and a database), or there is a dedicated team to keep local environments up and running, every change is scary. Not only is it easy to break things that you push from your new local environment, but even if your changes work, those same changes might then break on other developer laptops. So we avoid changes as much as possible.
This is the reason why industry wisdom is to "stick to a monolith for as long as you can."
Inability to Experiment with New Technologies
Imagine your architect or manager gives you a task to add a new kind of service, say Kafka or RabbitMQ. There is a slight chance you got the task because you are an expert with Kafka or RabbitMQ, but it’s unlikely. Developers are not experienced in setting up technologies, period, and this is easily a multi-day task. Not only do you have to get it up and running on your laptop, but also make it easy for others to set up too. There’s installing proper libraries and configuring services that will use the new service, then documenting every single step. It’s a daunting task.
Yes, docker-compose helps, but because you are not an expert, there is a lot to figure out, including configuration details and the inevitable exceptions. And the need to document doesn’t go away. This all makes the cost of experimentation difficult to justify.
The Many Ways Environments Get Out of Sync
One example is cache. We rarely configure and use cache technologies in local environments to save time in setting them up. Also, for reasons I do not want to delve into right now, they slow down development. But all other deployments, including test and staging environments, would have a cache service set up.
Differing database versions or other platform services in production or other deployment environments, compared to the local environment, cause a host of other issues. For security or stability reasons, versions are constantly upgraded in production. Test and staging environments are upgraded only occasionally. The local environment is the last to keep up (if it ever does).
The only time we upgrade versions or set up a service that we skipped on the local environment (as in the case of the caching service) is when there is a production bug or outage. Sometimes we will not be able to reproduce the bug or understand the cause of the outage until we upgrade the version.
Maybe you’re like me, and in most cases with production issues, you just log in to the production environment directly and debug there. That always feels much easier and faster than updating a local environment to reproduce the bug. It’s ok. I won’t tell.
How Torque Framework Solves Local Environment
Things look quite different when Torque Framework is on the job.
Before reading further, you’ll need to be familiar with Torque Framework’s basic concepts, like components, links, Architecture Dependency Graph, and deployment providers. Check out this blog post or watch the video on our website. It's a four-minute read or watch. Of course, you can always go to the Torque Framework's documentation.
What Torque Can Do Today
For Torque Framework, the local development environment is just another deployment provider. For local development deployments, we have implemented the docker-compose provider. To get your system up and running in the development environment, all you have to do is add a deployment object with torque deployment create dev <list of docker-compose providers>. Then all you need to do is:
torque deployment build dev
torque deployment apply dev
(dev in this context is the name of the deployment object. You can name it any way you like, and you can have multiple deployments that use the docker-compose deployment provider with different configuration setups for different development needs).
All the services, including all platform services like databases and caches, are configured and started with the same version as they will be deployed in any other environment.
To experiment with new technology on your laptop, all you need to do are a couple of commands like:
torque component create <name_of_a_new_service> <component_type>
torque component link <name_of_a_new_service> <name_of_application_service> –type <link_type>
After this, next time you execute build/apply commands on your laptop, the new service will be up and running, and the application service which you want to develop will be already configured. It will have a correct library for the service driver code, it will have environment variables, and most probably an example code to kickstart the coding process.
Torque Framework CLI has a command to list all available components and providers to easily find the one you are looking for.
Of course, there will always be details to take care of. For example, you may want to use a development-optimized docker image for your application service to enable hot reload and similar capabilities that make the development experience smooth. Or, you might want to disable the cache service on the development environment. These actions are now performed on the system level through the deployment Architecture Dependency Graph, or the development deployment object rather than in the weeds of the local development environment scripts.
A Peek Into the Future
Torque Framework makes things like "proxy components, links, and deployment providers" possible. What is that about? In many cases, you want to run your development environment against services that are deployed in some remote environment. To make the configuration switch from one setup to another– from local to remote service– these deployment providers' only role is to provide a correct configuration for a remote service connection. Now you switch between modes of development in a few commands. No down-in-the-weeds configuring, reconfiguring, forgetting, or breaking. Just use a different deployment object for a different setup.
Conclusion
While Torque Framework does not magically provide and run your entire development environment, and you still need to maintain the dev (or however you name it) deployment object, its configuration and composition are elevated to a higher level. Developers can get out of the weeds and instead think about and maintain their local environments at the system level.
Torque handles the mundane work that gets in the way of the high-value work developers have to do. Faster developer onboarding with quick laptop setups, less breakage due to out-of-sync environments across the team, less downtime updating local environments, and low-risk experimentation with new technologies are just a few of the ways Torque Framework reduces the operational burden of local development environments.