Engineering at Bunnyshell: A Journey to Cloud-Based Remote Development Environments

As we navigated the challenges of traditional development environments, we recognized the need for a more efficient, collaborative, and resource-friendly solution. This led us to explore the world of remote development.

Remote development, in its simplest form, is the practice of developing software in a cloud-based environment rather than on a local machine. This approach offers several advantages over traditional development methods, including improved collaboration, resource efficiency, and scalability.

Cloud Development Workspaces or Remote Development Workspaces are typically a virtual machine or a container where your developer tools, together with your application’s components are running as a container. Remote workspaces are easier to set up, can spin up faster, and are typically used for simpler and smaller applications. Examples of platforms in this category are Gitpod and GitHub Codespaces.

Cloud Development Environments (CDEs) or Remote Development Environments are production-like deployments of your application, typically in Kubernetes, and can also be connected to cloud-native services via IaC modules. Remote Dev Environments are more complex to set up, and they’re better tailored for applications that require more complex setups with 10+ components and services. Examples of platforms for remote dev environments are Bunnyshell and Release, among others.

The major difference between remote development environments and remote workspaces is the fact that one resembles the production environment, while the other simply moves the development runtime from your local laptop to a remote instance.

Remote IDE involve running your entire development tool stack in the cloud, typically together with your development environment or workspace and you as a developer are interacting only web-browser-based.

In this article, we will focus on Remote Development Environments, specifically Kubernetes-based ones, and share our journey to adopting this approach at Bunnyshell.

We'll delve into our experiences, the challenges we faced, the solutions we found, and the benefits we've reaped from this transition. We hope that our journey will provide valuable insights for other teams considering a similar shift.

Finally, before diving into Bunnyshell’s case study with remote development, let’s briefly go through the setup options one can have when developing using Kubernetes, as described in the project’s documentation.

Pure Off-line happens when you run both the cluster and the development environment locally on your laptop. Distribution like Minikube or Kind are great for working with Kubernetes on your local machine as they’re simpler and lightweight than K8s.

A Proxied setup means you’re developing against a remote cluster and forwarding traffic from and into the cluster for the components you’re developing on. The most used open-source tool for this type of setup is Telepresence.

A Live setup, as opposed to a Proxied one, involves syncing the code from your local machine into the remote cluster for the component/s you’re developing on. This is the model Bunnyshell uses for its remote development feature and works with hot-reload or hot-replace, depending on the programming language you use.

Similar to a Live setup, the Pure On-line also moves the code editor (IDE) as well as the application code into the Kubernetes cluster for a complete remote development experience. This capability doesn’t come out-of-the-box with Bunnyshell, however, you can automate an additional container in your environment to start with your pre-configured IDE and then access it as an endpoint in your browser.

If you're new to Kubernetes, it's important to understand that each setup option has its own strengths and weaknesses. The best option for you will depend on your specific needs and circumstances. For example, if you're working on a small project or just getting started with Kubernetes, a pure off-line setup might be the best option. But if you're working on a larger project or need to collaborate with a team, a proxied or live setup might be more suitable.

To illustrate this transition, we'll focus on the experiences of Aris and Georgian, our senior Backend and Frontend developers, respectively. They were the trailblazers for remote development within Bunnyshell’s R&D team. In this section, we'll delve into their individual journeys transitioning to remote development, providing both a backend and front-end developer perspective.

Bunnyshell's move to remote development was a strategic response to several critical challenges that were affecting our team's efficiency and productivity. These challenges, listed in order of importance, are illustrated with examples from our developers' experiences:

1. Resource-Intensive Local Environment Setup: The traditional development environment setup was resource-intensive, which often led to performance issues on developers' machines. For instance, Aris mentioned that running multiple services simultaneously was a real problem, especially when working with multiple projects. His laptop's resources were always used up by tools and services. Specifically, he mentioned that they had about 15 services running at the same time, many workers, and RAM was always used up by tools like PhpStorm, Visual Studio Code, and IntelliJ.

2. Time-Consuming Setup and Task Switching: Setting up the development environment for a new project or switching tasks was not straightforward. It was a time-consuming process that often led to delays and inefficiencies in the workflow. Additionally, Georgian for example didn’t have the know-how required to set up the backend locally to test his frontend code.

3. Inconsistencies Across the Team: With each developer setting up their own local environment, there were often inconsistencies across the team. This made collaboration and troubleshooting more difficult. Aris pointed out that when an issue was fixed by someone, it wasn't necessarily shared with others, leading to inconsistencies and repetitive work across the team.

4. Fear of System Updates: Developers were hesitant to update their system due to the fear of disrupting their local development environments. This fear often prevented them from taking advantage of the latest software updates and improvements. Georgian mentioned that he was afraid to upgrade anything, including his operating system, not to break his dev setup.

5. Challenges in Sharing Work in Progress: The traditional development environment made it difficult for developers to share their work with stakeholders. This hindered collaboration and made it harder to get timely feedback. Georgian used to do a Google Meet to show the changes and get feedback. Now, he just sends a link to his live app deployment.

Recognizing these issues, Bunnyshell decided to adopt remote development internally. The goal was to create a more efficient and collaborative development environment that would not only address these challenges but also leverage the benefits of cloud technologies.

Aris’s local environment consisted of a multitude of services and tools, including PhpStorm, Visual Studio Code, and IntelliJ and he was running around 15 services at once. The Bunnyshell platform included 5-10 microservices, a couple of database services and a monolith with 70-80 workers (that the team planned to split further into microservices over time). Each worker, even when idle, consumed over 100MB of RAM. This high resource consumption was a significant challenge, given the number of services running simultaneously.

The process of setting up this local environment was far from smooth. It often took 1-2 days to set up for a senior developer. This setup process involved configuring Docker Desktop and Docker Compose, as well as project specific configurations for packages and dependencies as well as make sure they worked well with one another.

Once set up, running the Bunnyshell application locally was not a complex process, it involved running a docker-compose up command and a shell script once, to set up additional configs and data.

The resource-intensive nature of the local setup posed significant challenges. To mitigate this, developers had to limit the number of workers running locally. However, this workaround often meant they couldn't catch concurrency problems effectively.

For Aris, one of the main benefits of remote development was improved resource efficiency. In a remote development setup, the development environment is hosted on a remote Kubernetes cluster, isolated by a namespace, rather than the local machine. This means that the resource consumption is offloaded to the cloud servers, freeing up the majority of his local machine's resources.

Aris’s computer now runs only the IDE and terminal sessions, keeping his resource consumption at a minimum. The setup process was streamlined using custom environment templates, and the time spent deploying a new environment was reduced to the click of a button or a CLI command run.

The adoption of remote development also brought new tools and features that improved Aris's workflow. Port forwarding, a technique used to allow external devices to access services on private networks, became an integral part of his workflow. This allowed Aris to connect to the database running in the remote development environment directly.

Environment cloning, a feature that allows developers to create a copy of their development environment, also became a part of his workflow. This feature is particularly useful when developers want to test new features or changes without affecting their main development environment.

Before the adoption of remote development, Georgian, a senior frontend developer at Bunnyshell, had a local setup that was based on Docker Desktop, Terminal for running Docker Compose, and Sequel Ace for database management. However, this setup was not without its challenges. For instance, any update to the system or the development environment could potentially disrupt his workflow. This fear of disruption led to a reluctance to update the MacOS or even restart the laptop, which could result in the local development environment not working.

One of the significant challenges Georgian faced as a frontend developer was setting up and running the backend. He was dependent on a backend developer for this task, which often resulted in a couple hours of pair debugging in case something broke.

Additionally, running the local development environment consumed a significant amount of battery life, limiting the duration of his work sessions.

Collaboration was also a challenge in the traditional development workflow. To showcase changes and receive feedback, Georgian had to conduct a Google Meet session. This process was not ideal and often hindered collaboration and needed synchronizing both parties.

The local setup was replaced with a more efficient and less demanding remote setup. The new workflow involved using a configuration file (config.yaml) to manage his remote development profile with token, organization, project, environments, and components. Starting the remote development environment was as simple as running a single CLI command (bns rdev up), which automatically used the profile from the configuration file.

The adoption of remote development also brought new tools and features that improved Georgian's workflow. One of the most beneficial features was the ability to share his development environment or a live app link with others. This feature simplified the process of showcasing changes and receiving feedback. Instead of conducting a Google Meet session, he could simply send a live work-in-progress app.

Despite facing challenges like an unstable internet connection, Georgian found that his productivity levels were improved after the switch to remote development. His laptop no longer overheated, and he could work 6 hours instead of 2 without having to charge his laptop, which made it easier to travel away from his desk.

Georgian's advice to other development teams considering a switch to remote development is to embrace the change. He believes that remote development simplifies the maintenance of the development environment, reduces stress, and provides more convenience. However, he also acknowledges that setting up a remote development environment as a frontend developer can be a blocker without help from Backend/DevOps engineer or clear instructions. The switch to remote development has allowed him to become more independent and less reliant on backend or DevOps personnel for setting up his local environment.

The transition to remote development was a team effort. We didn't have a dedicated Platform or DevOps team, so our developers took the initiative to implement the custom environment template that everyone uses to spin up Bunnyshell’s entire platform on Bunnyshell Environment as a Service. This collaborative approach not only made the transition smoother, but also fostered a sense of ownership and responsibility among our developers.

If you're new to hot-reloading on Kubernetes, it's important to understand that it's not just about reloading the app automatically with each source code change. It's about creating a fast and efficient feedback loop that allows you to see the effects of your changes in real-time. This can significantly speed up your development process and make it easier to catch and fix errors early on.

One of the main benefits of remote development for the developer experience is the centralized development environment template. This template is shared among all developers, allowing everyone to benefit from any changes or improvements. For example, if a developer finds a more efficient way to configure a certain tool or service, they can update the template and everyone else can benefit from this improvement. This not only improves the overall developer experience, but also fosters a culture of collaboration and continuous improvement.

Edit once, use all the time.

Technically speaking, the remote development environment deployment is immutable in a sense, making it a predictable experience for the developers.

Shifting an entire team's development to the cloud can be a daunting prospect for engineering leaders, primarily due to concerns about cloud costs. Since adopting remote development, our team at Bunnyshell has been actively tracking and optimizing our cloud expenses.

A significant strategy we've implemented is the introduction of availability rules for our environments. These rules ensure our environments automatically shut down at 6PM and remain inactive over the weekends. This approach reduces our usage to about 8 hours a day, a 70% decrease compared to continuous operation.

The average cost we incur per environment, per developer, per month, is between $60-$80. We find this cost quite reasonable, especially when compared to the expenses associated with leasing high-end laptops. Furthermore, the advantages of cloud development, such as a hassle-free setup, improved developer experience, and increased autonomy, make this a worthwhile investment.

As we continue to use and refine our remote development process, we're constantly looking for ways to improve and overcome new challenges. One of the challenges we're currently working on is improving the resilience of our remote development sessions to internet connection issues. We're exploring different solutions, such as automatic session recovery, to make our remote development process even more robust and reliable.

Our journey to cloud-based remote development at Bunnyshell has been transformative, not just for our workflow efficiency but also for our team's collaboration and overall developer experience. We've seen firsthand how this approach can address common challenges in traditional development environments, from resource-intensive local setups to inconsistencies across the team.

We've shared our experiences, the challenges we faced, and the solutions we found, all with the hope of providing valuable insights for other teams considering a similar shift. However, we believe that the best way to truly understand the benefits of remote development is to experience it for yourself.

If you're interested in seeing our platform in action, register for a free trial or reach out to us to book a demo.