Why I left Gitpod

Why I left Gitpod

Geoffrey Huntley

So, why did I leave Gitpod? It's hard to pick a particular singular reason but that internal all-company letter which was sent weeks after a wonderful, yet extravagant offsite, really spooked me; like I have been homeless before and I'm sure as shit ain't going back there...

I found it somewhat strange that a 65ish person company has scaled up so many folks in HR

I guess it would be easy for leadership to paint me as disgruntled but the reality is that as a holder of common stock in the company I'm constructively concerned about strategy, leadership, burn-rate in this economic climate and the following...

The admin panel after two years is still public on the internet

When I first joined the company I became very uneasy that the administration interface at https://gitpod.io/admin is public on the internet and raised it internally, but almost two years later all employees still have access to download customers' source code and environment variables via personal GitHub accounts.

"feat(admin): download a tarball of users workspace data"
Download workspace · Issue #552 · gitpod-io/gitpod
If I can't start my workspace anymore (e.g. because something is broken) I cannot access the data inside the workspace anymore. There should be a download link on the dashboard that allows me t...

Back in March 2021, there was a security incident where access to the administration panel was deployed in production - wide-open-by-default - without authentication, and it was never disclosed to customers. With a cumulative total of $36 million raised in venture capital, and after this incident occurred, you would think Gitpod would have hired a single security engineer by now...

Customers come last

I could write an entire blog post on this, but something structurally needs to change within Gitpod so that the experience of customers comes first and so that all employees serve the customer...

There is little point expending energy on awareness to bring in people into the funnel when the bottom is haemorrhaging due to product quality and service availability issues.

Preaching DevX but not shipping it

An incredible amount of energy is expended on preaching to others that companies should care about "Developer Experience" but little effort in my time was expended on actually shipping developer experience internally.

‎DevX Pod on Apple Podcasts
‎Technology · 2022
the people who run this podcast are amazing, but the product organisation is not aligned with what is preached

At last count, 11 employees of Gitpod are full-time doing software development building Gitpod with Gitpod from the JAPAC region with 300ms pings to clusters in either Europe or North America. Fun question:

"how do those employees feel when they see talk about the importance of DevX but see no action on the #1 that hinders their personal DevX productivity?"
Gitpod Singapore (sg) region · Issue #5534 · gitpod-io/gitpod
edit: 👋 hey there, please refer to #7489 as the source of truth for supporting more regions than just EU and US. We have discussed internally regarding creating a new region a few times since the t...
issue has been closed without resolution
Asia-Pacific Datacenter. · Issue #4526 · gitpod-io/gitpod
Please plan for a datacenter in asia pacific as applications like vnc requires low latency, please consider singapore / japan / sydney
issue has been closed without resolution
Gitpod in Mumbai region · Issue #6139 · gitpod-io/gitpod
edit: 👋 hey there, please refer to #7489 as the source of truth for supporting more regions than just EU and US. The latency experienced by me in Rajkot, Gujarat, India is about 300ms. This leads t...
issue has been closed without resolution

Spamming OSS projects as a growth strategy under OKR

If you look at the careers page for the replacement role of myself, you’ll see a term.

Gitpodification

Gitpodification literally means spamming popular open source repositories with pull requests designed as billboards that advertise the virtues of Gitpod under OKR where employees are micromanaged in retrospect as to why a maintainer didn’t accept/merge the .gitpod.yml and Open in Gitpod in the readme of the project.

https://twitter.com/joasahacker/status/1563443194183892993/

All evidence that this activity has been occurring is public. I tried, again and again, to educate leadership that spamming open-source as a growth strategy is a super bad idea.

Bad leadership

the quiet part that I didn't ship err until now...
the art of giving a shit
I was asked recently on the topic of leadership. In short, I’m an avid fan of servant leadership - being selflessly 100% focused on helping folks within my team / being a janitor. What is Servant Leadership? - Greenleaf Center for Servant LeadershipGreenleaf Center for Servant Leadership Back circ…

Microsoft is strategically evicting Gitpod

Microsoft is back doing anti-competitive things again but this time in the Visual Studio Code ecosystem. Microsoft has fractured their developer ecosystem in a way that makes anyone who uses VSCode in their service offering effectively an advertisement for two similar Microsoft services (GitHub Codespaces and the soon-to-be re-launched Visual Studio Codespaces) that don't have the purposefully designed developer experience paper-cuts that companies that consume the MIT source code will experience.

I can't say that I blame Microsoft's approach here - salaries of engineers who specialise in language tooling are expensive, and by my napkin math, it will at least take circa $9mill/year in additional salaries for Gitpod to be able to compete against [GitHub|Visual Studio] Codespaces. As detailed in the blog post below, there will soon be no Microsoft-maintained language servers  (what people refer to as "VSCode") in the future that Gitpod will be able to use legally...

Visual Studio Code is designed to fracture
I used to think GitHub Codespaces would help popularise Gitpod but now realize it is the other way around. Gitpod is currently permitted to exist in the Visual Studio Code ecosystem to popularise GitHub Codespaces, and Microsoft can step in at any moment to create legal crises that strategically div…

Competing against Microsoft is never fun. The way Microsoft enforces their competitive advantage is by making something the default. Paul Stovell, Founder and CEO of Octopus Deploy witnessed this first-hand back in 2016.

Overnight it became the default. We were exhibiting at Build 2016 at the time much of this was announced, and I remember people coming to our booth asking "so why should we use you over the Microsoft thing?". The "Microsoft thing" was announced only 5 minutes prior!
Why we terminated our partnership with Microsoft - Re: Next decade of open source
Aaron Stanard published a blog post about The Next Decade of .NET Open Source. It’s a good summary of recent conversations and there’s a lot to agree with in the post. In particular, I agree strongly with this point: If Autofac is genuinely the better tool for your business’ case

Reproducible developer environments is a movement and not a unique feature of any one particular product  

Reproducible developer environments are indeed a sleeper technology that’s going to ramp up for a decade in usage until, one day, everyone will be “behind the times” if they’re not already using them, but I think the industry is heading in a different direction to the techniques used (ie. workspace-images and .gitpod.yml) and advocated by Gitpod.

.gitpod.yml
Explore the documentation to learn more about Gitpod.io and Gitpod Self-Hosted.

The problem with workspace-images is that customers do not get security updates unless a Gitpod employee remembers to update this magical line in each one of the Docker chunks.

https://github.com/gitpod-io/workspace-images/blob/main/chunks/lang-c/Dockerfile#L7
Update gitpod/workspace-full to update Let’s Encrypts root cert · Issue #528 · gitpod-io/workspace-images
The current version of gitpod/workspace-full contains an outdated Let's Encrypt root certificate. This causes the problem that pull and push are not possible with self hosted GitLab instances t...
for 3 months last year the letsencrypt SSL bundle was expired cause no-one is tasked to bump the magic number

The problem with .gitpod.yml is that it dictates a particular way to achieve reproducible developer environments that are locked into a singular vendor vs the competing devcontainer.json open-standard which Microsoft is shipping as the default product experience in Visual Studio Code and GitHub (and soon Visual Studio) Codespaces.

If there's one thing I've learned in my 40 years around this sun is that whatever Microsoft ships as the default is the thing that will win.

However I think both approaches are missing the bigger picture that the industry is looking past Docker and towards tools such as Nix (or Guix) as they provide reproducible developer environments whilst also providing tooling for that enables complete ownership of ones software supply chain (via source/binary substitution) and the generation of software bill of materials.

GitHub - mitchellh/nixos-config: My NixOS configurations.
My NixOS configurations. Contribute to mitchellh/nixos-config development by creating an account on GitHub.
"I like to use macOS as the host OS and NixOS within a VM as my primary development environment. I use the graphical applications on the host (browser, calendars, mail app, iMessage, etc.) but I do almost everything dev-related in the VM (editor, compilation, databases, etc.)" - Mitchell Hashimoto

It's no secret that I'm a fan of nix, the build tooling stole my heart four years ago after a research detour into academia and after 19 years brewing in academia, nix is going mainstream...

and by utilising tools such as Cachix and nix one is able to achieve Gitpod's feature set of pre-builds + reproducible environments in a vendor agnostic manner which is super cool, however...

In this economic climate, I don't think any product should dictate reproducible developer environments (or tooling such as nix) before value can be derived as doing so introduces way too much product adoption friction.

The key to winning the valuable enterprise market is to integrate in with how enterprises currently work, minimise the amount of people/process change required and thus de-risk the political costs for a products internal advocate.

Requiring people to add .gitpod.yml in each one of their git repositories is way too much of an ask. I was quite vocal about this topic internally during my time at 'pod because open-source maintainers kept pushing back at the prospect of "adding another yml".

the Gitpod workflow will soon be replicated

self-explanatory

Kubernetes is the wrong abstraction layer

A development environment on Gitpod happens within a Kubernetes pod. It's pretty neat, but upon reflection, I think Kubernetes is the wrong abstraction layer for Gitpod for the following reasons:

  • A product designed around Kubernetes constrains an offering to software developers who use containers. In this economic climate, enterprise companies want a singular tool that works for all software development scenarios - including windows desktop development, macOS mobile development and data science (ie access to beefy GPUs).
  • Kubernetes is complex. Enterprises do not have significant experience with Kubernetes and thus selling/supporting Gitpod as an on-prem product is going to be a painful and expensive cultural transformation that I don't want to be part of.
a snapshot of a conversation with a mentor with on-prem experience about topics of GTM
  • I think containers are not a secure enough environment isolation technique for a remote code execution as a service business aka running untrusted public workloads. Gitpod does some really cool things with Linux namespacing but it isn't enough (security-wise), and it comes at a price (product-wise / see below)
  • For almost two years it has not been possible to run Kubernetes natively on Gitpod due to the above. One of the key motivators for customers moving a development environment away from a local laptop and into the cloud is to be able to run cloud-native workflows and applications. And on Gitpod, you can't.
Epic: Run k3s natively in Gitpod · Issue #4889 · gitpod-io/gitpod
Right now, to run k3s in Gitpod the only viable option is to use emulation to create a VM as showed here https://github.com/gitpod-io/template-k3s I did some analysis of why, even if we are able to...

and that's about it. In all, I don't regret working at 'pod, and it is a company that is full of really nice people, but there are structural, strategic and leadership issues that need resolving before the product/company will really shine.  

Thanks for reading btw.