Why I left Gitpod

Why I left Gitpod

Geoffrey Huntley

So, why did I leave Gitpod? It's hard to pick a particular singular reason but that internal all-company cost-cutting letter which was sent weeks after a wonderful, yet extravagant offsite, really spooked me; I guess it would be easy for leadership to paint me as disgruntled, but the reality is that as a holder of common stock in the company I'm constructively concerned about strategy, leadership, burn-rate in this economic climate and the following...

The admin panel after two years is still public on the internet

When I first joined the company, I became very uneasy that the administration interface at https://gitpod.io/admin is public on the internet and raised the problem internally. However, almost two years later all employees still have access to download customers' source code and environment variables via personal GitHub accounts.

"feat(admin): download a tarball of users workspace data"
Download workspace · Issue #552 · gitpod-io/gitpod
If I can't start my workspace anymore (e.g. because something is broken) I cannot access the data inside the workspace anymore. There should be a download link on the dashboard that allows me t...

Back in March 2021, there was a security incident where access to the administration panel was deployed in production - wide-open-by-default - without authentication, and it was never disclosed to customers. With a cumulative total of $36 million raised in venture capital, and after this incident occurred, you would think Gitpod would have hired a single security engineer by now...

Customers come last

I could write an entire blog post on this, but something structurally needs to change within Gitpod so that customers' experience comes first and that all employees serve the customer...

'pod only replied to this tweet after the publication of this blog post that highlighted Ryan's data loss woes. 

There is little point in expending energy on awareness to bring people into the funnel when the bottom is haemorrhaging due to product quality and service availability issues.

Preaching DevX but not shipping it

An incredible amount of energy is expended on preaching to others that companies should care about "Developer Experience" but little effort in my time was expended on actually shipping developer experience features internally.

‎DevX Pod on Apple Podcasts
‎Technology · 2022
the people who run this podcast are amazing, but the product organisation is not aligned with what is preached

At last count, 11 employees of Gitpod are full-time doing software development building Gitpod with Gitpod from the JAPAC region with 300ms pings to clusters in either Europe or North America. Fun question:

"how do those employees feel when they see talk about the importance of DevX but see no action on the #1 that hinders their personal DevX productivity?"
Gitpod Singapore (sg) region · Issue #5534 · gitpod-io/gitpod
edit: 👋 hey there, please refer to #7489 as the source of truth for supporting more regions than just EU and US. We have discussed internally regarding creating a new region a few times since the t...
issue has been closed without resolution
Asia-Pacific Datacenter. · Issue #4526 · gitpod-io/gitpod
Please plan for a datacenter in asia pacific as applications like vnc requires low latency, please consider singapore / japan / sydney
issue has been closed without resolution
Gitpod in Mumbai region · Issue #6139 · gitpod-io/gitpod
edit: 👋 hey there, please refer to #7489 as the source of truth for supporting more regions than just EU and US. The latency experienced by me in Rajkot, Gujarat, India is about 300ms. This leads t...
issue has been closed without resolution

Spamming OSS projects as a growth strategy under OKR

If you look at the careers page for the replacement role of myself, you’ll see a term.

Gitpodification

Gitpodification means spamming popular open source repositories with pull requests designed as billboards that advertise the virtues of Gitpod under OKR where employees are micromanaged in retrospect as to why a maintainer didn’t accept/merge the .gitpod.yml and Open in Gitpod in the readme of the project.

https://twitter.com/joasahacker/status/1563443194183892993/

All evidence that this activity has been occurring is public. I tried, again and again, to educate leadership that spamming open-source as a growth strategy is a super bad idea. Gitpod has tried Gitpodification once before, and when feedback from the open-source community that this activity was harming the brand was relayed back to my 1-up my concerns were dismissed and told to "just do it anyway using your personal GitHub account".

Bad leadership

the quiet part that I didn't ship err until now...
the art of giving a shit
I was asked recently on the topic of leadership. In short, I’m an avid fan of servant leadership - being selflessly 100% focused on helping folks within my team / being a janitor. What is Servant Leadership? - Greenleaf Center for Servant LeadershipGreenleaf Center for Servant Leadership Back circ…

Microsoft is strategically evicting Gitpod

Microsoft is back doing anti-competitive things again but this time in the Visual Studio Code ecosystem. Microsoft has fractured their developer ecosystem in a way that makes anyone who uses VSCode in their service offering effectively an advertisement for two similar Microsoft services (GitHub Codespaces and the soon-to-be re-launched Visual Studio Codespaces) that don't have the purposefully designed developer experience paper-cuts that companies that consume the MIT source code will experience.

I can't say that I blame Microsoft's approach here - salaries of engineers who specialise in language tooling are expensive, and by my napkin math, it will at least take circa $9mill/year in additional salaries for Gitpod to be able to compete against [GitHub|Visual Studio] Codespaces. As detailed in the blog post below, there will soon be no Microsoft-maintained language servers  (what people refer to as "VSCode") in the future that Gitpod will be able to use legally...

Visual Studio Code is designed to fracture
I used to think GitHub Codespaces would help popularise Gitpod but now realize it is the other way around. Gitpod is currently permitted to exist in the Visual Studio Code ecosystem to popularise GitHub Codespaces, and Microsoft can step in at any moment to create legal crises that strategically div…

Competing against Microsoft is never fun. The way Microsoft enforces their competitive advantage is by making something the default. Paul Stovell, Founder and CEO of Octopus Deploy witnessed this first-hand in 2016.

Overnight it became the default. We were exhibiting at Build 2016 at the time much of this was announced, and I remember people coming to our booth asking "so why should we use you over the Microsoft thing?". The "Microsoft thing" was announced only 5 minutes prior!
Why we terminated our partnership with Microsoft - Re: Next decade of open source
Aaron Stanard published a blog post about The Next Decade of .NET Open Source. It’s a good summary of recent conversations and there’s a lot to agree with in the post. In particular, I agree strongly with this point: If Autofac is genuinely the better tool for your business’ case

Reproducible developer environments are a movement and not a unique feature of any one particular product  

Reproducible developer environments are indeed a sleeper technology that’s going to ramp up for a decade in usage until, one day, everyone will be “behind the times” if they’re not already using them, but I think the industry is heading in a different direction to the techniques used (ie. workspace-images and .gitpod.yml) and advocated by Gitpod.

.gitpod.yml
Explore the documentation to learn more about Gitpod.io and Gitpod Self-Hosted.

The problem with workspace-images is that customers do not get security updates unless a Gitpod employee remembers to update this magical line in each one of the Docker chunks.

https://github.com/gitpod-io/workspace-images/blob/main/chunks/lang-c/Dockerfile#L7
Update gitpod/workspace-full to update Let’s Encrypts root cert · Issue #528 · gitpod-io/workspace-images
The current version of gitpod/workspace-full contains an outdated Let's Encrypt root certificate. This causes the problem that pull and push are not possible with self hosted GitLab instances t...
for 3 months last year the letsencrypt SSL bundle was expired cause no-one is tasked to bump the magic number

The problem with .gitpod.yml is that it dictates a particular way to achieve reproducible developer environments that are locked into a singular vendor vs the competing devcontainer.json open-standard which Microsoft is shipping as the default product experience in Visual Studio Code and GitHub (and soon Visual Studio) Codespaces.

If there's one thing I've learned in my 40 years around this sun is that whatever Microsoft ships as the default is the thing that will win.

However, I think both approaches are missing the bigger picture that the industry is looking past Docker and towards tools such as Nix (or Guix) as they provide reproducible developer environments whilst also providing tooling that enables complete ownership of ones software supply chain (via source/binary substitution) and the generation of software bill of materials.

GitHub - mitchellh/nixos-config: My NixOS configurations.
My NixOS configurations. Contribute to mitchellh/nixos-config development by creating an account on GitHub.
"I like to use macOS as the host OS and NixOS within a VM as my primary development environment. I use the graphical applications on the host (browser, calendars, mail app, iMessage, etc.) but I do almost everything dev-related in the VM (editor, compilation, databases, etc.)" - Mitchell Hashimoto

It's no secret that I'm a fan of nix, the build tooling stole my heart four years ago after a research detour into academia and after 19 years brewing in academia, nix is going mainstream...

and by utilising tools such as Cachix and nix one is able to achieve Gitpod's feature set of pre-builds + reproducible environments in a vendor agnostic manner which is super cool, however...

In this economic climate, I don't think any product should dictate reproducible developer environments (or tooling such as nix) before value can be derived as doing so introduces way too much product adoption friction.

The key to winning the valuable enterprise market is to integrate in with how enterprises currently work, minimise the amount of people/process change required and thus de-risk the political costs for a products internal advocate.

Requiring people to add .gitpod.yml in each one of their git repositories is way too much of an ask. I was quite vocal about this topic internally during my time at 'pod because open-source maintainers kept pushing back at the prospect of "adding another yml".

Gitpod's innovation has ground to a halt

When I first heard of Gitpod (circa 3-years ago) and they later asked me to join them I was excited about three features: prebuilds, workflow and ephemeral workspaces.

prebuilds
Gitpod was 1st to market with the concept of pre-building a developer environment, but in this small window of time, GitHub has since re-implemented the concept in a better way via GitHub Actions, the devcontainer.json open-standard and through advertising/educating their customers about the feature in the GitHub user interface.

Prebuilding your codespaces - GitHub Docs
To speed up codespace creation, you can configure your project to prebuild codespaces for specific branches in specific regions.

the Gitpod workflow
Typically creating a workspace is done by prefixing any GitHub URL with https://gitpod.io/# thus https://gitpod.io#https://github.com/gitpod-io/gitpod.  However, GitHub has since implemented and popularised the convention of converting GitHub URLs from "dot com" to "dot dev".

17.8k likes on social media -> platform advantages at work.

Thus to open https://github.com/gitpod-io/gitpod in [GitHub|Visual Studio] Codespaces it's as simple as either pressing "." on your keyboard or converting the URL to https://github.dev/gitpod-io/gitpod.

Alternatively, one can just click on the giant green create codespace button in the GitHub user interface by default.

Whereas Gitpod requires people to know about their browser extension to achieve the same functionality.

ephemeral workspaces
One of the key differences between Gitpod and [GitHub|Visual Studio] Codespaces is the workflow that Gitpod prescribes of one workspace per branch but ...

self-explanatory

In three years Gitpod hasn't shipped new innovations. Yes, there was that JetBrains partnership announcement, but feedback so far from customers has not been great.

This sector moves fast; GitHub has already cloned many of the key value propositions of Gitpod. Newer competitors such as Okteto have sprung up and are piggybacking off Gitpod's marketing spend to promote Okteto's product - which to be clear - is a competitor of Gitpod...

A Better Developer Experience With Gitpod & Okteto | Videos
Empower developers to innovate and deliver cloud-native applications faster than ever.

Kubernetes is the wrong abstraction layer

A development environment on Gitpod happens within a Kubernetes pod. It's pretty neat, but upon reflection, I think Kubernetes is the wrong abstraction layer for Gitpod for the following reasons:

  • A product designed around Kubernetes constrains an offering to software developers who use containers. In this economic climate, enterprise companies want a singular tool that works for all software development scenarios - including windows desktop development, macOS mobile development and data science (ie access to beefy GPUs).
  • Kubernetes is complex. Enterprises do not have significant experience with Kubernetes and thus selling/supporting Gitpod as an on-prem product is going to be a painful and expensive cultural transformation that I don't want to be part of.
a snapshot of a conversation with a mentor with on-prem experience about topics of GTM
  • I think containers are not a secure enough environment isolation technique for a remote code execution as a service business aka running untrusted public workloads. Gitpod does some really cool things with Linux namespacing, but it isn't enough (security-wise), and it comes at a price (product-wise / see below)
  • For almost two years it has not been possible to run Kubernetes natively on Gitpod due to the above. One of the key motivators for customers moving a development environment away from a local laptop and into the cloud is to be able to run cloud-native workflows and applications. And on Gitpod, you can't.
Epic: Run k3s natively in Gitpod · Issue #4889 · gitpod-io/gitpod
Right now, to run k3s in Gitpod the only viable option is to use emulation to create a VM as showed here https://github.com/gitpod-io/template-k3s I did some analysis of why, even if we are able to...

and that's about it. In all, I don't regret working at 'pod, and it is a company that is full of really nice people, but there are structural, strategic and leadership issues that need resolving before the product/company will shine.  Overall it saddens me that the company has switched into value extraction mode over innovation and hasn't kept up with the competition.

Thanks for reading btw.