Why Development Experience Matters

In September, Ramiro and John traveled to Tel Aviv for a hosted event with Okteto customers and local engineering leaders. The evening included presentations from monday.com, Yotpo and a panel discussion about Why Developer Experience Matters.

We captured that conversation here…

Introduction

Ramiro Berrelleza, CEO & Co-Founder, Okteto: When Pablo, Ramon and I started Okteto, the idea came out of common frustrations we each had. We had worked in similar companies, building SaaS products in the early days of Kubernetes and Docker. We saw these same problems across companies; complex environments, slow onboarding, it was hard to validate if changes worked or not until you got production, we had shared test environments that were broken half the time and the other half they just didn't work. These pain points led us to starting Okteto.

I'm excited to talk to Lior and Ron about what they're building, the why behind their decisions and the benefits and challenges of building DevEx initiatives.

How did you start recognizing that Development Experience mattered for your company?

Ron Barabash, Chief Architect, Yotpo: For us it’s a story of growth in headcount and especially in R&D. COVID also changed a lot of the way we work in our day to day.

In the past, you have your developer paired with likely a more senior developer or somebody that is your buddy when you're onboarding, so you can speak with them and they help you out with setup of things like a development environment. And as we kept growing in headcount and with COVID everything moved to be more distributed globally - you no longer had that person sitting at the desk next to you. We saw that it became very difficult to onboard new teammates.

I started my current position 2 years ago. At first, I mapped out all phases of the SDLC (Software Development Lifecycle). I analyzed stages from creating new services, to development, testing, and build to CI/CD deployment of services. I started mapping out where the challenges were for the entire R&D team and we saw that the first major pain point for us was setting up a remote environment for our monolith application. And that journey of high-growth in headcount, moving to a distributed company policy and also the decomposition of our monolith to microservices is what led us to evaluate our internal development experience.

Lior Rabin, DevEx Team Lead, monday.com: Yeah, I totally relate with what Ron said. I started at Monday.com in the height of COVID. I was assigned an onboarding task and my goal (and every new developer joining) was to deploy something to production in the first week. I started off by setting up my dev environment. Here I was, a developer with 10 years of experience, trying to setup the environment and it doesn't work. I'm sitting at home and I tell my buddy on Slack that this setup doesn’t work. I’m sending him snapshots and trying to figure it out but it’s really hard to do async. It’s like the last thing you want to do on your first week as the new person is to have a zoom call for setting up your environment - that looks a little strange. So in summary, I didn’t hit my goal and I deployed to production on the second week.

That experience really annoyed me. We had also just moved to Docker and Docker Compose, and this was the shiny new environment. And I said, oh my god, what was this like a month ago? I think this was one of the first moments that I thought I came to a big company and everything should go smoothly and this can’t be how it should work.

And as Ron mentioned, we were experiencing hyper-growth at monday. We almost tripled ourselves in the R&D department. At monday, the developers have full ownership end-to-end, so they're a bit of a product people, a bit of QA, (we don't have dedicated QA) they also own metrics of their features and the data. They don't need to "waste" their time on setting up stuff for their local environments. They just need to sit on their laptops and start developing and doing impactful work.

This was one of the big motivations that drove us to creating a DevEx team.

What is the team structure?

Lior, monday.com: As we grew, we understood that we needed to split the Infrastructure group into Production Engineering and DevEx. The production team are the engineers serving monday’s customers. And the DevEx team is "serving" a different type of customer, which are the monday engineers. This is also how we present it in job interviews - you have a responsibility to your customers which happen to also be monday employees.

How do you see your team's responsibilities evolving as you continue to grow?

Ron, Yotpo: We are currently building what I call our entire platform engineering department at Yotpo. Since it’s so new, we thought about having a developer platform. You see that in a lot of companies right now, where R&D is growing and you want to have a centralized, internal and kind of like an opinionated platform. This is your product as an internal team. And it’s the team's responsibility to figure out how to expose your services to your [internal] customer.

The way I treat it is that we have our developer platform and part of it consists of our cloud environments. Developer Experience something that is being molded into and thought about with everything that you do. Like if you create a new service, what is the developer experience there? If you do monitoring, observability, etc, how is the development experience? So basically in every part of the software pipeline we ask ourselves is the developer experiencing something that we are proud of and something we are satisfied with.

Another aspect, especially with Kubernetes and cloud services becoming a lot more distributed is that everything is becoming a little bit more complex. And similar to Monday, at Yotpo developers have an end-to-end responsibility and ownership of their services. So we really want to reduce the cognitive load for them.

And because of that, we're using our platform engineering groups and improving developer experience on all fronts. I think it's a blowing up trend and I'm not seeing it go away anytime soon.

Lior, monday.com: Yeah, I totally agree. It’s funny, it’s like we are working at the same place - our story is so similar. We did not prepare for this talk, we’ve never met before, I promise!

At monday, we're also building an internal platform and my team, the Developer Experience team, is the team responsible for it. It’s a collaborative effort and not only us building it, but we lead it. In this platform we create a lot of services and we want this flow seamless, fast and as fail-proof as possible.

For example, when a developer creates a microservice, it's a click of a button and two check boxes. After two minutes, you can start developing because it's on Okteto from a template and it's all working with no friction. A few more button clicks and it goes to staging. It's amazing. It's a process that used to take us a couple of weeks, plus a few internal tickets for the team to create those resources and the helm charts or whatever. Today it’s just automated and it's a super, super good experience for the developers.

With this workflow the infra team is no longer the bottleneck. We're around one infra team member for every nine developers. The developers work a lot and we [the infra team] were a huge bottleneck to their work. Now that we have this platform and these automated processes, it reduces our tickets influx and things are moving fast while we can reinvest our time back to important infrastructure work like monitoring, building more regions, more clusters, etc.

Another benefit of this has been improved developer experience for the infra team. Our goal and vision is to automate as much as possible. The first part was setting up dev environments but then by using those environments we can reduce the load from staging environments. With faster inner loops, we don’t need multiple staging environments now, we could have one staging which is production-like. At one point I think we had 5 to 6 staging environments and it just doesn’t make sense when you can develop remotely. We aren’t completely there yet, but this is the vision.

Do you think Development Experience is something only for larger teams at big companies, or should startups be thinking about it from the beginning?

Ron, Yotpo: Yes, I think you should take a proactive approach when it comes to building a decent developer experience. Because it’s something that can actually grow, in a bad way, and blow up in your face.

As we saw with both of us at Monday and Yotpo, our first use case for Okteto started with our legacy applications, our monolith. Mostly because it's very hard to work with and having a one click deployment made it really easy.

So in that regard, smaller startups might not have these legacy apps and monoliths to see the immediate use case, but eventually you’re going to want to think about how your teams are organized and what they consume, and teams that need services from other teams. It’s going to happen where you’ll need a way to deploy something that is not your direct responsibility or project. And if you can deploy that in one-click and work against it in a cloud development environment with no friction, that’s super powerful.

Lior, monday.com: I think the main advantage is that you can deploy it and just forget about it. So I think for a small team it’s still very powerful to think about this workflow. Maybe you didn’t develop a service yourself, you didn’t work on that project and maybe the other developer is unavailable and you don’t have context or know how it works exactly, but you just need it to work. It can be simplified from the beginning.

Maybe at a three-person startup you don’t need a dev platform in your first days, but there are things like what we are talking about that will help you along the way. In some ways, at a startup you might have more to work on as an individual than at a big company. So if you can save time and some headaches by using cloud environments, why not? The technology is there - just use it!

How do you get Developer buy-in for something that’s a big shift in culture or workflow?

Ramiro, Okteto: I’ve worked in developer tooling for a long time and I’ve learned over the years that it’s really hard to convince developers to change their tools or the way they work. You can’t force developers to adopt something new. At Okteto, we of course dogfood and use cloud environments because they believe in the technology and see the value. But if I we’re to say hey let’s use Jira (disclaimer: I worked for Atlassian for many years), they would say “yeah, yeah, you use Jira, we’re going to use something else”. My point is, it’s a hard drive and I think you have to really pull out the use case for them. What have you found to be helpful as you shape new development experience paths for your teams?

Ron, Yotpo: We started off by launching an internal survey for our R&D organization. The whole purpose of the survey was to identify what the most painful points of their jobs were. We learned that their highest pain point was development environments. So we took that information and found a solution. We said hey, if that's your biggest problem, try this tool, Okteto, that we just implemented and see if it helps. And today I think Okteto is the most adopted tool across the entire R&D organization at Yotpo.

That’s because we actually solved a really big pain for our customer (internal developers) The way we went about it was to do a POC and a POC team which created the entire infrastructure for it. They led the POC and worked alongside our developers to create a really good experience for them. Because of that, the adoption just blew up.

In summary, if you want developers to adopt something, solve something that is really painful for them and they will come.

Lior, monday.com: I think that is an important point. If you help them, you don't need to force them.

We did find a similar thing. We already knew that people had a lot of problems with their environments. We started the POC with Okteto and defined every touch point with one microservice to work on. I posted a question in Slack asking if some developers wanted to join the experiment to solve the environment issue. I needed help on this because I’m not a developer on my day-to-day. I wouldn’t be the one using this every day and I needed their input.

I was a bit surprised when 10 developers said they wanted to join the POC. And it quickly spread across the organization and in a matter of days the POC experiment team grew to 30 developers. It was really crazy the adoption rate with Okteto. All 120 developers we had at that time moved to Okteto quickly after the POC was done and we finalized an agreement.

How do you measure success in your DevEx initiatives?

Ron, Yotpo: It's a really good question. Right now one of the biggest things that we actively seek to measure is your R&D velocity. You can start off by looking at DORA metrics and lead time is probably the most meaningful one when you talk cloud development environments.

This can also be one of the hardest things to measure when not everything is streamlined and you have unified pipelines for example. It's very hard to measure so the first thing that we do in order to prove the value of that kind of project, or any developer experience initiative, is to show adoption usage and also what I call customer testimonials.

I did this recently when I was presenting to management. I wanted to communicate how successful the initiative was. So I presented customer testimonials, stuff that the developers said to me when they started using Okteto. I also showed adoption and how many people are using it.

Those are helpful when showing success and I also believe DORA metrics is a very good way to kind of understand if the things you are doing are actually meaningful in terms of velocity. So we are not there yet, but we’re working on it.

Ramiro, Okteto: How do you track DORA metrics? What are using?

Ron, Yotpo: We did a POC with a bunch of companies like LinearB and OKAY Engineering. Basically those tools are sitting on top of your GitHub and JIRA and are digesting all the activity that is happening and showing you lead time, which is how long it takes to take something from development into production.

Again, we had some trouble initially understanding if those metrics are right for us and how they are being measured. We need to do a lot of work to streamline the interim process along GitHub and Jira before we get value out of using a tool like that.

Lior, monday.com: For us it’s almost entirely usage that we’re tracking. We have some manual surveys around developer experience that gives us good feedback, but we don’t have too many success metrics outside of usage.

What were some challenges you had integrating Okteto into your environment?

Ron, Yotpo: One initial challenge was that we had to pick the use case that we wanted to do the first POC implementation around. It was important to us that the project solved a real, painful problem for developers, and that was the key success criteria for the POC. So for us we started off with something small and we didn’t build anything new and in the end it didn’t solve a big problem, so it didn’t seem successful to us.

But eventually we found the right use case. It was the biggest pain point and most hard thing to do for a dev at Yotpo. So we figured if we can solve that, it’s very valuable.

My advice when selecting or introducing any new technology is to frame it in regards to solving a real problem and work with your customer or employee peers to scope and solve it. Usage is super important so I’m going to make sure I do everything I can to troubleshoot and help the customers, because onboarding any new tool is a big lift. Developers have many other things to do so you need to be very proactive if you are going to ask them to experiment and onboard a new tool. Take a very hands on, engaging and supportive approach.

Lior, monday.com: The most challenging thing was to make our monolith work because it had a lot of moving parts at the time (and has even more today). It was a very deprecated old version of rails and it had a lot of dependencies of some very hard coded stuff that was mostly technical. We chose this as the POC project because it was the most difficult thing for us. We said, if we solve this, all those other node.js microservices are peanuts.

During the first month of the POC, Okteto’s CTO Pablo, really dug into the repos - he probably knows our code better than me now - and he fixed and optimized a lot of the code to make it work. But we have some technical limitations on our side that are preventing us from using all of Okteto’s features like Preview Environments. We still need to solve that part.

What’s next in terms of Dev Experience that has you excited?

Ron, Yotpo: In terms of vision, I'm sticking with the developer platform. This is the biggest initiative that I'm trying to promote in my company. It’s all about having a centralized piece of infrastructure where you can go and create new services, understand knowledge bases and know where to see ownership of things. I want to keep investing in that.

And the other thing that has my interest is remote IDEs like Code Spaces and GitPod. We are interested in using one of those tools with Okteto then we have a full cloud experience when developing so you’ll never need to come back and install an IDE. You could work from an iPad if you wanted and that’s very cool to me.

Lior, monday.com: It’s also about the developer platform for us. It’s about providing developers a way of self-service for the infrastructure, where it makes sense of course. The vision is that you come to work, you open Slack, you open monday internal platform (Sphera), you run "okteto up" and that’s all you need. You can go from your CI and your monitoring tools, there are some notifications and diff detections. You can create secrets, create services, whatever else you could want in that same platform. That’s great DevEx.

Ashlynn PericachoMarketing / Mom-in-training 👩‍👦View all posts