Validating Real-World Usefulness of AI Models - Julia Kasper

On this episode of Coffee with Developers - recorded on location at the GitHub offices in San Francisco - we sit down with Julia Kasper, Senior Product Manager for VS Code and GitHub Copilot at Microsoft, to talk about how she and her team validate the real-world usefulness of AI models.

----------------------------------------
Welcome to WeAreDevelopers, the #1 developer community in Europe!
This is your one-stop destination for the latest tech insights, tutorials, and career advice to elevate your developer career.

Stay updated Dev Digest, with our weekly newsletter featuring the most recent tech trends, career guidance, and original content crafted by developers, for developers. Subscribe now

Interested in advancing your career? Browse through our job board featuring over 190,000 jobs. Unlock job opportunities with a free developer profile

Don't miss out on the annual highlight of every developer's calendar - the WeAreDevelopers World Congress. Network with 15,000 peers and learn from over 500 speakers.  Secure your spot and save 10% with "wearedevs_yt"

#CareerInTech #Tech #ProgrammingTutorials #DevRel #CodingTools #TechJobs #TechTalks #DeveloperSkill...

Transcript

SPEAKER_00 0:01

Welcome to another coffee with developers. We hear live at the GitHub offices in San Francisco. Well, in the library of it, not everybody works with a place like that, I guess.

SPEAKER_01 0:10

I wish.

SPEAKER_00 0:12

So Judia, you and what are you doing here?

SPEAKER_01 0:14

Thanks for having me. My name is Julia. I'm a product manager on the VS Code team. In particular, I support all of the model launches in VS Code and in GitHub Copilot in VS Code and our chat extension. So yeah, it's been a fun journey. I joined the team last year in August. So it's now been roughly over half a year. Lots of learnings, lots of growth, and been super exciting so far.

SPEAKER_00 0:40

So you're part of Microsoft and working with GitHub rather than GitHub?

SPEAKER_01 0:46

It's you know, we all consider ourselves very close friends. Um so we are in this very lucky position that we work very closely with GitHub, but also with the Microsoft team. Um so if you look at my old chart, yes, I like direct in Microsoft, but we have day-to-day conversations, we work very close with the GitHub folks, so it really goes hand in hand.

SPEAKER_00 1:09

So when people talk about VS Code or old school people like me, we're always like, oh, cool new editing feature, cool new code folding, whatever, and these kind of things. But with the integration of uh of Copilot, it becomes much more than that. It feels like more like an integrated IDE writer now than a text editor. How are the numbers of people actually jumping into the fully like chat integrated thing in Visual Studio Code, or is it more like coders that use it like me, command I when I need it?

SPEAKER_01 1:37

I think we are getting a lot of different use. Like we have the ones that grew up with VS Code as the pure IDE code editor, who are not that often yet using our chat extension. But then we also get the new developers that have a very AI-first mindset who go in and purely want to use the Copilot chat extension, who immediately go in with the intention of I want to use the AI features, the AI functionalities in here. Um the numbers are I can't share too many details, but um they are constantly growing, and we see more and more users, also from the more old school, how you kind of called it, and maybe they are more and more using these AI integration more natively. Um and we truly see a shift on how developers are working with AI these days now.

SPEAKER_00 2:29

Now, your dedicated role is to talk about the model integration into the V into VS Code. And uh I mean it was interesting because at first it started with a few models and then Microsoft announced like we allow any of them in there. So, how many are in uh available out of the box right now? And how fast do you how long do you get time to actually get acquainted with them before new ones come out?

SPEAKER_01 2:52

Yeah, good question. Um so GitHub Copilep has different SKUs. So we have enterprise skew, we have an individual screw. So it does depend on which is the subscription, which is the license. First of all, you've subscribed to, and then depending on this, what are the models that you're allowed to do? As an enterprise, um, certain enterprises have certain policies, so it might be that your admin is restricting what are some of the models that you're allowed to see because of their security or ceiler requirements. Um in theory, um, we support all of the three big model providers from Google, Anthropic to OpenAI. And we also work very closely with them before these model launches to make sure that it really works end-to-end for all of our developers who immediately from day one want to use these new models to make sure it works in VS Code and with our chat extension. Um typically it really depends. I thought, so I joined the team and I thought, oh, we must have such good processes and like it works very smoothly. The reality is that these model providers sometimes get a checkpoint, a model checkpoint that is very promising, and they're like, oh, we have a very exciting new checkpoint. Can can you please go and test this and give us some results back and make sure and let us know how this new model checkpoint is doing within your harness. So it's very ad hoc sometimes. Um typically I would say we probably get like two-ish weeks in advance where we know oh, there might be a new model, um, uh yeah, a new model release coming in, and that's when our engineering team, our offline eval team started to get to work with these new models.

SPEAKER_00 4:33

Now you talked about e bills, what are they and what do people what why are they important?

SPEAKER_01 4:40

Yeah. Um so there are different kinds of evals. Usually when I talk about evaluations, um we talk about online and offline. Um so one of the very interesting things, AI is non-deterministic. So you might have experienced this yourself. You go in, you open the chat, and you ask one question, oh, um scaffold me a new web API. And every time you do this, you're gonna get a different result, which makes it so hard if we think for us to test before a new model gets released, how do we test and how do we evaluate how these new models perform, especially because of their non-deterministic nature? So, how it works is we very similar back then we had um tests, we had test scripts. We still do this in the evaluation or offline eval space. These are called assertions. So we write a test case which is scaffolding a web API, and then we have certain deterministic assertions to say, oh, a web API should have, for this specific task, it should have five endpoints or things like that. So we write up these assertions, so then we have a way to test if this non-deterministic model is able to achieve all of our assertions, only maybe a certain amount of these, and how many do we need to make sure that it actually works end-to-end.

SPEAKER_00 6:08

So it feels like a good prompt, actually. Like instead of just giving it an open-ended question, you're already limited to the things that you want, and then you get better quality results back.

SPEAKER_01 6:18

Well, yeah, I mean, if everyone is, would be as efficient as prompting, but we all know, including myself, sometimes I go in, I ramble, I don't really know what I want the model to do. I have a fair amount of an idea what I want it to do, and but not every prompt is always as um straightforward as it probably should be.

SPEAKER_00 6:38

One uh one thing you said was that companies might limit it down to only using a few models or different ones. Now, the big thing right now is that everybody has got worried about is like token pricing and like uh our engineers burning through like lots and lots of money for different things. Now, different models are differently efficient at different tasks. There's lots of third-party software that I see that actually allows you to see your cloud status, for example, in the taskbar. Is there an idea to do that inside VS Code to say, like, okay, here's the limit how many tokens you can use, here's which model is the most efficient one for the tasks that you need to do? Is there a way to get a token saving mechanism inside VS Code?

SPEAKER_01 7:17

Yeah, that's a very spicy question. I'm not allowed to answer all of them just yet. Um one of the things, um, so one of the benefits of GitHub Copilot is that we don't just want to limit you to one model provider. It truly is you can pick across different model providers whatever model you think works best for your task. Um we are, or we do want to be more transparent in sharing some of our own internal benchmarks across these different model providers to give our end users a little bit of a better, roughly fair estimate. Hey, maybe for this task, uh it would be super cool to use this model or another one. At the end of the day, something including myself, you get to know models and they all have their own certain personalities in a way. So you're getting used to a certain style, a certain way, maybe for a certain task. And I see people picking models based on tasks that they are more familiar with, the personality or the style that they like. So on a lot of use cases, I see, for example, um claude models being used for more web dev, um backend development being more used with the GPT models. Um so yeah, it's uh it is a tough question.

SPEAKER_00 8:37

There's also political decisions sometimes. I mean, there's lots of models that are from countries you might not want to send your data to, uh, and uh but that seemed to be very, very powerful. But it's it's such an overwhelming market. I mean, I look at at Hugging Face and I looked, I used to look up like a specialized model for different tasks, and I'm like, cool that I could run locally and things. I kind of gave up because there's a new model every two days. How do you keep up with the demand?

SPEAKER_01 9:04

Yeah, so now it's part of my job, so I basically have to keep up with the demand. Um it is very interesting for me personally, because now that I that we work so closely with the model providers, we can really see the shift every time a new model has, and what are some of the improvements, also so some of what are some of the personality, even like kind of like personality traits for these new models. So we now watch more carefully. Um but yeah, that it feels almost like we've also reached a saturation of how good is there even going to be a next big jump? Um, because right now it feels like GPD 5.5 and the Opus 4.7 one, they're very close in um quality at this point, so I can see why users are also like the quality is really good. Do we need a new model? Um we do see a lot of improvements every time.

SPEAKER_00 10:01

Do you see that there's a performance improvement or decrease from model to model? It feels like, of course, they can do more, but also are they more efficient than they used to be before?

SPEAKER_01 10:11

So so so far, um looking at our internal benchmarks, we've always seen a jump every time we've released a new model. Um, some might be smaller, um, for example, 4.6 and 4.7, there was an increase, it was just a little bit smaller than what we've seen with 4.5, 2.4.6. But every time there is um uh somewhat of an improvement. Um token efficiency is a different um different story. We do see uh all kinds of degrees every time a new model is being um released.

SPEAKER_00 10:45

Is there a way to connect a local model that I have already running on my machine with something like Olama?

SPEAKER_01 10:50

So we have in VS Code bring your own model, um, bring your own key, so you can um bring in, you can connect through your own API key to these models, so you can bring any kind of model that you would want. Um the nice thing for built-in models like GBD 5.5, if you don't bring your own model, and we do optimize our coding hardness for these models. That's why we put, especially before launch, so much time into making sure the system prompt is updated. The system prompt works um great with these new models because we have a little bit of time before the launch to make sure it really works end-to-end in hours. So that's one of the advantages of using the built-in ones. But we you can always bring whatever model you want as well.

SPEAKER_00 11:34

Can you say something about numbers? How many people do that, or am I a freak for doing that?

SPEAKER_01 11:39

I actually I actually don't no no no. I think there are. I actually don't know the correct number, just to be very transparent. But we talk a lot and I see this coming up and a lot of times where people are asking, oh no, I want this specific model because for yeah, whatever reason. Um and that's a very fair point. I just don't know the number, unfortunately. So so you're not a freak. We are getting the question more and more, and I see a lot of also um issues in now we as code um repo where people are asking constantly about hey, we would like support for these kind of um models or edge cases. So yeah.

SPEAKER_00 12:14

As an engineer yourself, how does that how does that that challenge feel? Like you seem to be, it seems like you're you're you're beholden to a third-party provider who does the models, and oftentimes models get like, oh, we might release it, we might release it, we might release it, and then like the newest model is the best thing ever. And how much time do you get from like as a as an integrator? Do you get preview time or do you also get it when it's when it's finally out there?

SPEAKER_01 12:42

Um so we typically get like the two weeks-ish weeks before a model launch. And usually what we do during this time of the two weeks is one very important thing, we are optimizing our coding harness for it. So, most importantly, we are looking at our system prompt and make sure that it works with the new nuances, how the new model works. Um we do this by running offline evolts. We constantly, like in this two weeks, we run so many offline evolts, we are tweaking certain system prompts, we are tweaking certain things in our coding harness to make sure the quality keeps going up, the resolution rate of how many of our test cases are being solved just goes up, so we are providing the best quality overall. At the same time, or in parallel, we're still doing a lot of internal dog fooding and testing, and which is interesting because these are the two types of feedback the model providers are really looking to us as the integrators. They're always asking about um eval numbers and they are also asking about what we kind of call vibe checks. Um so they are they love some quotes that we are getting from our internal dog fooders. So before model launch, I go in, I have a fun project that I'm currently working on, and I just test the new model and I'm like, do this. Sometimes we get some recommendations from the model providers. For example, GBD 5.5, they've done a lot of work into more front-end dev work. So they said, hey, our research team focused a lot on more front-end stuff. So I test a lot a lot of different tasks about front-end web development to see if whatever they claim to be better at, it truly holds off. And then we are sharing this kind of feedback with them as well. So they get a we're all hoping the model in providers including that the quality holds and um it truly reaches the promises what they are saying.

SPEAKER_00 14:34

Can you do a direct A B test in VS Code that you say, like, okay, I got this task, run both of the models, and then I will take the better result?

SPEAKER_01 14:45

Um so that's where sessions come in in VS Code. So um there we have different types of agents. We have local agents or um remote agents. Remote agents use um GitHub Copilot CLI, so you can just send it off, it does its work, and it comes back, or your local agent, you're in your editor, you see these local changes also being um applied at the same time. So typically what I do is I spin off different local sessions and I ask the same prompt across different models, across different um yeah, across different models, and see what they come up with, and then I let faith or whatever I feel like in that moment kind of decide. So I kind of do my own A-B testing as well.

SPEAKER_00 15:35

Is there anything I can do as an outside contributor to help with like integrating the new things? Is there a different integration in uh in Canary, for example, in VS Code Canary, or is it just that's more about UX of the system itself?

SPEAKER_01 15:50

Um so all of our system prompts for VS Code, because we are open source, they are public. Um, and every model provider gets their own system prompt because how these models behave, um so anybody can look at the system prompt, especially once we release a new model, everyone can immediately see what the new model prompt looks like, and you can kind of also tell and see what are some of the changes that has been applied that the model provider has shared, and we are always open for community suggestions. So if you have done your local testing and you feel like, hey, with maybe we want to make some or tweaks to the system prompt based on yours, we are always open for these kind of suggestions as well. Um so yeah, that's usually what I recommend people to look into and also to familiarize themselves a little bit more with these different models as well.

SPEAKER_00 16:42

What is a request you always get that you think like people this should be something that should be obvious? What is a stumbling block that people have that that comes in as a as a question to you?

SPEAKER_01 16:52

Usually the the very first one I get is what's the best model? Um and I wish I had this here is the best model, but you already touched on a point where different models are good for different certain tasks, like dev front-end versus back-end dev. Um so I typically, including myself or looking at the VS Code engineering team, I always recommend switching models from time to time. And also always please use the latest, just because they are better better, and we do see a lot of enterprises still using some of the older models as well, which um it makes sense maybe sometimes for them because of CELA requirements or enterprise requirements, but I always recommend um go ideally to the latest, just because we do know it holds the best of all quality. Um so yeah, usually it's the question well which which is the best model I should be using.

SPEAKER_00 17:51

And um, how does it feel that? I mean, that might be a spicy question again, but it always cracks me up that there's so many editors out there from different companies that are just a fork of Visual Studio Code with security turned off, or like uh or like special features. Like I love that Monaco and the environment that Visual Studio Code is has become the blueprint for all of them. But it does feel weird to say, like, oh, you gotta have start with this editor to be a good developer or with this editor. Like, how does it feel to be like in the OG of the of the development environment and other people like, yeah, I'm using that one, not touching yours anymore?

SPEAKER_01 18:30

Of course, I'm not gonna lie, of course, it hurts sometimes where you're like, ah, you know, it is a fork of MPS code. Um the re but also the reality is we've always been open source. The team has always been open source. So open source has always been a core part of the team culture of how we think of the product. So we always knew there could have been the opportunity, like the possibility of somebody forking our repo and making their own. So as much as it does hurt from time to time, we still value the open source community and we also benefit a lot. Like we get PRs from the community themselves. So there's always a like pros and cons towards being um open source with yeah, people taking that also for their own advantages. But overall, we just value the open source community mindset, and I think nothing has changed towards that mindset, and we're still very proud that we we love that open transparency, um and we don't want to give up on it, so there's never been any talks in the team that to making it um close to anything.

SPEAKER_00 19:37

It feels weird when one of the forks get becomes closed source. That's really the weird thing about it. And I I want to call that out because I worked on VS Code and Edge and Edge inside VS Code. There was an extension that we did, and a lot of people feel like they can't contribute from the outside. They're either not they feel they're not good enough because Microsoft has thousands of board developers sitting around that will do it anyways, or they feel like, why should I do it for a company like Microsoft or or why should I support that they have enough money? It's a great way to learn, I think. Contributing to an open source project, especially a big one like that, is a great way to learn as a developer, and it's also every contribution I found, at least in my team back then, was very much appreciated, and there is no dumb contribution that feels simple enough.

SPEAKER_01 20:25

No, totally agree. Um, and I think even especially with AI nowadays, um, I mean I joined the team half a year ago, and I feel like I I feel very empowered with AI because the first thing I did is I um phoned via Scode Repo and I asked, explain this to me, and in this way I was able to narrow down how certain things first of all work. Um so it's so much easier to grasp a big GitHub repo to better understand how it works, but then also to make these small contributions and start small, pick one issue that is open that the team has somebody assigned to, but we always love these contributions and we are a small team overall. Um so it is really seen and it is um appreciated. We call out these PRs even in our stand-up from time to time, where we are like, hey, somebody contributed it. This is one of the reasons why we are open source. Um so I always recommend use now that we have AI, use the power of AI to understand how it works and then make a small first contribution. It is seen definitely across the team.

SPEAKER_00 21:35

Cool. Well, thanks very much. This was insightful, this was good. Um are you coming to our event in September?

SPEAKER_01 21:41

I mean I hope so. Um so yeah, we haven't finalized times yet, but I would love to be there for sure.

SPEAKER_00 21:47

Yeah, and if people have any questions about models inside uh like AI models inside VS Code, that's the right person to talk to. Thanks very much.

SPEAKER_01 21:57

Thank you.