AI Vector Search at Scale - Ewa Szyszka

On this episode of Coffee with Developers, recorded in the sunshine live at AgentCon in Berlin, we speak to Ewa Szyszka, DevRel Engineer at Qdrant, about how developers can build for production-grade AI search with their vector search engine and database, and what she's looking forward to seeing next in the realm of vector DBs.

----------------------------------------
Welcome to WeAreDevelopers, the #1 developer community in Europe!
This is your one-stop destination for the latest tech insights, tutorials, and career advice to elevate your developer career.

Stay updated Dev Digest, with our weekly newsletter featuring the most recent tech trends, career guidance, and original content crafted by developers, for developers. Subscribe now

Interested in advancing your career? Browse through our job board featuring over 190,000 jobs. Unlock job opportunities with a free developer profile

Don't miss out on the annual highlight of every developer's calendar - the WeAreDevelopers World Congress. Network with 15,000 peers and learn from over 500 speakers.  Secure your spot and save 10% with "wearedevs_yt"

#CareerInTech #Tech #ProgrammingTutorials #DevRel #CodingTools #TechJobs #TechTalks #DeveloperSkill...

Transcript

Copy Transcript Copied

SPEAKER_01 0:00

Hello and welcome to another coffee with developers live at the AgentConf in Berlin. I'm here with Eva who gave the second talk today. What did you cover?

SPEAKER_00 0:08

Ooh, so I covered today ways that you can save your token usage and cut that nasty bill and not hit rate limits.

SPEAKER_01 0:17

So you talked about optimizing, uh no, maximizing your data and minimizing your tokens was the title. How do you do that?

SPEAKER_00 0:24

Exactly. So well I come as a developer relations engineer from Quadrant, so we think that the approach is to use vector search. So we've conducted multiple experiments to figure out how we can retrieve information more efficiently. So the whole thesis of the talk was that the retrieval part is really important in making sure that you don't burn through all the tokens.

SPEAKER_01 0:48

So interestingly enough, I'm gonna talk in like 25 minutes and I'm gonna be talking about how to optimize your code output so that other people don't burn too many tokens by reading the content out there. Because I mean it fascinates me how the formats that we use these days are just so unoptimized. People are basically throwing five agents at something and wonder when uh that battle each other and wonder why the data is so so strong. In terms of uh vector s vector databases, of course, like how does that work? Do you do you install it locally, do you use a service, or what what would you tell people is the easiest way to get started with that?

SPEAKER_00 1:23

I think the first way is to figure out what kind of data you're dealing with, what's your use case, and also just not shove everything into the context window and take a pause and figure out, okay, so what data I'm dealing with, how can I embed it, how can I chunk it, uh, before I even store it as vectors so I can very quickly retrieve my most relevant answers. So I've spent quite a long time figuring out what kind of data sets uh I'm dealing with and then load it to a vector store and search your most relevant results.

SPEAKER_01 1:59

We do the same actually at We Are Developers, we actually have 4,500 hours of footage every every year of our events. Uh like 4K video footage, so we create like transcripts from them and then we run it through our own vector database to actually make it available for you to search things. So if you want to know about MCP, we give you a playlist of where everybody talked about MCP in each of the talks. And we realized that the caching in the database was the big winner here as well. But uh as somebody who came from originally SQL databases, like how do you get your head around a vector database? What's the main difference for you?

SPEAKER_00 2:35

Oh, okay, this is a really good one. So I would say if you're conducting search at scale, this is where you might run into bottlenecks. And also you said video uh data. So I think one big strength of vector databases is that you can have multimodal data. So you can have anything from audio to video, um, and you can store it very efficiently um at scale. So quadrant specifically, we sit at that like um we like to say when the search gets serious, this is when you should use Quadrant. Uh because you can parse it very efficiently through like the HNSW algorithm, or recently uh what came out was TurboQuant, which is another very uh fast way to go through uh through the data that you have stored. So I would say that that puts us on the bleeding edge uh among other things. And other use cases would be if you're using like IoT devices, so quadrant edge is one solution that helps you um in those um environments if you need um to have like a specific data residency, it's very easy to switch it with like um quadrant cloud or hybrid solutions too.

SPEAKER_01 3:48

Or Neo4j and all the other vector databases out there that are available as well. Sure. I'm just saying, okay. Uh but you said that you're actually working in this space as a DevRel person. Like uh what what is that like? Like do you come up with the use cases that you showed, or did you have some customers information, some some case studies that you could show here as well?

SPEAKER_00 4:11

Sure. So developer relations engineer is a fairly new role, I would say, in the industry. It's just a couple years old. And um it depends on the company, how they define it. What I do day to day is I come up with ideas of how to build workshops based on uh both customer stories of what um uh different teams are building um and also what people are asking from conferences from client side. Uh so I get to figure out how does that fit in the ecosystem, what can we build for people so they see themselves in the solutions that we build.

SPEAKER_01 4:47

In terms of uh in terms of people uh demos that you showed or things that you talked about, what was the most impressive gain that you've seen so far?

SPEAKER_00 4:57

I think I'm really looking forward to one demo that's uh being currently built, uh which is multimodal data and parsing specifically audio data. Um so looking into beyond LLMs, uh, I think this is like a sweet spot where people can think of all of that unstructured data uh beyond like the first use cases and first chat GPT-like uh solutions. So one of the um demos, for example, that was built by my colleague was looking at CCTV cameras and looking for anomaly detection uh for potential um criminal activity. And I think that just opens the world of possibilities out there.

SPEAKER_01 5:42

But you're not working for Palantir, are you?

SPEAKER_00 5:45

Cannot disclose. No, cannot disclose.

SPEAKER_01 5:47

Fair enough. Uh it's an interesting space where people uh where people don't realize that data comes in so many different formats. And always when I worked when I worked on Microsoft, we always got this demo databases that we showed people and in the workshops and they were like perfectly aligned and perfectly clean. I never had that in real life. I never ever had a database that was not full of errors or very bad uh very bad data. How do you deal with that, those outliers? Like when you when you basically uh is the cleanup process one of the first steps before you put it into the database?

SPEAKER_00 6:19

That's one of them, but um actually Agentic evals, it's a really big space. So um I would say the process never stops. So once you put something to production, you have to evaluate it, and um anything, whether that's a tool, that's a skill, needs to go through the continuous evaluation process. Um today's keynote from uh Microsoft mentioned that uh continuous cycle as well, and this is what both we recommend and we do. It's never the end. It's um have we caught those cases or not? One way to do it is to have some sort of uh evaluation data sets with sample uh perfect golden answers that you can compare things to as a starting point.

SPEAKER_01 7:01

It would be interesting to turn that around. I remember when when we talked about talking to the press, we always create a rude QA with like the most annoying questions that they could ask us so we can prepare for them. It would be similar to have like a really messed up database as a as a first step.

SPEAKER_00 7:16

Absolutely. I think that's because you want to figure that out before you push things to production.

SPEAKER_01 7:23

Cool. Now, uh one of the things in terms of your career, how you ended up where you are right now. You told me before there was a wild thing. You had your startup, you you you did this, like how does how does that compare to like I had my own thing and then I now uh am now doing DevRel for somebody else? Is it is it more relaxing? Is it stress less stressful? What what is the career that you would tell people is a better thing to do?

SPEAKER_00 7:46

Oh um, so first thing that I did out of college was to try to run a startup uh here in Berlin actually. Um there were so many things that I haven't learned, but I think it was the best decision to give it a like a legit shot because you learn a lot of things by doing, and that's what I think turned me um into a person who really likes developer relations because you have to build in front of people. Um so if you don't know something, you have this uh feeling that like okay, I have to keep up my game, I have to stay very sharp, as sharp as I can. Uh, and I like that environment both in startups um and developer relations. I think that's uh like if there's a Venn diagram, this is where the two overlap. Uh it's obviously very different to work on like a Series B company and a larger team versus when you're the one in the steering wheel.

SPEAKER_01 8:40

Do you think that with the world that we have right now with a genetic software development, it's much easier for you to start a company? Because I remember the be the meetups here in Berlin, and when you said that you're technical, it was like locust jumping on you. Everybody wanted to have a technical co-founder because they were actually happy to be a founder of a company but didn't know how to do it or didn't even know if it's possible. Do you think nowadays these iterations much faster that you can actually do it yourself without needing an extra technical co-founder?

SPEAKER_00 9:08

I would say the moat is the technical expertise that is like a domain expertise. So having fluency in a technical like software development domain is fantastic, but then having on top of that another like domain expertise, um it really is very powerful because with the tools, I don't think anymore it's about the next feature, it's about how fast can I figure out what people want to be solved and uh what's a nice have. And you probably want to get as close to okay, this is a pressing issue that someone would actually put a dollar sign to um to resolve and not a nice have. Um so yeah, domain expertise plus technical skills is where I think uh Stronger's founders lie.

SPEAKER_01 9:59

Which is tricky though, because now that we say like we you have to do uh agentic development, you have to do download 50,000 models on your machine every day, or you're falling behind. How do you get to become a domain expert when you actually try to be uh a check of all trades in the AI world?

SPEAKER_00 10:16

Yeah, um I think you like even people who are at a bleeding edge are asking themselves questions, so you can't like be the know-it-all. Um in a way, it it is such a fast-evolving environment that you you just don't sleep. But uh learning how to delegate for founders, um, like I think um working together with great founders around, uh they just find people who are absolutely obsessed about a certain topic and they bring them on the team.

SPEAKER_01 10:46

Cool. Now we're coming uh here in Berlin uh 2nd of July for the VR Developers World Congress. Are you gonna be part of it? Are you gonna be come around?

SPEAKER_00 10:54

Am I invited?

SPEAKER_01 10:55

Well, we can we can sort something out. We always need moderators, we need people that help us uh there as well. And uh yeah, I mean in general, I guess we can get you in. It would be fun to have you on the show as well. Fifteen thousand people, six hundred speakers, so we can try something there. Cool.

SPEAKER_00 11:11

Thank you so much.

SPEAKER_01 11:11

Thank you very much. Bye bye.

SPEAKER_00 11:13

Bye.