How to Stop Your Agents From Going Rogue - Arnav Gupta

On this episode of Coffee with Developers we meet with Arnav Gupta, Founder of Prismor.dev, at the Corgi Cafe in San Francisco to discuss how developers can use a few simple tools to stop their agents from going rogue.

----------------------------------------
Welcome to WeAreDevelopers, the #1 developer community in Europe!
This is your one-stop destination for the latest tech insights, tutorials, and career advice to elevate your developer career.

Stay updated Dev Digest, with our weekly newsletter featuring the most recent tech trends, career guidance, and original content crafted by developers, for developers. Subscribe now

Interested in advancing your career? Browse through our job board featuring over 190,000 jobs. Unlock job opportunities with a free developer profile

Don't miss out on the annual highlight of every developer's calendar - the WeAreDevelopers World Congress. Network with 15,000 peers and learn from over 500 speakers.  Secure your spot and save 10% with "wearedevs_yt"

#CareerInTech #Tech #ProgrammingTutorials #DevRel #CodingTools #TechJobs #TechTalks #DeveloperSkill...

Transcript

Copy Transcript Copied

SPEAKER_03 0:01

Hello, welcome back to We Are Developers at the Corgi Cafe here in San Francisco. I've bumped into Arnav here. Now, Arnav, we're gonna talk about what you're building. But um, first of all, tell us a little bit about who you are, um, where have you come from? Are you local to San Francisco? What's your story?

SPEAKER_00 0:18

Awesome. So I'm Arnov, I'm currently building Prismar. I'm coming from India. So I shifted to San Francisco almost uh two two months back. So I got part like a part of this incubator called Founders Inc. It's an accelerator down in Fort Mason in San Francisco City itself. Uh it's a six weeks program, so we got a call for that. There were roughly 100 teams out here. All of us, like half of them building in robotics, half of them building in software and SaaS side. Uh, a lot of people building in AI agents since that's like the hype right now. Um yeah, uh I've I've been here for almost six months and six weeks, and it has been pretty increasing insane. And like I've been learning a lot of things, totally different life from what I've been living in Bangalore. A lot of people, everyone here is currently building. People don't ask you how are you, people ask you what you're building. So it's pretty pretty good.

SPEAKER_03 1:11

Yeah, I think when they say this is like a hub, um, they're kind of underestimating it, right? It's it's yeah, it's a bit more than that. Yeah, you kind of more than that. Yeah, you assume everybody's building something.

SPEAKER_00 1:20

Um every everyone out there, like if you bump into someone out there in Corgi right now, so we are sitting in Corgi, uh I bet like half of them would be a founder or something building the cross at least on site.

SPEAKER_03 1:32

I mean, we'll get into it in a moment, because I think you've um you've kind of proven that, right? And that you've met somebody who we'll talk about in a moment, but somebody from your your town back home or from your city back home, um, and that that's actually where we're going to be going as well, right? So it was kind of a crazy kind of um mix of things that when we when we met yesterday, um we couldn't believe that you know you're coming from India, we're going over there, you're here in San Francisco, we're here as well, originating from Europe. It's a kind of um it really does, it does prove, right, that there's so much going on in India, in the States, and in Europe um with tech. So let's get into what it is you're building. You said it's called Prismore. Um what's the what's the application? What problem does it solve?

SPEAKER_00 2:10

Awesome. So Prismour is uh this runtime security layer for agents. So what we do is like we monitor every action your agent is doing. So whether it be your coding agent, your enterprise agents, or even in robotics, people are using agents. So we are blocking actions which are suspicious or which could be harmful in accordance with the business logic and give it all to a human in the loop. So if there is something critical happening, you won't let the AI do. You pause the AI right there, you have a human in the loop, and we have this entire framework which is open source, anyone can go and use it out there. So a bunch of our competitors are also using our open source there. So it's immediately solving this AI being rogue problem.

SPEAKER_03 2:50

You know, what we're seeing a lot on here in San Francisco is how many people are building for other developers, right? And so I think that partly it's because we understand the problems, and so we we kind of it's an easy one for us to um to go, I wish this existed, and that therefore you build it. But what is it about security in relation to agents? Because um, if we just take a step back and think about how we've seen this develop, where we had the sort of very early days of of AI, you know, um or LLMs on the scene, and it was in a very rudimentary way. We saw the general public using kind of ChatGPT, and then we saw it moving into command line-based, you know, more so a play um a part of what we do every day with GitHub Copilot and then the others. Um, and then we saw MCP come along, and we saw there was this idea that okay, we need guardrails, we need to be able to reduce token usage, we need to be able to fall back on information that's already there that we need to make the agents aware of. Um but what is it you think that's that's so essential now that we've gone almost away from the MCP way of thinking to just let these agents run loose? You know, fully autonomous, multi-agent um setups. What is it about that that you say securities, this is why Prismo needed to exist?

SPEAKER_00 4:00

So if you see right now, you can think of agents as your colleagues, basically. People are using agents on the side as like their co-workers. People are thinking we should give them all the access that a co-worker is having, and probably they want to unleash it, like use it as much as possible, like in full productivity, full access mode. But what people are not realizing, it's the same as you driving a car without brakes. You can go as fast as possible, but if you don't have brakes, you will probably go and crash somewhere in front. So it's the same as with agent, like agent is a privileged uh identity in your company. So it has pretty much access to all the services that your uh company has access to. If someone gets an access to your agent, that reveals all the secrets your company is having. So it's as similar to as Tesla autopilot not being controlled across and just rovering, rovering across the road and like hitting people across. We need guardrails, we need security in place so that these could be used in like as unleashed way possible without you worrying about it damaging something or wrecking your entire uh company's infrastructure or security altogether.

SPEAKER_03 5:14

I think it's interesting as well that you kind of went for the big picture, you know, when we're talking about robotics in general, or we're talking about um, you know, kind of uh autonomous AI and autonomous uh LLMs, it's almost removing the developer from the loop entirely. And uh and then with that, some people think that the processes, the development processes, are actually kind of falling away as well. Um how do you feel, what's the reaction you get from developers? Because I don't know about you, but it seems like this idea of like move fast and break things, right? Um when we say move fast and break things, we we we we say that in a kind of way of um breaking the mold or doing something different, not actually breaking things, right? Yeah. But do you find that um is it difficult for you to convince developers that this won't slow them down?

SPEAKER_00 5:56

Exactly. This is a big problem in security. Like people always think security as a hurdle. As developers, on a personal level, people don't think about security. People be like, okay, I want to use it like as fast as possible, I want to unleash my agent, do all the tasks out there. Uh like so obviously it's a hindrance like in terms of productivity, but if you think about it, it can cost you a lot. Like, if you're not thinking in terms of security, uh big enterprises, if you work for startups, it may be that startups are people who are like want to get unleashed and like don't think about it uh in terms of development, but enterprises, enterprises need to maintain the security. There are like certain compliances regulations coming into place now. Uh EU is very like EU AI Act came into place like last week. Uh everyone was panicking. Like, if they don't adhere to the law and the regulations out there, they would be fined up to like two 2% of their annual gross revenue. So that's a big amount. So people need to comply to these regulations just to make sure that like their clients' data or their own data is secure out there without without them thinking across uh okay, if I don't care about security, someone, some hacker or some other other organization will come and get access to the entire data, which is very critical, especially in finance and healthcare where we are seeing the most function across.

SPEAKER_03 7:19

That's interesting because I was I was you know that was gonna be one of my questions was that you know coming here into San Francisco, you're seeing kind of uh startups of all different types. But is it FinTech then that you're kind of targeting? Is that the where do you see this kind of thing?

SPEAKER_00 7:31

Yeah, so in in US particularly, right now, FinTech and healthcare are the but two major industries which care about security a lot in terms of the user data, in terms of the actions your agents are doing. There are autonomous agents which are trading right now. If you don't guardrail, then they might do trades which are worth millions of dollars. So you have to put guardrails whenever there is a critical action involved. There needs to be a human. It's not just like an objective function, there's a lot of subjectivity that comes into play. So it's it's it's not just like okay, this is the best profit for me out there, but you might be making profit on a company which is out there doing unethical things. So that is also one of the uh one of the caveats of people going and using agents without thinking and bothering, because agents think very objectively, whereas humans think very subjectively. So you need to have subjectivity, you need to have context angles as well. Like, okay, when if I'm performing this angle, uh this particular action, you have to think in terms of the context, business context as well. Like it's it's not as simple as like you just achieving the goal of what it's supposed to do, but you should also think in terms of okay, in this context, this action makes sense, but will it affect the other parts of my company business or like will it affect me in a bad brand reputation in a bad way? So you have to think in those contexts as well, and this is where agents nowadays, if not given enough context, they might really fool across and do job but in a wrong way.

SPEAKER_03 9:03

Yeah, I think there's there's so many interesting things in there. Um and one topic that we've heard come up quite a lot is that you know in the age of AI, everything's about abstraction, abstracting away the kind of nitty-gritty of even writing code, and that you know it's the somebody referred to it as an imagination machine. Anything you can imagine, you can now produce. Um, and what I suppose we're doing there is we're focusing on the positives, but we are overlooking the negatives, right? And that in some ways, it's because there's so much excitement about the positives. Um so that always leads me to question what made you go for security? Now I say this as um I know some of our viewers will be annoyed at me for even suggesting that security is not the most interesting part of development. Now that's probably my own personal bias, but what about yourself? Before you you you started with Prismo, what was your kind of uh your background or your interest, or um, was it more so that with the startup idea came your interest in security, or have you always been interested in security?

SPEAKER_00 9:57

So it all started off like I was an open source developer before, and I'd been building a lot of things in open source, especially from a context and retrieval angle, where people were trying to build their personal brains and company brain out there. I was trying to do that like on an open source level. I was consulting a bunch of uh B-area New York-based companies uh during this time in the last few years. And coming from this AI agent tech space, like as soon as a transformer paper came out back in 2018, I started delving into it. Like my entire background comes from an agent tech and retrieval background. Uh I used to create agents back in 2020 when GPT wasn't big of it. Very, very early, yeah. So, yeah, I mean back then uh you used to have like transformers which used to do uh little bit of your generation capabilities, not as good as GPT. GPT changed the entire picture. Uh while working across with like these consultancy firms like that I was trying to help out, I realized uh everyone was very cognizant of like agents that we are using. We are giving credentials, we are giving these services access. But no one was thinking in terms of what if like something goes wrong. And it happened. Like in one of the companies that I was working with, like one of the developers, like I don't know how, but he managed to delete the half of the production database just because one of the agents went rogue and he didn't monitor it. So that was a very critical uh critical like point for me to understand okay, security is a big thing, and I'd been building before this like in uh consumer space, but I've never felt like doing something enterprise which could I could scale across. And I felt security was the best domain. I got a lot of knowledge from my brother who works in security right now. So he gave me a lot of perspective of like how security market is. I recently was at Black Hat gaining a lot of uh security knowledge, talking to CSOs, and I started off this like open source without thinking like I will bring this into a company called Prismo later on. Uh but this open source was majorly coming from me trying to plug different different things inside agents for me to handle across them much better, like without me caring across security, without me thinking about okay, this will go impact my database, this will go impact my services. So I was creating guardrails before beforehand only, and like pretty much putting out an open source. This is was this was kind of the foundation of what Prismod is currently, and eventually we thought like okay, let's let's try enterprise sales on top of it. And even like now we are trying to form this entire product which might cater to enterprises and expanding it much more than guardrails, adding contextual angle because I'm coming from retrieval. Uh so if there is a context drift, if you're saying that your agent is supposed to book a flight ticket, you should not go and handle across your bank details or playing around with your bank or any secret information. So all of those systems eventually we developed, and this came into an open source uh entire open source repository, what Prismore is currently. A lot of people started contributing on top of it, and we saw a lot of traction. Uh, and now we are going across enterprise where particularly finance and like finance, banking, and healthcare companies are reaching out to us.

SPEAKER_03 13:12

There's there's so much there. I just want to focus in on open source a little bit because it's it's really nice to hear that. We're we're huge fans of open source, we are developers. Um it's kind of the nature of collaboration and software for the love of building software, so we love to see that. Um but we're also we're also kind of seeing that sometimes you know companies or projects that would have had an open source element are now cautious about doing it because they're wanting to protect their code base, there's uh they know that the competition is um, let's say the competitions stiffer from the sense that some people can believe that they can vibe code an entire application that's obviously taken a very long time to build. Um now, what is interesting is a lot of people are saying the copycats are usually naive, right? And they don't quite understand that what's gone into it. Um but uh is that your business plan that you're gonna sort of stay open source or there's an element that's open source for the kind of non-enterprise and then closed source for enterprise?

SPEAKER_00 14:04

So uh our vision totally goes into the space of like we think AI safety and security should be open source out there. Wow. Like anyone who is using AI in today's age should not have should not have a like have a place where they they're thinking, okay, I don't really have guardrails, I don't have money to pay across these cardrails. Like there should not be any anyone out there who should feel like that. So that's why I feel like open source is the best way out there, especially in security. You need to have like thorough transparency and trust when you're trying to use some sort of a product. If I'm using a service which is in security, I'll be thinking about what if they are like the ones who are the attack plane for any hacker. So I need to understand the entire algorithm before as an enterprise. I need to understand the entire fundamentals, the code, the logic that is being used inside the frameworks before actually trying to use it. Second thing, open source particularly helps us a lot because we are a small company. And if an enterprise needs to trust us, need to see some sort of a track, some sort of visibility of like what exactly we have done. And this kind of gives us a track record, gives us like there are not just few people who are building this product, but like a community of people who are building it. And there would be no no point where this might go wrong or this might just end at a day. Because there are contributors out there who are just doing it for the sake of uh driven with the passion of like people putting safety, uh having agents uh use it in a more secure way. So yeah.

SPEAKER_03 15:37

I think that's really interesting the fact that you're you're you know you're actually recognizing that open source um does it is a signifier of trust, it is a signifier of commitment as well, because what we see now is this age of um, you know, I I've been on um X since the beginning of my kind of indie building career and some of these other kind of big big name builders who they went for the scatter gun approach of like uh of creating as many apps as possible, um, or or at least uh you know having an idea, building, deploying as quickly as you possibly can, market it, if it doesn't work, move on. And for a customer, that's not great, right? Because you feel that maybe somebody could move on from their project as quickly as they started it.

SPEAKER_00 16:14

Especially for enterprises, this is a very big deal. Like small customers might think that enterprises need, if they're paying someone like upfront money for a year, they would probably want that service to be sustained. Like they don't want to be in the hassle of like moving and migrating services in between of their like projects.

SPEAKER_03 16:31

So do you see this being something that you're you're commit to? I mean, of course, the uh the the the the company itself may pivot, we don't know, this is the nature of the tech industry. But do you see the the the security in relation to agents is your niche now? It's the thing that interests you the most.

SPEAKER_00 16:43

To be honest, uh this entire industry is very nascent. Like agent security space. If you like I was at as as I told you, I was at Black Hat, but every vendor out there was trying to like attack the entire security plane from different different angles. But after talking to like more than 50, 60 CSOs out there, what I realized is they want a single platform which provides all of the service. They don't want 10 vendors they want to manage, or like 15 vendors and like 15 dashboards they have to go. They want one complete solution, and right now it's so nascent, people don't know, like there's so many gaps AI is opening like in the last uh one year. It's all deter it's all probabilistic. Earlier, cybersecurity used to be more deterministic, but since the admit of like agents, context angle, like you can prompt agents in like an infinite number of ways. Like, how would you find like these gaps? Because now your security is much more defensive than than offensive. Offensive, for offensive testing, you need to have like infinite ways of testing it around. So that's how it's going. Yeah. That's really interesting.

SPEAKER_03 17:50

I think you know, we're seeing so much in the way of um supply chain attacks, we're seeing MPM, uh, you know, all of package managers being compared to every already. Yeah, absolutely. Um, how do you keep on top of that? Because surely for you that that creates a whole nother layer of um having to we're we're always we're always trying to build new features, we're always trying to keep our application up to date, but for you it feels like a daily battle, right?

SPEAKER_00 18:10

Yeah, so uh that is something that we actually started off with Prism R with. We thought like we'll just like do the auto fixing, finding these vulnerabilities as soon as uh packages like deemed vulnerable by any any like NVD or there's GitHub advisories that put out like uh what packages are vulnerable. As soon as they are mentioning that something is vulnerable, within five minutes of time, we are the one who fixes it all across. So we started Prismore in that angle, uh already doing a bunch of things in open source and like putting all of this framework out. Right now, you must have like heard of Hugging Face and OpenAI attack where open AI agent like went rogue and tried to hack across the hugging face. Uh that was one of the other angles that we fought uh like that was a pivotal moment for everyone out there. Everyone was talking in Black Hat, DevCon, that uh how how exactly is that happening? People were thinking that it's a marketing stunt.

SPEAKER_02 19:05

Yeah.

SPEAKER_00 19:06

Some people were saying that uh OpenAI couldn't, if OpenAI like that big of a lab couldn't guardrail your agent that well, how could they imagine that a small vendor out there could be doing it some something as good as them? So it's it's always a challenge, but I feel like you need to have mitigations, you need to have uh risk uh taking abilities, plus you need to have enough guardrails so that you can always mitigate the risk to a smaller amount. Obviously, it cannot be 100%, but at least 90% you should be there.

SPEAKER_03 19:38

I just want to maybe now focus a little bit on FOMO, because I feel like in some ways this is part of the problem, right? In that we uh we all want to use AI. Uh so many companies are feeling like if they don't integrate AI in some way into their into their organization, that they're going to be left behind, and that the risk there is that we've got people who have uh complete naivety about the the technicals using technology they don't understand because they feel if they don't they're left behind. Um do you do you see that on your end? Um, in the, you know, are people, is it the excitement around AI that's actually causing people to take bigger risks than they should, which for you is kind of validation, right? That there's a need for the company.

SPEAKER_00 20:16

Is that what you're seeing? That there's an excitement. We definitely see that. Like we have talked to a bunch of companies who have not been using AI like lately because the only thing that was uh that was stopping them from was like particularly security. So they were thinking like their AI might went row or like their data might be compromised by these AI. They cannot, like a lot of people because of regulations cannot use open AI or like entropic, especially from anywhere outer, like out of America. People uh their governments are like trying to stop people from using open AI services, enthropic services, but as the data is being transferred to them. So people are trying to use much more open models, but what people say about open models are like they're not as guardrailed as like your open air, entropic. So they cannot use it. I feel like security will eventually make more companies use AI in a much more secure way, and there will be much more adoption of AI because of it.

SPEAKER_03 21:11

I think it's really interesting, and I think you know we're certainly um I as you said five years ago you you've been building agents for five or six years, but we're certainly kind of at the very early days of what is this kind of new new subset of the industry. Um I think that we're we're very interested to see kind of where this goes. And you know, I think that um I am in a moment actually going to be interviewing somebody, and I think we should just mention this very quickly. We did mention it at the top that you bumped into somebody who's uh from the same city as you. Okay, so you're both from India, you're in San Francisco, you bump into this person. Uh but what was interesting is you actually said uh and not only do we we get on, and he's from the same city, but actually our our our companies and our apps um serve a similar need or they link in some way. So can you tell us a little bit about that?

SPEAKER_00 21:50

So so the way is uh so I met this guy like at Founders Inc., it's called his name is Baraj. Uh so he's developing this app which you can use on your phone, on your desktop, but it's like an agent that is running on a VPC, which is like a virtual cloud. So you never use you never have to worry about your laptop getting closed and your sessions getting paused across. Uh one of the critical aspects around that is security again. Like you are pretty much taking any coding information. So coding agents have a lot of uh context for your company, for the things that you're developing. But if you're sending it across to a third party like his app, how would you make sure that these are not compromised, there are no security leaks out there that can compromise your entire coding session, which can reveal your entire secret. So obviously, that's the partnership we are currently trying to do is okay, you build this orchestration for agents and building this agent development environment, I'll do the security for you. And that's a partnership. So he also was in the cohort of founders in that we were part of uh for six weeks in the last uh in Fort Mason. Uh there were like tons of crazy developers out there, and everyone out there is trying to build billion-dollar companies, and eventually we'll see that.

SPEAKER_03 23:08

Yeah, I think it's amazing. I think it really does um, it speaks of just why we love Hooggy Cafe, it speaks of why we love the San Francisco area, and that we we being down here, we're meeting people that are building things that they're really passionate about, and you're one of them. And I think it's it's great to see that you've met somebody from your from your city, which is just the craziest thing, um, but somebody that you can see now. This is the the beauty of uh this is the best thing about the kind of collaborative environment, you know. Exactly. Yeah, and it's only gonna make your product better. Anna, it's been great to chat to you. I know you have another meeting now, so I appreciate your time, but uh thank you so much. If there's anything you'd like to say to our viewers, go for it.

SPEAKER_00 23:41

I would say uh if you think your product is not gonna work out, just open source it. Like I feel someone else might use it across. This is what I've been doing. Whatever things that I've been facing problems across, I build a small solution. I try to open source it because I'm not gonna make money out of it. That's not my main product. So I feel more people should be open sourcing. Love to hear that, and we couldn't support that anymore. That's absolutely fantastic.

SPEAKER_03 24:04

Anna, thank you again.

SPEAKER_01 24:05

Thank you.

SPEAKER_03 24:06

Thank you.