The Vulnpocalypse

Hi everyone and welcome to Yells at Cloud. This time brought to you by PHP. 31 years old, runs most of the web, and this week it finally made the front page of the Wall Street Journal. All it took was an unauthenticated remote code execution bug. My name is Gunnar Grosch, and I am joined again by Danielle Heberling, Chris Farris, and Ken Collins. Welcome all to the show.

Thanks, Gunnar.

Glad to be here.

Danielle, I hear you've been at a gaming expo, but not any type of gaming expo, right?

Yeah, it's probably one of the most Portland things ever. it's the retro gaming expo. So focus on like older systems, a lot of stuff from the 70s and 80s. And then another awesome thing about it was they had arcades and pinball machines all throughout the expo hall, and you could just go up in free play. So like

Massive.

that alone was worth the the ticket.

Yeah, that's very cool. Well, some of us on this show are from the seventies or eighties, so that's one thing. We're retro. But then just take a look at my background. I'm all for retro gaming for sure. Then Chris, tell us about something you've built that actually worked on the first try.

it wasn't AgentCore in EU Sovereign Cloud because apparently that doesn't exist. But no, so I've been trying to you know, build something to allow me to have multiple accounts in Claude and so yeah, I had Claude I had Claude go and build me a multi account, multi provider MCP tool.

And you know, it does resource firewalling, so I can be like, this user doesn't get to send emails, and this Google Drive is off limits to this account. you know, there was a ton of OAuth stuff, and I really didn't actually understand it, but you know, Claude managed to you know write it all for me cleanly, and then I ran it through my normal SAST and I ran it through Security Agent, I mean Continuum, code reviews, and it came back with no findings.

So I'm kind of now convinced that either the machines are colluding against me, or Claude actually writes better code than humans and we should just fire all software developers. we'll see which ends

Mm-hmm.

up being correct after about a week or so.

Alright, we'll we will get back to that topic then. And Ken you survived the first episode and you came back? I think last

Thank you.

time last time you talked about church windows, if I remember correctly. Have

you have you done anything else that has nothing to do with with AWS or the cloud?

Well, ignoring the work that I've done in the garden, planting collards and garlic, I think I just went to a Product Fest in Richmond. So I live down here in Portsmouth, Virginia, which is right over by Virginia Beach. And Richmond is our like little Washington, DC. It's pretty close. It's only about an hour away and there's lots of big companies up there. And so I have to turn my technical side off and go hang out with the normies and talk a lot of product stuff. And it's been really fun. I saw some folks from the Duckbill Group there. saw

know, folks from Capital One, CarMax. It's just really good to kind of sit down, talk to people, hear how they're trying to solve for problems. And, you know, of course everything comes back to AI at some point.

I'm sure we're gonna get back to AI as well on this show. Wouldn't be Yells at Cloud otherwise. So yeah,

Did you say yells at Claude?

yells at Claude, yeah. Sometimes

Yeah, might be more appropriate.

Claude is my

best friend, don't yell

at him.

well I definitely yell at Claude. Maybe that's why he doesn't respond well every time. So as usual, we have a lot to get through and we are gonna start, sadly, with a goodbye. AWS is shutting down something that you, Danielle, originally helped build. Tell us about what happened.

Yeah, I will try to keep this short and not offensive. you might actually need to edit some of this. We'll see. yeah, so the the thing that you're seeing that's being deprecated or retired by AWS is I guess sunset is the correct word, is a service that they called Application Composer.

And then they changed it to Infrastructure Composer. For the listeners that may not know what it was, is it was a a canvas where you could drag and drop AWS resources in a visual way, you'd get a diagram, you could connect the resources, and then behind the scenes, it was creating an AWS SAM template. SAM is Serverless Application Model, it's a superset of CloudFormation.

And what a lot of folks may not know is originally that was built by a company called Stackery, a part of what we built actually. We we had a whole serverless platform. we had CI/CD, we had like spinning up ephemeral on PRs, like stuff like that. Really cool.

and a great

DX story to go along with it.

Yeah, that was actually originally how I met Ken, because we were talking at a conference before we were both heroes. but anyway, yeah, so AWS acquired Stackery, part of what we built became that Visual Canvas, which was super cool. it was present in the, well, still is until October 29th, present in the console. that is the part that is getting sunset, I guess is the correct word.

it will still be available if you download the VS Code toolkit, which is cool, but also kind of confusing because you say you're sunsetting something, but it's still there. but one thing that I will definitely say is I very much grew up with this visualizer. Like that was how I learned infrastructure as code and CloudFormation and it just really helped me like stretch

my ability into being able to see what was possible with AWS in particular. And I think my Yells at Cloud hot take about this is it was awesome and

really fantastic. And I know this is not meant as a knock at individual AWS employees, but AWS as a company, I just really feel like it deserved a much better steward than it got. and AWS as a company is really not good at things like developer experience. they're great at like the Lego building blocks, but like the hardest part

is how do you put them together? So

Yeah, and especially serverless has always struggled with that part. So many small different services that you use, but people don't always grasp it. I'm a huge fan of of Application Composer, Infrastructure Composer, and I've used it heavily throughout the years. So yeah, I'm very sad to see this go, even though it's I guess remains in one form.

Yeah, I think the, the product building, it's always been like easier to stand up a database in AWS with something like PlanetScale than it was just to go in and spin up RDS or do some other things because not necessarily with wires and boxes, it's just that they had a better click stream, right? There's things that people don't care about and wiring all that stuff up together. Danielle, do you think it's a, do you think it's a, do you think this is the death of wires and boxes or do you think they'll, you know, like even for

That's

agentic stuff, right? Wires

a good question.

and boxes was really big. Like here's how you put your agents together. Do you think the builders like Step Functions will all stay?

Yeah, I think to a degree they will still stay. and I actually almost wonder if like I I had no insight into the decisioning here. Maybe part of it is is like, we can just have AI do this. You know, let's just invest more in the agent toolkit or whatever AWS is currently building. And that that is very much true. And it can visualize stuff. so

Once you fire

the expensive humans, you don't need the user

you know, user or human user experience anymore. It's all about AX, agentic experience.

And there really are two sides to Infrastructure Composer. One is the the building part where you are using boxes and putting them together. I guess that is the part that's that perhaps agents will be doing instead. The visualization part of it is gonna be there as long as you keep using the the toolkit, I guess.

Yeah, curious to see what this CloudFormation visualizer or whatever is gonna be replaced with. Cause currently the bones of that is Application Composer. And in the sunsetting announcement, it did say it was going to be replaced with something better, which is hilarious, because side story, when I first joined Stackery.

like AWS had an existing, you know, visualizer for CloudFormation in the console. And that was actually a question that I had asked during the interview. I said, what if AWS like, you know, makes this thing better and then our company goes out of business, like, you know, what how are how are you gonna manage that? And it's funny because the opposite happened. we got acquired and we be became that thing. it was really bad. So

Yeah, the well the old CloudFormation visualization was not not great. I think a problem has always been as well that it's not usually one stack. You have

multiple stacks of things and it doesn't

really work across that way. You you have stack sets and so on, which then it doesn't work well with.

Yeah, in in defense of that, like it it was terrible and ugly, but like being able to build a visualized graph of AWS resources in a way that's like easy for a human to read is definitely not a small feat. 'cause I've seen firsthand

with Stackery. and then it's like then they release something new in CloudFormation, then you have to update it in your thing and sometimes things aren't always in sync. We had

Yeah, edit.

originally tried to also support the serverless framework in addition to SAM. And like it just got to the point where it was hard to keep both libraries up to date. So we just focused on SAM.

And I remember the, it's hard enough to delete a CloudFormation stack as it is. Bless your hearts if you've been using foreign keys between stacks.

Yeah, and edge functions, things that need to be provisioned in us-east-1. That was a a fun issue because we were a pre-launch partner for Lambda@Edge. So I remember like that was such a new thing of like this is only in us-east-1, so we had to figure out I I don't take credit for it, it was one of my co-workers that I sat by, but I got to hear him swearing as he was trying to figure it out and he he figured it out. yeah.

think you can take credit for it now. It's it's in the past, so

There you go.

I guess this this also makes me think about Fig. So many of us have used Fig in the CLI, which I guess became I think it briefly was part of CodeWhisperer then Q CLI, and then Kiro CLI, and I don't know if it's a thing anymore. And that was something super useful, having that autocomplete part in in in your CLI. so another, I think acqui-hire type thing.

Where they hired the people building the tool and then it disappears.

Yeah, I did

That was

recent too.

see a notification

they're getting rid of the autocomplete in the Kiro CLI , so that was sad to see.

Yeah. So it's it's a theme now, I think.

Agents don't need it.

No. Alright, so pour one out for Infrastructure Composer. And it's going out quietly. It started with a keynote with Werner and now it's in a quarterly bulletin. But this week, a lot of other things went out a bit louder, and most of them had CVE numbers attached. Chris, I think you named it. Let's see if I can pronounce this.

Vulnpocalypse?

The Vulnpocalypse, yes.

which is actually something that people have been predicting since you know Mythos landed on the market and you know Dario had to go and you know start proclaiming the end of the world to to juice his you know IPO price. But we actually kind of saw within the span of a few days a slew of fa sh my god

You know, so we I think it started and maybe I have this timeline wrong, but it started in my head at least with eMudhra which is I guess an Indian certificate authority, but I guess they LLM trained their you know, trained their LLM on an old XKCD cartoon which just had like return random number four. I used a dice to to roll this so it's guaranteed to be random. Because literally their private key

or one of their private keys was just the number four. anyway, so somebody figured this out, brute forced the certificate, brute forced the private key, issued a bunch of certificates. I don't think anybody's bank accounts got hacked, but when the browser manufacturers found out, they like immediately untrusted the whole company. they have said that, you know, they're out of business. They're not issuing certificates anymore because well

Somebody's gonna retrust them in the browser chains. And honestly, for our listeners, you should probably go and check your certifi packages that are in your OSs because they're now out of date, and your agents and/or applications may be connecting to something that isn't what you think it is.

and then you

started off talking about PHP being 31, but PHP is 31 and has had its midlife crisis and you know blown a disc and everything else. yeah, this one actually made the Wall Street Journal and The Economist. So, you know, when the normies are talking about CVEs, you know it's a doozy. and this was a great one because this was an example of uncoordinated disclosure.

One of the read-in package maintainers released their fixes early. one of the discovering firms released their exploit video hours before the patches were out.

Amazon actually reported, CJ Moses, one of their CISOs, reported that their honeypot technology was detecting this thing in the wild four days before the actual patches and announcements were released. So, you know, mmm. But you know, like Azure, PHP is probably a technology you need to run, not walk away from now.

Wait, wait, are you saying you should run PHP? okay, okay, okay.

Should run away from PHP. Yes.

Run far fast. Keep going. Don't look back. and you know, keep going past Node.js, which, you know, not to be outdone, had their own slightly less apocalyptic remote code execution bug. this included you know, the patch now includes the

Bundled fixes for the eMudhra stuff. So if you're running PHP or Node, which is everybody, make sure you're up to date. And then in the breach news, we have Railway, which is an AI cloud wannabe company. They had a bunch of their customer keys leaked. Which of course goes back to my adage, which is: you know, when you upload your secret strings to the internet, they become just strings. But

Lots of API keys that need to be out there. And I hope none of you run TiDB, I've never heard of it. But actually if you do, you're probably rotating keys and not listening to this podcast.

they too got compromised and the threat actors found an AKIA in their JIRA, which was then used to pivot into their cloud offerings. Round out the mix, you know, the company that funded the most expensive Nepo baby acquisition ever, Warner Brothers and Skydance. you know, that company, Oracle, patched forty-five RCEs. Fortinet, which maybe is responsible for I don't know what, forty-five percent of the CVEs issued last year.

You know, also had, you know, more ones. GitLab dropped a CVSS 10. And then for those of you who only use Route 53 as a database, probably want to go check your BIND servers because they too have a major vulnerability.

and then you know, we talk about supply chain, but you know, sometimes we actually have to worry about the real supply chain, you know, the one that gets us our food. you know, a threat actor group has compromised a South Korean IT firm that specializes in the maritime industry. So it seems to be now mostly a hack and leak attack. So, you know, put up your confidential data and demand a random.

a ransom. but who knows what else they got and you know which government may have it and you know what's that gonna do to global shipping if they manage to shut it down again. So my advice might be time to stock up on lead, gold, and canned goods.

Chickens and liquor.

you know, why does it you know it's not just AI code that's writing slop, people are doing it too. Humans are lazy and write bad code. and frankly companies aren't ready to address their massive tech debt. So yeah.

Chris, do you feel like a lot of this stuff is coming from the research firms that have access to Mythos or is it coming from other angles?

susp maybe the PHP bug. I think the PHP bug might have come from Wiz. or at least like I think it was discovered by a couple folks kind of simultaneously, which does kind of hint to hey it's AI. you know, whether it's Mythos or whatever the OpenAI SOL version of that is.

Can't I I can't say or I didn't, you know, read the details deep enough to to to find out. you know, but AI as a whole, definitely, you know, I'm seeing significantly more vulnerabilities. I think if you look at Chrome has fixed like 152 things in some of its releases, like fifty-five vulnerabilities patched in the latest Oracle thing. So I mean it's definitely

you're starting to see that if you don't have a good good process for continuously you know updating that FROM line in your Docker file, you're gonna be in a world of hurt. So yeah. Maybe the one

Something that

Yeah. sorry.

No, go ahead, Danielle.

Should I go? Okay.

yeah, I was gonna say I was reading the PHP Foundation did release like a post mortem of it and it mentioned that it was first found in 2024. I I actually wonder if some of this might just be a result of these foundations not having enough funding and a lot of open source maintainers doing free work. Like, what's your thoughts on that, Chris?

It's definitely a problem. you know, if you're outside of the the reasonably well-funded Linux Foundation, you know, if you know, it's that old XKCD cartoon where, you know, it's this massive Jenga tower, and then you've got, you know, one package maintained by a guy in Nebraska for the last 30 years. and we we

We know a couple of those people.

even saw that with the xz vulnerability, where the threat actor was gaslighting and harassing.

The poor maintainer of this compression library asking for patches, and the guy was like, dude, I got a life. I don't want to do this shit. Here, why don't you you here, I'll give you commit access, you can go to do it. and then it turned out to be somebody who didn't actually exist and then put a bug in SSH. So yeah, I mean open source.

Probably was a great idea in the 90s. and let's face it, you know, the entire modern eco computing ecosystem and the cloud is built on Linux. But right, it's is it still sustainable and fundable? I don't know. interestingly enough, I've stopped open sourcing.

You know, I I don't see the point in it anymore. I don't think I'm getting market value out of, you know, releasing the things that I do. So like this latest thing that I'm building, yeah, I just said, no, I'm not gonna bother figuring out how to open source it. I'll just keep it private and share what I did and somebody else can go prompt their Claude to go do it.

Yeah, that's one way. But I guess we still have the seventy percent of of the web that's running on PHP, so we can't stop with the ones that are already running. And I I think I read somewhere that the PHP Foundation has thirteen employees, something like that. Which is kind of wild.

my goodness.

Yeah. Yeah. Yeah. Given that seventy percent of the internet is running on PHP, probably half of that is WordPress underneath it. and you do have two commercial companies that are sort of behind WordPress and I think they hate each other. or at least maybe their CEOs hate each other. There there's some drama

Peace.

going on there too.

Definite.

but like it's also because, right, like I don't have to pay to use open source, so why would I?

and that's it, right? You know, short of the big companies, and all right, I will say something nice about Amazon, you know, and Azure and Google, you know, they are funding the Linux Foundation. and so they are for the core stuff that is keeping us all alive, you know, they are funding to to make sure that that happens. So but it's the small stuff, yeah.

Yeah, and and but there are two there are

two sides to it as well. One one is the funding, actually put in money. And I I think Automattic for instance. I think they're the biggest contributor to the PHP Foundation, which makes sense because WordPress. But that

but then there's the other side, which is contribution as well, because even with funding, you need to have people who actually do the stuff, do the work. and I think that's where a lot of the open source world is struggling.

having people that are able and willing to put in the time to actually contribute, no matter

the funding.

un unless it's their full time job. Like,

Yeah, exactly.

you know, I I I think of Mike whose full-time job is keeping Python safe. And I'm like, cool, I'm glad that, you know, you're getting paid to do this and that you this is not this the thing you do on the weekend when you are bored and don't have something else you need to do.

Exactly.

Anytime

I see PHP hit the news like that, it reminds me that maybe Ruby or Rails is kind of next, right? So the shout out to my sys admin in the late 1990s that founded the

Ruby user groups with Chad Fowler, who would prototype all of his PHP back in the day with Ruby. you know, it's got to be coming soon, right? Like I've not heard a lot of the vulnerabilities from the Ruby side, but.

you there's all those little packages and lib whatever underneath everything that is still there as well.

so much that is unmaintained these days as well.

Mm-hmm.

So something that caught my eye was how many companies and individuals who wanted to take credit for finding this. There was such a long list of companies that said we were the ones who discovered this, which I find kind weird. Of course, they they want the exposure if you're a security firm.

But there were just so many this time. Did they all just find it at the same time or

To be honest, I don't know. I could s I so to to what Danielle said, right? Like in the PHP post-mortem, they said, yeah, we kind of knew about this thing since 2024. So it could very well have just been three companies, you know, like, hey, we've got our new Mythos token budget replenished. Let's see what's in PHP. and in fact, I think I think Chainguard.

Am I getting this one

right? Chainguard tweeted or posted on LinkedIn that was like, yeah, we were looking at this, and we basically just said, go find prob vulnerabilities in PHP. And that was the extent of their prompt. like we

Ha ha!

Yeah. So

were all told, go become prompt

engineering. It's hard, it'll be the next thing, it'll keep you employed.

And then they go and find this, you know, thing that breaks the internet with go find vulnerability. And I'm like, all right, make

You forgot

the /goal part. /goal.

Yeah. Yeah.

So I think one part of that is Chainguard trying to explain why they broke the embargo six hours before they were meant to.

And what we're seeing right now is just internet is flooded with people who say they they know how to do the exploit, which doesn't seem to be super easy either. So it is there, but it's not something that

Yeah. So

just anyone can make use of. And that probably explains why it's been there for such a long time.

It's 5 o'clock Lisbon time on Friday the 23rd when we recorded this. And as of about 45 minutes ago, I had not seen anybody actually talking about actual exploits. there's a lot of noise, there's a lot of probing, but like nobody has you know, actually been popped.

So right, it's a really bad CVE, but maybe it's not as easily exploitable as everybody thinks. or, you know, Claude hasn't gotten around to, you know, figuring it out yet.

Right. But will this continue now then? this has been all the ones you mentioned, that's

just the last few days. Or is this just the tip of the iceberg?

You know, that's an interesting one. I think it I think it will eventually die down. because I think eventually all the crap code that us meatbags have written over the past 40 years

will eventually get solved and scanned by AI. And AI will eventually come up with patches for all of that. at which point, right, like we're we're we're at the peak now.

WAH-WAH-WAH!

said, I think maybe AI writes better code than we do. So maybe there's a light at the end of the tunnel, we'll just all be dead. Or irrelevant.

But yeah.

maybe. But I think yeah, I think we're gonna keep seeing this for for quite some time now. if AI ever solves all of the crappy code that we've written in the past, I don't know, honestly.

It can. If it just deleted GitHub, like accidentally agents, yeah, drop database, you know, as it was trying to fix it, you know, that's probably something Copilot would do, right? Then that's gonna get rid of a ton of bad code right there. So maybe that's the answer

Ha ha.

is just delete GitHub

and see what happens.

Feels like GitHub is trying that on their own though, right?

I think that's it. Copilot'll be just like, I don't know how to solve

this. Maybe if I just deleted all of GitHub, hey look, this the metrics are stable again.

if that opens up another issue though, because then if there's no code for these agents to train and steal ideas off of, but if it's all bad code that gets deleted, I'm all for it. But yeah.

Yeah. Well.

Well I I've been saying for a while

that we're gonna end up with black boxes eventually, so the code won't matter.

It's just

gonna be input output, that's all that matters, and it can be assembly or something in there.

Cause let's Well that's it exactly.

Like we go back to nobody knows what the bits are in a punch card anymore, right?

Nobody writes assembler anymore. Hell, nobody writes C anymore. So, right, like it's all Rust and Python, it's interpreted languages, and now we're

Mm-hmm.

gonna get rid of that and it's just gonna be prompt, write me a thing. at this point, half the time I will prompt Claude to go rewrite something rather than go find it in my GitHub.

All right. So I believe you mentioned Node as as one of the things on your list, Chris. and it happens to be that node is the only runtime for the newest way to run code on AWS Lambda. And since this is, I think, the one piece of good news on the show, I'm gonna take this one. And I wanna talk about Lambda Web Functions. So on October twelfth,

AWS launched something new called Lambda Web Functions, where you deploy a normal HTTP server, think Express or Hono, Fastify, something that listens on a port. There's no handler, no JSON event. You don't have to think about Lambda Web Adapter. One execution environment that serves many requests at the same time. And billing here is active CPU time plus the memory that it holds. So that's

gonna mean that time spent waiting on a database or some downstream API mostly is gonna stop costing money. At the same time, all the existing AWS Lambda functions, well, they got renamed to Lambda Event Functions. And I think this is the lambda that many of us wanted years ago. Web frameworks that run as is IO heavy APIs get a lot cheaper and

Single deploy that can be multi-region and you get one global URL. But it also, of course, then adds a new branch to a decision tree that keeps growing. We talked about earlier with Infrastructure Composer that one of the tricky parts with serverless has been figuring everything out. And now in less than 12 months, Lambda went from having one compute model to now having four.

Event functions, the old one, Managed Instances, Micro VMs, and now Web Functions, each with its own billing model, concurrency model, and set of knobs. So

left off CloudFront Functions, Lambda Cloudfront Lambda of Lambda@Edge, right? You know, there's there's more there

Provisioned, provisioned concurrency.

than just the the normal ones, so

Yeah, what about

Durable Functions? Do you count that as separate?

No, no, it's just a tab within your Event Functions. but yeah, you're proving my point. So I think Web Functions, it's a great release. I'm super happy about it. But the bigger question is what happened to serverless as a whole? I think serverless used to be hard for a lot of people to to understand. we have three serverless heroes on this show, and and we have all knowing Chris as well. So

I think many of us we've kind of figured it out by now, but it has been hard for people to understand because the model when you're building serverless is a bit unfamiliar. And I think it's getting harder because now there's just so much of it. You used to have one dial with Lambda, you remember? Memory. And now you have so much configuration you can do. There's so many options. And across these different form factors.

But maybe then the answer is that this isn't meant for humans to understand anymore. So we're coming back to that. Maybe it's enough if agents are able to understand this and just pick and choose the right one.

So Gunnar, do you think you could speak a little bit to the scaling model as you understand it? Now, and I'll ask that question with a statement. I love the old way because I could understand the evented. And, you know, there's counterparts on serverless heroes like Matt, who does the Bref PHP framework. Bless his heart, he's probably busy right now. the, you know, me, I stuffed Rails inside of a Lambda and scaled it up for Custom Ink.

and ran a lot of our workloads like a product catalog and things like that were, you know, directly competing with K8s and the scaling model. I loved it so much because, you know, just give me one virtual CPU. That's all rails needed. Right. And if another request came along, I relied on Lambda scaling functions, what I basically called sand versus rocks or boulders. Because when competing with K8s, like you still have to hurt the pod in some way to get at the scale, right? Like it has to feel some pain.

in some way, but with Lambda I was able, and the eventing model, which I think actually works really well with HTTP, it's in and out, it's one single event, it's share nothing architecture, we were able to scale so much better with Lambda than Kubernetes because it was just really hard to figure out how to hurt the pod and which way you wanted to scale it. do you feel like there's a, is there a new scaling rule here? Right, is there, is it?

laissez faire, it kind of hands off or do you just kind of have to put your hands in a different place?

So as of now, there isn't you don't have that many dials. It's a fixed size, I believe when it comes to the memory and you're paying for the CPU time. it does behind the scenes scale differently though. So with Event Functions, have to get used to that name. Lambda Event Functions. You you had the concurrency model where you had one of these.

Firecracker VMs spin up. And then if you had a second request come in at the same time, well, it needed another environment to run until the

Mm-hmm. Yep.

old one was ready. this one scales.

So request

cone could be measured by cold start in a sense.

But this one scales differently where you are able to have multiple requests to that same environment. So I think it opens up for a different concurrency model in the way that it's built, which I think is good.

Can you measure,

can you measure queuing time? That was what we had to do with Kubernetes, right? We had to measure the trade-offs between if it was queuing for a cold start or queuing for within side of the pod that you would be hard to see.

That's a good question. I actually don't know exactly what the metrics are and what what you're able to see natively as of now.

Sounds like we need some science.

Definitely need some science.

Yeah, I actually wonder since like you could just toss a whole express server in there without needing an adapter, if if

Mm-hmm.

people realize that, you know, this exists, and I think right now it's probably just mostly folks like us that are the power users. I actually wonder if maybe this will get more people using Lambda because a lot of people have wanted that use case. And it also makes it easier to just more intuitively run your handler code locally.

And validate real quick that it works.

Mm-hmm.

no, I fully agree on that. I think now people can build something locally using Express, for instance, and then you just push that to the cloud and it just works. What that also means though is that for me, that means that I take on a bit more because now I also have to own that server that is running the code. Sure, it it's it's running in AWS Lambda Web Functions, but

It's my server that's running there compared to previously, where it was just a handler.

Yeah, it fits the mental model

What does this mean for

much better.

So what does this mean for API Gateway? Because that was how we like turned HTTP into you know OG Lambda

Mm-hmm.

Functions. what happens to API Gateway? Is it gonna go the the Stackery route and be on a sunset list in a few years?

I don't think Event Functions are going anywhere though. I think there's still lots of

No no no no no. OG

Lambda will always be there because it's it's

Yeah yeah. But

how I do things with cloud EventBridge and everything else. But I mean like

Yeah. But

API Gateway itself, does it have a function now or does it have a purpose?

I would say it still does. So you still building APIs using Event Functions for sure, and and even direct integrations with API Gateway where you don't have Lambda functions,

Mm-hmm.

which is a story in itself, how how

Exactly.

that works. But I do think API Gateway is gonna be around for sure. Maybe it'll get an update, who knows?

V3 here it comes.

I mean it did get the HTTP

version at some point, right? So

we don't talk about HTTP APIs, no.

Yeah, what Ken said with the the V three, 'cause there are two API Gateways now and that's not confusing.

Right.

Wow, that sounds so Google of them.

the Google thing would be to to kill it off, right?

Well, th yeah, you've got three versions, two of them

Yeah, once it was successful.

have been deprecated

and one is undocumented. Yeah, and you're not sure which one you're calling.

Yeah. No, but I'm just to reiterate, I'm super happy about this launch. I think it's very cool. I think it's a great way to go. And like you said, Danielle, I think it makes it easier for a lot of developers to use Lambda. I think the confusing part is probably the bigger picture where you have all these different flavors and with Lambda Managed Instances where you can

Then run Lambda functions for up to ninety minutes now, which is another thing. should you?

like Amazon Q. I mean you can't just say Q, you have to say which Q. Now when you say Lambda, you have to say which Lambda. Totally serious here.

Yes. That's right.

And if you don't, then I you mean OG Lambda,

the Event Functions thing.

So this actually has me.

Yes, Lambda Event Functions.

generally horrified because it's that much more now attack surface that's gonna get dropped on the internet.

So something as well, like I said, I I think this is partly meant for agents to understand. Something that I noticed is that when you do a deploy, it actually generates an AGENTS.md for you as well. I think that tells the story about who the audience for for running this actually is. so and as of now, obviously.

Get rid of the

The agents don't have the knowledge about Web Functions, how it works. Documentation is brand new and they don't know how this works. So having that there, yeah, it probably makes it easier for Claude or whatever to understand how to actually build and deploy this.

Where does it write that AGENTS.md file to?

So when you run the the deploy command, it ends up in that directory.

So you use the AWS CLI. I think it's lambda-web is the is the new syntax you use for web functions.

And then you can deploy it through there. Very cool. Very demure. All right.

And you're right.

I just asked Claude, can you re implement my project cheaper using Lambda Web Functions? Is like I have no idea what Lambda Web Functions are. Search the internet. So there you go.

There we go.

It took my agents such a long time because I've been working a lot with Lambda Durable Functions, and they just didn't want to know what Durable Functions was. No, no, that's an Azure thing because they apparently have something called durable functions. So I had to

always remind them it didn't matter if I put it in claude.md or had the skills.

Is still wanted to go the Azure route every time, and I can assure you that I was not going the Azure route.

All right, so AGENTS.md when you deploy.

And maybe the agents will help pick the Lambda setup for everyone. And I think Ken, your topic is what happens when the agents form a team and start rewriting the tools they run in. Ken.

Mm-hmm.

Is Claude teamin' and modin' ?

It is and not a lot of people know about this and I find it I love the track what these agentic harnesses are doing and when you look under the prompt and See the tools that are there and those little subtle keywords, you know back in the day It was ultrathink or something like that would trigger the the thinking modes It gives you a little bit of a hint about where we're headed. So what I'm talking about today is

what's called Claude Teams. And this is a feature that they enabled under feature flag. It is still under feature flag and it was released in early February and it's called Claude Teams. And they've been changing how you move your mental model from like sub-agents to teammates. So traditionally you might say, spawn a sub-agent to do this particular task. It'll take on the context load, the tokens and the burden out of the main session, push it out somewhere else and then report back.

You can even have Claude for a long time now, name agents in different windows and have them talk to each other. This is basically pulling that up just a little bit more. And the way it works now is that for every session that you start, if you are feature flagged into this particular new way of working, you have a main level team called sort of a team lead. And this team lead can dynamically spawn.

not sub-agents, but teammates. And the reason they're not just sub-agents, but they're teammates is that they can share a task list so they understand the goal and each of them can talk to each other as well. So this kind of gets back into what I called the Sims. Like this was very early on in the AI days where things such as like Crew AI are other things that I kind of built called experts AI built on OpenAI that where people just, you know, it was all basically persona modeling.

back in the day and that's kind of coming back around again because we now have this sort of larger harness that will manage these things. So the thing that I think is interesting is that they recently came out I believe earlier this month in early October with what are called mods. So mods are basically packaged through the plugins but they can do something a little bit different. They're primarily focused on changing the user interface and they can do that both in the CLI as well as the desktop app.

And so that could be novel things such as tracking your token usage or better yet, keeping track of your teammates, which ones they're talking to, clicking through them and seeing how they're orchestrating longer horizon work together. The thing that caught my eye is, that mods no longer execute in a sandbox. So this should just really perk Chris's ears up. The traditionally, if you were on Claude.

I'm fine with things

as they're progressing, you know, sitting here with burning coming with a coffee cup.

It's going great.

Things are fine. So if you, you know, there was typically like a firewall here, right? Like you had folks like me who have my entire product team using Claude Code CLI, because Claude desktop, if you use skills that relied on your computer being set up a certain way, a natural thing from the developer experience, any skill that you had would run in a sandbox environment.

And I think it's incredibly fascinating now with mods being able to run outside of the sandbox that you're going to be able to have Claude do things. And if you've ever run a long horizon task and see Claude do something weird, I swear, I swear to you, I was doing some work the other day where I was building a mod and it actually

patched its own session information to bring in the mod to test it. Right? So it's like it, it knows where its memories are stored on disk and it changed

Right.

its session while it was building the mod so that it could test the mod. things are going to get really weird soon.

And that that part about not being sandboxed, I read up on it, and it's the first sentence in the security documentation. So

Mm-hmm.

at least they're pointing it out.

Yeah, right. And it's behind like, this is the thing that helps you build user interfaces, which, you know, I used to be a graphic designer. love user interfaces. I love customization. I also love not running outside of a sandbox, right? Like to me, to me it's normal. So like one of the things I would put out to y'all is like, would you argue or would you be comfortable letting AI traverse your computer? And I'll frame that around the question, right? So recently here,

I believe I have to go back and look at the news, but Amazon blocked Meta's Muse from making purchases on amazon.com. And this is something we talked about at the Product Fest last night as well. And that makes sense, right? Like I'm not a big Open AI user, but I would imagine their consumer tools do the same thing, right? They create a sandbox, they create some form of compute. Maybe they're using Lambda Web Functions now.

and that Lambda Web Function or other Amazon tools spins up a sort of a trusted browser and it will proxy your credentials through it. If you're Elon Musk giving it your credit card information to have it going out making purchases for you, please don't do that. Just don't do that with any system. But you can kind of see this natural progression of things are hard to do when you take somebody's personal computing and extend it out into the cloud. You have to do all sorts of things. So if my computer had a VPN connection,

and I wanted to help automate a piece of work as some sort of general knowledge worker, you would imagine that it would be better off if I ran that on my local computer. And I think you're gonna see this trend of my browser locally being run by your agent. And then guess what? Nobody can tell the difference if it's me or Claude using my browser. So there's no such thing as blocking purchases anymore from an agent.

Did I get you right that you think that Amazon blocked Muse out of concern for the consumer?

I didn't make such a claim.

Good question.

ha ha.

They didn't want to lose the advertising revenue.

yeah that makes sense. And they also block the same for Grok Bot now. So you mentioned Elon. He apparently

Mm-hmm.

what is it called these days? Not tweeted, he shitposted that he's gonna start

Well, it's X right so

Zittered, yeah, I don't know.

you just

call it the cat butt

So yeah, he's starting an Amazon competitor now, I think.

think about what's on your computer that now Claude has access to. Tilde AWS slash SSO slash whatever that token file is, right? It can go and like,

Mm-hmm.

you logged into AWS, let me go help you, and it will figure out, here's the production account and here's my admin key. And you didn't give it those admin keys, but it found them.

and logged in. And same thing with, hey, I can get into your password manager. Let me go and transfer some money out of your you know brokerage account. You know, I was like, yeah, you've got money in his 401k. Here, let's use that to go buy something. It congratulations, you just now hit massive tax penalties for an early withdrawal.

This is just, you know, and we saw this a little bit earlier in the year when OpenClaw came out, right? Like, there were cases where people's OpenClaws were like, you know, making crypto moves that they didn't authorize. Frickin' VP of AI safety at Meta had her entire mailbox deleted while she was apparently running across the apartment to unplug the damn Mac Mini that was running OpenClaw.

So right, like I don't think things have gotten any safer in the last ten months.

Well, and it is weird because in recent news, it's all about AI broke out and did this AI broke out and did that. And I find it fascinating. You know, I am for one, a big proponent of computer control, but that is a responsibility that I think, you know, we as engineers get to see the sort of glimpse or the future. Right. And we also know that how many of us have helped make their families printer work. Right. Like, so

This is not going to, I want it, I want it, but it's probably going to be weird. I maybe predict that they will roll this back, right? Because it's hard to imagine why you would do a mod system for UX enhancements and bring this capability in. it, wow, like it's, I like it and I'm also afraid for it because from a product perspective, when you see...

end users using your system, they're always almost doing it wrong. Right? So like there's that GIF joke where you like hand a glass of water and they're trying to drink it out the bottom of the cup or something like this, or they're pouring like just end users will do the wildest things. And then you give end users Claude, they're going to do doubly wild things.

And and looking at that Amazon versus Perplexity legal case as well. Well, it doesn't matter if it's the agent, it is the user that's doing the actual shopping as well. So this can definitely have legal ramifications when these mods start moddin'.

Because you will be prosecuted. Sam Altman will not, but you will, because you haven't donated enough to the right political causes.

that brings us to the next segment, which is it's time for us to take a short break. But that for you means sponsored messages. We'll be right back.

Alright everyone, welcome back. It is now time for burst mode. And as usual, we are gonna start with Danielle.

Yeah, so something that I saw this week that was really interesting, and I'm not sure it's gonna be super widespread outside of the PHP community, is this developer Eric Mann, he gave a closing keynote at a PHP conference in Texas. And disclaimer I am good friends with this guy, but he did not ask me to talk about this or anything like that. It just had a really good impact on me.

it's like about a half hour long talk. It's called The Commit You Can't Take Back. And it's not a lesson about morality or, you know, like the purity of, you know, how some people act in tech and elsewhere. it's more along the lines of like, Hey, you are a builder, you're gonna be building some stuff. And in your head, you're gonna be thinking, you know, A, B, and C are going to be the use cases. but what happens?

when someone wants to do D with it. So one of the examples that he gave was he was working for a defense contractor and they were building tracking for boats on, you know, some river or something. And him and his team were thinking, you know, this is for safety. This is for like making sure that like, you know, the barges keep moving and commerce keeps going. And like that was what they were thinking. And then one day a PM just had a drop down

That added destroy to the boat. And his point was not like, you know, a judgment on, you know, this or that is better. It was kind of like you really need to know what your comfort level is for how people might use the tech that you built. And once that line gets crossed, you need to think about how how you're going to react to it.

And I just thought it was a a really great talk. it's something that happens a lot in our industry, probably in other industries as well. And I I think my yelling at the cloud for this is just more I I would like to see more conversations like this. it was really great.

definitely feel the, you know, ooh, have I done something incredibly wrong? But I'll I'll save my you know confessions for the war crimes tribunals.

I really want to talk about the crappy state of MCPs. you know, I would have thought, you know, it's year 2026, you know, everybody's doing MCPs that you know we would have better support, but my god.

Like AWS's MCP is just a wrapper around a little Python environment thing. It doesn't really do great multi-account support. Claude, you know, no, you have three Google Gmail accounts. Sorry, I can only ever connect to one and read the emails. Outlook, you know, requires you to have a Copilot credits to actually run their MCP. yeah, so right, like I'm generally completely

frustrated at this and so I was like, fine, I'll do the Sauron thing and I'll create one MCP to rule them all. and so now I've got this hundred dollar, you know, a month infrastructure running in one of my AWS accounts just to make this all work. But

I can at least ensure that all my GDPR Article 9 data is staying safely inside the EU and is not getting transferred over to the US to be inferred in Anthropic's Cloud. Because now I have all the classifiers and everything else, you know, running locally and running deterministically. And so AgentGateway.dev, really cool. More to come.

Chris, it sounds like I could interest you in some computer control and full access so that the AWS CLI can just run from your computer and do everything that you need to do.

Funny that the last

thing that I did open source was Claude YOLO, which basically is my container infrastructure that allows me to run --dangerously-skip-permissions. And so I do have, like, yeah, here's the AWS account, you know, the one that I'm allowed to actually do anything in because we're topic that shall not be named. But right, like here's your admin, go go dude, go deploy, and it has access to one account.

You know, those credentials expire every four hours. and yeah, I'm I'm reasonably happy giving AWS access or giving Claude access to one of my AWS accounts. My home folder? No, not so much.

I guess you have one agent building and then you used another one for grading, which I guess is half a loop. Ken, I think you're on something similar, recursive self-improvement, right?

So the recursive self-improvement or what we're calling RSI now is the loop that keeps on giving. the, you don't know, if you don't know about what OODA loops are or whatever ADLC or product product SDLC, it's just a loop. It's always a loop. But the recursive self-improvement is something that has been happening. It just may not be so obvious. So every time Claude comes out with a new version and it

understands how to write its own skills and those skills are used by you. There's a form of self-improvement there, but you can actually turn those skills backwards towards Claude. So the idea is, is wherever you kind of make a human decision, you can use AI to set it to a goal. You give it enough tools to look at the data, to make all these decisions and it can pretty much do things a lot faster than you can. So the recursive self-improvement, you're to see a lot of it. I've seen some startups are coming out now saying they focus on it.

A fun thing, it's not a product you can buy, but if you want to buy RSI, can sell it to you. It's just a, it's just a, it's a technique and it's a process and it's incredibly powerful. So if you're a product person and you've thought about things such as evaluations, a lot of times over the past year, folks think of evaluations as something like they would do that we engineers called application performance monitoring, right? Like it's a thing that happens at the end of the line while you watch the system run.

And sure those are useful, but if you bring evaluations further into the product, sort of discovery and development life cycle of a system, you can use those as rules for the agent to improve itself on anything that you're doing. So this is actually what the, this is actually what all the cool kids and the AI companies are doing themselves, right? They're using AI to help write their AI to write the systems to improve it. And it's, it's going to start catching on really heavily, I think in like for us end users as well.

I know Anthropic, they I think it was a white paper that they called When AI Builds Itself. and they're calling RSI, I had to look up exactly what it was, fully autonomously designing and developing its own successor. Which I guess is exactly what you're talking about, where it's it's then using a previous model to build the next model in that case.

And in the case for us, that's usually just the prompts, right? So it's

a, or the, some cases, the system or the process that the prompts also operate under.

Cool. I also saw a number that over 80% of the merged code at Anthropic is written by Claude today. So

I'm surprised

makes sense.

that low. you know.

Well, we're surprised to still be employed based on what Dario has said.

I don't know, after my comments when I might not be, but yeah.

True. I guess define employed.

So, what I wanna know is when somebody finally starts doing RSI and then just creates a loop that doesn't have an exit condition, you know, what is the while true you know version of RSI, and then it just keeps going for till it either runs out of tokens or eliminates us.

That's it. As long as you can pay for it, it'll keep going.

Stop writing prompts. Someone has

to write the first prompt, right?

It's the goal, not the prompt.

Let there be light, that was the first prompt.

Yeah. Alright. So Ken is racing to not write prompts. Labs are racing to build AI that builds AI. And you, Chris, you want to have data kept in Europe. And I think the one European lab still in the race actually named its newest model after a cat meme. And on our last episode, I asked.

bit jokingly, if Mistral was still around. And now at the beginning of October, they actually released Mistral Large 4, which is nicknamed Le Chonk it

let's get to the name first. It started as a fan joke. in June, after Mistral renamed Le Chat to vibe, people on Reddit and X invented a fake Mistral model called Le Chaton Fat

the fat kitten. and instead of getting upset about it, Mistral, I guess, just played along. Arthur Mensch, the CEO, he said that it was really called le gros chaton, the big kitten. But anyway, now

they've launched Mistral Large 4, which they are nicknaming Le Chonk about a trillion parameters, and it's using what's called mixture of experts.

which basically means that even though you have to have all of it in memory, you have this router that picks between the different subnetworks within the model. So a trillion parameters in total, but it's using around 50 billion to do any work on a given token. which is pretty cool. So they've announced it, but as of now, it is still not.

available. I think Hugging Face says that it should be available October 31st. And the license still says coming soon. So I guess we'll have to have to wait and see. But

So will it be in Bedrock in twenty twenty seven or twenty twenty eight?

It will not be generally available. It might be available eventually in Bedrock. I guess we'll have to wait and see.

put a challenge out. I want somebody to name a model Big Chungus because I think that should come back.

Well send a message to to the CEO of Mistral, Arthur Mensch. He might play along. That's Mistral Large 5 maybe.

but so yeah, I think that's pretty cool. That there might be I don't know if it's gonna be you call I think you called Mistral tier three last time, Chris. let's see if this is tier one, at least tier two, hopefully.

gotta be available first. And as as as I have to point

Available, shveilable.

out to all of my colleagues on in America who are like, why can't we do this in Europe? I'm like, because we don't bulldoze our homes in national parks to build data centers. So we we may

Ha ha ha.

have a great model, but if we don't actually have the massive polluting infrastructure to run it, then is it really useful?

Yes. But on the same note, there is another model that got released beginning of October, another European one called Aleph Alpha Kolibri-1. So this is a small one, seventy-eight billion with about three and a half billion active parameters trained in Germany and Finland, and it's aimed at German language and regulated industries. So part of this entire German

data sovereignty type thing, I would say. So

Much like the XKCD random number generator that only returns four, this just only returns nine.

Also, how much would you trust a model that only has like 21,000 downloads? It's like picking the NPM package that just came out last week with 10 stars.

Ha ha ha.

Don't be that guy, Ken.

We have to look positive at things now. Europe has models too.

Alright, so I guess we'll do what we normally do when we get towards the end. I want to know what you're looking forward to now. Chris, I know you want to find out if you're still employed. I guess that's number one. But do you have anything else you're looking forward to?

After that last comment, I'm not.

What am I looking forward to? I am looking forward

to leaving the computer in Portugal and going to the UK with my kid and actually like seeing Windsor Castle and Tower of London and and all of that stuff and kind of just hanging out with a boys' weekend.

Wasn't that the same thing you looked forward to last time?

I thought last maybe? I don't know. When did we last report?

Feels like I've heard it before at least.

Gotta watch the old episode to find out.

Yep. Ken, what are you looking forward to?

Well, you I'm a long-term thinker and I like to think about strategy and things like that. My forward is just very short-term. It's right after this call, I'm going to go to Chick-fil-A and get some dinner. That's it. That's it.

That's

Putting the blinders on, it's just

I miss Chick fil A.

the next half hour. That's the only thing in front of me.

Perfect. Danielle?

Yeah, kind of similar. this weekend I'm gonna go see that Elizabeth Holmes movie, the Nathan Fielder one.

So I'm curious to see how entertaining it is.

Do the look.

that's gonna be super interesting. Yeah, do let us know. I think the trailers are amazing. it's Nathan Fielder is pretty cool as well. So what am I looking forward to? I think for the past few episodes I've been talking about upcoming events that I'm looking forward to, which isn't super interesting anymore because that's that's always the case. Bunch of fun events happening all the time. So what I'm looking forward to right now.

Is and this might might frighten a few people, but I'm actually looking forward to winter. We are now in that weird

period where it's pretty cold and wet and dark. I know, I know, Chris. It doesn't look that way for you, but here in Sweden, okay. Yeah, it's

It it's cloudy. Yeah.

cold and wet and dark pretty much the entire time.

I like it. It's time

again. It's time again for the cold and dark.

Yeah. So I wanna get snow. I wanna get some some cold outside so we can get some some proper winter for once. That's what I'm looking forward to.

there a word for that Gunnar? Is it like a huga or something like that? something that sums up just being cold and cozy.

you mean hygge?

Yeah, there you go.

That's well, it's a Danish word which means to be cozy. so it can happen during summer or it can happen anytime. But yeah, hygge is nice. And one more thing, since we're talking near term as well, I'm gonna hit the sauna after this, so I'm looking forward to that as well.

Are we gonna do one of these live at re:Invent?

we should. re:Invent is coming up very quickly now. Is it five weeks until we're

kind of there? So we probably should.

I don't think Ken's

going though. Ken will have to phone in.

All right, so we started with PHP finally getting famous. If you haven't patched yet, I think it's about to get famous on your servers too. So do patch everything. Yeah. All

You're about to get famous.

right. So this has been fun. We've covered a lot of stuff. And if you haven't already, subscribe and

Like and leave a comment. Let us know what you think.

And you can subscribe wherever you get your podcasts. With that, I want to thank Danielle, Ken, and Chris for joining again. And thank you to all our listeners. And we will be back. Bye bye.