Vibe coding, and the rise of the product engineer

00:06James Hartt Hello there. So this is, the second episode of the AgentSee podcast and, Abrar's going to take, sort of the main role of this. This is, this is all about his, good work at, GradeDrive and another startup. So yeah, it's, episode two, vibe coding and the rise of Product engineering.

00:22Abrar Mahmood Yep. Yep, alright, so I'll just start off by doing a little bit of a summary of what I've been working on recently. So I work at a startup called GradeDrive so we use AI to help schools and Tuition centres

00:37James Hartt Thank

00:38Abrar Mahmood Mark

00:38James Hartt You.

00:38Abrar Mahmood Exam papers I work with a non-technical co-founder to build it And of course there's a lot of sort of challenges in that in terms of ensuring velocity and keeping the product going Forwards so we'll go into that a little bit. I also have been helping out at a startup just sort of part-time Where we've got a similar sort of idea where we are working with product domain experts And some of the work I've been doing is to sort of like introduce some enablement on that so they can move as fast as possible

01:19James Hartt Yeah.

01:19Abrar Mahmood Very important in the sort of startup environments that we of course move as fast as we can so it is particularly relevant in these Sorts of places I'll sort of be going into like what we could ship before versus

01:33James Hartt Thank you.

01:34Abrar Mahmood After sort of some of the things that I've been working working on So in terms of how we used to work it was it was very much sort of like the pre-AI sort of ways Of working where the sort of middle part that takes the longest which is the engineering

01:54James Hartt Yeah.

01:55Abrar Mahmood Is AI-assisted so it would go from sort of like ideation of like going through getting the ideas of what we're going to be working on so that's Very much sort of like product team led and then it's validated with an engineer where it's then sort of gone into tickets with some Technical implementation and then like some time has been spent with the engineer to actually implement the work the first sort of Introduction of AI was to sort of assist that so of course that was quite a few years ago and then you still kind of have that same sort of that long Feedback loop of like engineer spends a whole bunch of time building to the ticket spec and then like oh actually like we've now got that Into QA and it doesn't quite meet what the product team or even the customer quite expected so there's quite

02:41James Hartt Yeah,

02:41Abrar Mahmood A long sort of feedback loop around that

02:43James Hartt And some balls dropped. Yeah.

02:45Abrar Mahmood Yes and I think I think like that is of course still better than pre-AI because the part that took the Longest which is the engineering is now contracted and it's very AI-assisted which

02:59James Hartt Yeah.

02:59Abrar Mahmood Which is of course AI already showing its merits sort of back then so I think like that's kind of how things Have been for quite a while and I think a lot of what I've been working on sort of looks at reshaping that and

03:15James Hartt Yeah.

03:15Abrar Mahmood Making that iteration loop as quick as possible So I think I'll be going into how we can sort of make that practical and make that safe but I think one thing that would Be good to sort of highlight as well is sort of vibe coding in general so whilst doing that and working in that way We're still maintaining quality because pieces of work are going through these different sort of of checkpoints You could call them so and there's levels of validation that are in place for example when engineers are Vibe coding Or Using AI to build whatever they're building it still goes through the checks of pull requests and there's still quality standards we're Meeting

04:03James Hartt Yep.

04:03Abrar Mahmood And when tickets are written up they're still sort of engineer-led suggested implementation details so all of that is still in place The next sort of thing that I found was happening a lot was the whole idea of vibe coding and when that was sort of introduced a few Years ago and that kind of like scrapped the entire thing

04:25James Hartt Yep.

04:25Abrar Mahmood And was like okay I've got a I've got an app I want to build I'm just going to go in and start knocking some stuff Out I'm going to be prompting the agent to Build whatever my heart desires and quite often It presents itself as being a very sort of like motivational Approach because the feedback loop is so quick and you're getting this sense of progress which is kind Of like feeding into itself to actually get you to really work

05:00James Hartt Yeah, it's

05:00Abrar Mahmood And

05:00James Hartt One

05:00Abrar Mahmood Spend

05:01James Hartt Dopamine

05:01Abrar Mahmood The time

05:01James Hartt Hit after the next. Yeah,

05:03Abrar Mahmood Yeah exactly

05:03James Hartt Yeah

05:04Abrar Mahmood So you're getting that dopamine hit and you really just want to keep going and sort of like work hours on end and Like spend loads of time building things and I think like when I first got into engineering I was very excited to be building things and I think like I kind of was getting that same dopamine hit back then as well and And I think When someone non-technical now finally has the power to build Without having that sort of barrier of Engineering and that engineering bottleneck, someone from a product team can now go in and vibe code and they can really start Building whatever sort of comes to mind.

05:49James Hartt Yeah.

05:50Abrar Mahmood And I think that was really good because it kind of sparked a lot of Good new ideas that people have come up with and I think equally it would also tell People with lots of ideas why whatever they're building might not necessarily work because

06:08James Hartt Yeah.

06:08Abrar Mahmood They can get straight into that prototype

06:12James Hartt And yeah, if it's got the full context of the code base, it's going to make, well, certainly the models now are going to make good decisions of why that's a terrible Idea, given what's

06:21Abrar Mahmood Yeah.

06:21James Hartt Already there yeah

06:22Abrar Mahmood Yeah I think the other really good thing about it is if you want to build a whole new feature and you can vibe code it like because that iteration Loop is so quick you can get to a point where you've got something that's roughly working and you might actually decide actually this doesn't Really quite solve the problem I thought it would so like you can without having that engineering bottleneck you can get that point really quickly which

06:46James Hartt Yeah,

06:46Abrar Mahmood Is

06:46James Hartt And then

06:47Abrar Mahmood Which

06:47James Hartt Just

06:47Abrar Mahmood Is really good

06:47James Hartt Cast it aside. Like that's like one of the great things is because it is there's no real cost particularly for sort Of trying out these ideas you can get it to eight percent and be like oh no actually that's just terrible like it's a terrible

07:02Abrar Mahmood Yeah yeah

07:02James Hartt Idea we should we should not carry on rather than yeah try like not knowing and having to force it through the pipeline

07:10Abrar Mahmood Yeah.

07:10James Hartt And yeah like that it definitely the trial and error loop is so good

07:18Abrar Mahmood So I think the other thing is when going through this whole vibe coding approach I found that vibe coding A new feature in an existing code base was actually really good for the idea of prototyping new features so one of The approaches was essentially product team approaches us with this new feature we want to build they would they would Essentially just check out a new branch which is purely for the purpose of a prototype of a feature and they'll just Knock something out and be like okay this is roughly what I want it to look like but you as an engineer you'd have to go and build that Feature from scratch because what they've built is probably just like not following the sort of engineering best practices and all the

08:06James Hartt You.

08:06Abrar Mahmood Experience that sort comes with it as well so that was an approach that we did take in various sort of Features that we were building so like I would often work with a prototype that the product team would vibe code and then actually Building it meant sort of starting from scratch like you couldn't really take that prototype it was kind of a throwaway thing

08:27James Hartt Yeah.

08:28Abrar Mahmood But because you get that iteration you then have the product team develop confidence in what they actually want to build so it is a Useful thing to have it was also really good to kind of transfer from the product team to the engineering team of like how This feature is actually supposed to work so you might have like some mock buttons and whatever else so like one of the features involved using AI to Extract data and like present it to the user So the prototype would like just have like the UI for what you should be doing then when you click a button it doesn't actually do anything just transitions To the next screen

09:05James Hartt Yep.

09:06Abrar Mahmood And then like present the user with like what's being like what was extracted like in this sort of mock piece of data so it's Really good for that transferring that sort of understanding of what the feature would be doing I think of course the real problem With that or not the problem so to speak I think it's more the difference between when we when we used to be doing that Versus doing that now is when we've got product the product team working on these prototypes back Then the models weren't quite as good as they are now. So they would kind of like give the vibe Coded feel of what's being built and by that I mean it wouldn't be following the practices that are already established in the code Base it wouldn't be very good at self-iterating like I think like Ralph loops was a really big thing that was

10:01James Hartt Yeah.

10:01Abrar Mahmood Like around but to be honest like now that is like completely dead like

10:05James Hartt Yeah, the Ralph

10:06Abrar Mahmood Like

10:07James Hartt Wiggum

10:07Abrar Mahmood I think that

10:07James Hartt Loop,

10:07Abrar Mahmood Agents

10:07James Hartt Yeah.

10:08Abrar Mahmood Are just really getting smarter at what they're doing so I think what that means is that like because the agents are sort of Iterating and just getting so much better at what they're doing a lot of these workarounds I would I see Ralph loops as being like a workaround For

10:23James Hartt Yeah, absolutely.

10:25Abrar Mahmood For like what an agent should really be doing Where time spent at the moment for me specifically is firstly making it is the enablement Of the product team

10:40James Hartt Mm-hmm.

10:40Abrar Mahmood So if they've got an idea I want to make sure that they can go ahead and build whatever it is that They're looking to build and to do it in a way that fits with all best practices within

10:53James Hartt Yep.

10:54Abrar Mahmood The actual code base so there's a few ways we've been doing that so of course like there's course all the agent documentation in the Repo itself there's Like in terms of like security There's a really big thing there where essentially you want to make sure that when the product team is working on these Features we make sure that the for example like we have a particular role with like the like the cloud Infrastructure environment for that particular user

11:26James Hartt Yeah.

11:26Abrar Mahmood Which has read-only access to a particular environment so they can use that as like a database to work with

11:33James Hartt Yeah.

11:33Abrar Mahmood So there's things like that and making sure that their environment is enabling them to do what they need to do Without actually like compromising ourselves a little in any way

11:46James Hartt Yes, and the agent knows that as well, so they're happy SSH-ing into a box, knowing that it's not prod they can do Whatever they want write secret queries like yeah mutate

11:57Abrar Mahmood Yeah

11:57James Hartt Things like yeah that sounds like the way to go like make sure that it's completely hermetically sealed Right.

12:06Abrar Mahmood Yeah yeah absolutely and I think like the other part of that is just like the ways of working itself so like my co-founder he's Now he's still using the Claude desktop app he's not very comfortable using a terminal at all

12:19James Hartt Yeah, right, yeah.

12:20Abrar Mahmood So he's using the Claude desktop app however I've got a particular skills in the project that I've Told him to use that for example spin up a worktree

12:33James Hartt Yeah.

12:33Abrar Mahmood Which gives us an isolated environment for him to work on whatever it is he wants to be looking at at that particular time

12:41James Hartt Yeah yeah and

12:41Abrar Mahmood And

12:41James Hartt Does

12:42Abrar Mahmood Spinning

12:42James Hartt It

12:42Abrar Mahmood Up

12:42James Hartt Run builds and like

12:44Abrar Mahmood Yeah

12:44James Hartt Yeah and

12:45Abrar Mahmood So

12:45James Hartt There's

12:45Abrar Mahmood It essentially

12:46James Hartt There's

12:46Abrar Mahmood Does

12:46James Hartt Some waiting around like when he's iterating like he can do other things and

12:51Abrar Mahmood Yeah.

12:51James Hartt Sort of

12:52Abrar Mahmood Yeah

12:52James Hartt Make it make a coffee or whilst it's spinning around and executing his idea,

12:57Abrar Mahmood Yeah

12:57James Hartt Like...

12:57Abrar Mahmood So the skill essentially does what you as an engineer would probably be doing working in a sort of in a web App so

13:06James Hartt -Hm. Right.

13:06Abrar Mahmood Take for example a Next.js app you're probably gonna run npm run dev but before that you'd probably be installing doing an npm install and Like getting all your other dependencies so I've got a skill which is basically just worktree up

13:21James Hartt Right, yeah.

13:21Abrar Mahmood And I've got the skill to be smart enough to basically spin up an isolated environment For the particular for the particular feature that that my co-founder wants to work on

13:37James Hartt Right, yeah.

13:37Abrar Mahmood And they're all self contained as well so you can have multiple worktrees running and they're all running their own npm run dev on their own Ports

13:46James Hartt Right, yeah.

13:47Abrar Mahmood And they've all got their own databases as well so they've all got their own Docker Compose files to run their own databases so So it's all self -Contained And I think that's one of the big things that has really propelled my co-founder to actually work on This stuff really quickly because the iteration loop for him now is really good. So it's as fast As it is for us. So

14:15James Hartt Yeah.

14:15Abrar Mahmood He doesn't need to be going through that massive sort of like loop of like idea validate with engineer ticket Wait and then actually what I've built doesn't quite line up with his vision he's literally working on the worktree and talking To his agent to say okay this is what I want to build go ahead and build it he gets that a few minutes later on his web Browser and he can go in and change things as quickly as he wants

14:46James Hartt Have

14:46Abrar Mahmood Of course

14:46James Hartt You had any instances of it going completely off the rails

14:52Abrar Mahmood So

14:52James Hartt And him

14:52Abrar Mahmood I

14:52James Hartt Having

14:52Abrar Mahmood Think

14:53James Hartt To give up a half a day's worth of vibing

14:57Abrar Mahmood Yeah I mean I think there's been a little bit more than that so there was a particular feature that that was vibed. Which was fundamentally not implemented correctly

15:09James Hartt Yeah.

15:10Abrar Mahmood So this is essentially so at GradeDrive we've got this feature which is to take a PDF file of All your scanned exam papers So if it's one large PDF file of an entire class's worth Of scanned exam papers, the feature is to automatically split it by student. So if you've got

15:33James Hartt Right, yeah.

15:34Abrar Mahmood Students' worth of exam papers, it will identify the start and end of each student's papers. Papers so when it was vibed originally it worked but on a larger on a larger set on a Larger classroom

15:50James Hartt O(n³). Yeah. No.

15:54Abrar Mahmood Well

15:55James Hartt No. Yeah.

15:55Abrar Mahmood Yeah I mean not quite not quite that being the problem I think

15:59James Hartt Yeah.

16:00Abrar Mahmood I think in terms of complexity it was actually all right. I think the main issue was that the whole logic was running on The user's browser

16:09James Hartt Right, yeah.

16:11Abrar Mahmood Rather than being implemented to run asynchronously on the backend.

16:16James Hartt Yeah.

16:17Abrar Mahmood So one of the

16:18James Hartt I mean,

16:18Abrar Mahmood Key

16:18James Hartt That's, yeah,

16:18Abrar Mahmood Issues.

16:19James Hartt That's a classic with sort of, like, heavy stuff happening

16:23Abrar Mahmood Yeah.

16:23James Hartt On browser. Yeah. Yeah.

16:25Abrar Mahmood Yeah, so I think like one of the immediate issues was if you refresh your browser you kind of lose that state

16:32James Hartt Yeah. Yeah.

16:32Abrar Mahmood So that was one of the immediate problems but of course the other problem is on a larger document you just like it just crashed the User's browser and like you're immediately dependent

16:43James Hartt You're relying on the specs of

16:45Abrar Mahmood Yeah

16:45James Hartt Someone's like stressed out potentially on memory and so on laptop yeah

16:51Abrar Mahmood Yeah so like I think like

16:52James Hartt Not being able to stream things or yeah like I said not being implemented correctly

16:57Abrar Mahmood Yeah so you're kind of at the mercy of a school teacher who's really just trying to trying to trying to help themselves by Using GradeDrive and like they've got a laptop in front of them and the fans are probably kicking off at like full speeds

17:12James Hartt A Chromebook, yeah.

17:13Abrar Mahmood A Chromebook and like draining its battery and then all of a sudden you get an oh snap on your Google Chrome Browser

17:21James Hartt Yeah,

17:22Abrar Mahmood Or whatever

17:22James Hartt Yeah.

17:23Abrar Mahmood And yeah it's just I think the intent of the feature was very good it's a really good feature but I think the technical Execution of it if it had the correct guidelines in place that's an example of something that could have been vibed Correctly and I think going forwards I don't expect that to happen again because we've got we've got the Correct Agent guidelines in place to talk about implementing async workflows and making sure they are recoverable and also Immutability as well so that's obviously a big thing and idempotency so like we've got that explicitly called out in our In our agent guidelines

18:08James Hartt And

18:08Abrar Mahmood I

18:08James Hartt What

18:08Abrar Mahmood Think the other

18:08James Hartt If

18:08Abrar Mahmood Thing

18:09James Hartt Have you noticed it deferring and saying like you might want to talk to Abrar about this or like have you noticed it sort of not sort Of giving the user what it wants after these things have been added

18:21Abrar Mahmood I think that's a really good point because I think that's an example of something I've still not really seen it do of course with agents is very keen To go ahead and give you something

18:33James Hartt Yeah give you

18:34Abrar Mahmood It's

18:35James Hartt Throw some

18:35Abrar Mahmood Not

18:35James Hartt Tokens

18:35Abrar Mahmood Very good at

18:36James Hartt At

18:36Abrar Mahmood Saying

18:36James Hartt You

18:36Abrar Mahmood No

18:36James Hartt Yeah yeah and prioritizing

18:38Abrar Mahmood Yeah

18:39James Hartt Or getting the bigger picture yeah

18:41Abrar Mahmood Yeah so that's definitely one of the things that I think is a little bit tricky

18:47James Hartt It's got

18:47Abrar Mahmood To

18:47James Hartt It

18:47Abrar Mahmood Get

18:47James Hartt Feels

18:47Abrar Mahmood Right

18:48James Hartt Like that it's that classic game of whack-a-mole right you're just You're just going to be like, yeah, you're going to have, yeah. They might get smaller and smaller as you go along and like more and more subtle, like

18:59Abrar Mahmood Yeah

18:59James Hartt Subtle issues.

19:03Abrar Mahmood One thing where things have moved on quite a lot for when I'm using agents is the idea of it being Able to self-iterate so we spoke about the whole Ralph loop thing earlier but I think another thing is like now that We've got a ton of MCP useful MCPs available that you can just go in and just give it access to whatever It needs to have access to and then like also the other thing which has been really powerful for me personally is The whole integration with browsers and

19:38James Hartt Yeah.

19:38Abrar Mahmood So you can actually get it to of course build it build the feature and then self it and then self-iterate and validate What it's built to make sure it's actually functioning

19:49James Hartt Yeah, actually clicking through the screens, making sure the buttons aren't off screen or like yeah, that kind of stuff. Yeah.

19:56Abrar Mahmood Yeah and having them screenshots come back to you as well to show you that this is actually working is really powerful so that's that For me is something that's a real big development and I think that's why that's like where literally the agents are seeing the Work

20:14James Hartt Yeah.

20:14Abrar Mahmood So I think that's

20:16James Hartt Yeah. Yeah, I since

20:16Abrar Mahmood Definitely

20:17James Hartt Our last episode, I played a lot with the like yeah cloud sorry the claude browser extensions and a few other ones and it is just amazing what You can do and it's got it's obviously got access to the DOM it's obviously

20:30Abrar Mahmood Yes.

20:30James Hartt Got access to like the actual sort of visual stuff it's making sort of the right decisions you can sort of loosely but Also fairly resiliently automate basically any browser workflow

20:43Abrar Mahmood Yeah.

20:43James Hartt That you can think of because it's kind of behaving like sort of like a user would yeah so it just yeah like I said it just Enables so much more Closing the delete

20:55Abrar Mahmood So I think one of the things that I've actually been very impressed by is, So when I'm working on any of these projects, I'm actually running it in a headless Virtual machine. So I'm running it on a headless Chrome a headless browser And it's still able to make use of that so of course that's not using the Claude Code browser extension

21:22James Hartt Yeah.

21:23Abrar Mahmood And that's like that's literally using a headless browser and taking screenshots and I think like the self-iteration on that is like a Million miles ahead of where it was I would say like a year ago

21:34James Hartt Yeah.

21:35Abrar Mahmood That's basically been my main workflow and web apps that I've just been just like working on a headless container and just getting it to knock stuff out

21:45James Hartt Have you, like, any of the GradeDrive or the other stuff that you're working on? Do they do any sort of mobile stuff so is there can you I'm guessing can you set up the headless browser to be x size and make sure that it's sort of get making all the mobile workflows like Particularly good like that's my bias from doing sort of lots of mobile work in the

22:06Abrar Mahmood So

22:06James Hartt Past

22:07Abrar Mahmood So web apps yes so I do test on multiple viewports

22:12James Hartt Yeah

22:12Abrar Mahmood So the I'm I've essentially got a fixed set of viewports that I always get the get the agent to validate the Work on I've not done any actual sort of like iOS native or any Expo or any sort Of Android stuff

22:30James Hartt Yeah

22:30Abrar Mahmood Using

22:32James Hartt Yeah that was a sort of follow

22:33Abrar Mahmood This sort

22:33James Hartt -On question

22:33Abrar Mahmood Of workflow yeah

22:35James Hartt It's like how easy is it to do how easy is it to do these workflows with native when you sort of do have to build a binary and have to

22:42Abrar Mahmood Yeah

22:43James Hartt Have to wait a decent amount of time and have to get it to run on a bleeding Mac. Like not just a nice headless virtual machine in Linux.

22:53Abrar Mahmood Yeah so I think yeah that's an interesting one so I do have a an Expo app called Article FM which I've built And that I essentially would describe as having vibe coded my experience with that was the first thing I thought is does Expo have an MCP okay yes it does I think I had to pay quid a month just to be able to

23:14James Hartt Right.

23:14Abrar Mahmood Use their MCP but it turns out it wasn't very good Like it just and I guess that question is that what does not very good mean but like it was just struggling To Self-iterate So again like I think that was down to the MCP just not having enough visibility or not Giving the agent enough visibility of like what it was doing like I think it just like took ages to like find places to click on and It eventually just I just eventually fall back to using like one of the CLI tools for like clicking on simulators I can't remember what it is Off top of my head I'm not really an iOS developer but I think there is a CLI tool where you can pass coordinates to click on

24:00James Hartt Yeah. Yeah, the build, it's just all the iOS, the Xcode build tools,

24:05Abrar Mahmood Yeah so that's kind of what I ended up doing it was more painful to be honest

24:11James Hartt Yeah.

24:12Abrar Mahmood So yeah my experience and positive experience so far has been predominantly web development

24:18James Hartt Yeah, it's just The fast feedback loop in web development is just so, it's so ingrained into the tooling,

24:26Abrar Mahmood Yeah

24:27James Hartt Whereas iOS, it's just, it's not, unfortunately,

24:30Abrar Mahmood Yeah

24:31James Hartt Same with Kotlin Android. And so on it's just it's never I think yeah we're gonna have to see some innovations in that I'm sure they're coming but yeah like they're not quite there

24:42Abrar Mahmood So is it essentially just the like even without agents is there not literally like a watch mode or whatever to

24:50James Hartt I yeah

24:50Abrar Mahmood To

24:50James Hartt I'm

24:50Abrar Mahmood Use

24:50James Hartt Sure

24:51Abrar Mahmood Or

24:51James Hartt There are yeah there's watch modes in SwiftUI apps for sure But, yeah, I think it's really just if you're, Yeah, if you're, say, trying to work in an Android app, you're going to have to have an extra loop that, like, re -implements it Or Based on PRs or something that could be agent-led.

25:11Abrar Mahmood Yeah.

25:12James Hartt But, yeah, you're never going to end up with that sort of clean loop of web tech. So, Yeah, it's always going to be a bit more complex. Yeah the tension between yeah native and web is getting definitely mixed up at the moment Like

25:26Abrar Mahmood Yeah.

25:27James Hartt You could say it's never been easier to do two different native code bases because again yeah you can just chuck agents at it but like You're also still going to come up against a lot of issues that we're describing now

25:41Abrar Mahmood I think yeah so I think like in terms of the actual engineering iteration loop so like of course we spoke about Like web developments obviously works really well with this whereas iOS or well any native build is a little bit more tricky I think like from what I'm seeing with agents they're getting so good at understanding what the intent is For the work

26:06James Hartt Yeah

26:07Abrar Mahmood And what that means is one-shotting stuff

26:11James Hartt Is a little

26:11Abrar Mahmood Is just

26:11James Hartt Bit yeah,

26:11Abrar Mahmood Getting

26:12James Hartt It's getting much

26:13Abrar Mahmood So much

26:13James Hartt Better.

26:13Abrar Mahmood Better

26:14James Hartt Yeah

26:14Abrar Mahmood Which kind of leads me to think that having a running web server to actually like baby what The agent's doing is becoming less and less important So that leads to the next thing which I wanted to talk about which is pull requests so now I think like I'm at a point Where agents like I've got I've got agents so where I work I've got my agents all set up in a way where they're really good At knocking stuff out and it's kind of like I kind of see it like I've now got a I've got an entire consult I've got an entire offshore team that will do anything I need them to do and I kind of see it like that where like Agents are an entire offshore team that will just do whatever it is I need them to do I'm coming to them with the ideas now My and they're going to knock it

27:09James Hartt You.

27:09Abrar Mahmood Out between them as quickly like in like insane speeds and they need to find a way of validating their Output And... And of course I would say that a lot of big tech firms that are very heavily using agents are kind Of sort of seeing this as the problem as well but like how do we validate their outputs so of course naturally from an engineering perspective That means a pull request so in terms of that iteration loop so we were talking up we're talking a little bit like about web development so we've Got npm run dev which then gives you the iteration loop and like I think like because the agents are getting so good like we can just get them to one-shot That so if we take that out of the equation hopefully that applies to mobile development as well we probably don't need

27:54James Hartt Mm-hmm.

27:54Abrar Mahmood To see every little iteration if the agents are really going to get that good the next sort of thing is okay we've got a pull request and this pull request Is for an entire feature so what does it look like to spin up this pull request so make perhaps like that's kind Of like the next sort of evolution in terms of like the journey so I think that's something that we've been doing quite a bit in sort of The startup that I've been working at So we've got an idea of this pull request is now in a draft state. So for example, in GitHub, you've got a draft pull request that can then be promoted to being fully open.

28:32James Hartt Yeah.

28:33Abrar Mahmood So we create a draft PR, which then has an environment that it spins up, again, using the similar sort of Seeds planted for the worktree. So it's got its own isolated database and all of that. So it's essentially just spinning up a worktree. But on the deployed environment and then that gives us an that kind of gives us the feedback loop so if we're going to be one-shotting a Massive feature which the agents are very capable of doing these days it then goes away builds it comes back with a pull request In draft state and then spins up an environment and then we've got a URL test it out So that's kind of like where we are like where we are at the moment and it seems to work really well it's like Okay we've got this big feature it goes away and builds it takes its time to do it does its own sort of validations in terms of Like running the sort of headless browser simulating that clicking away self-iterating then it's happy with it and then it comes in Does a pull request and then we've got like a an environment that it spins up and that's for us to go in and have a look at it

29:40James Hartt Yeah.

29:40Abrar Mahmood I think one of the big problems is because it's so hands-off you go in with a feature that you've That you want to build it goes away it's very hands-off it goes away and builds it comes back with a pull request And because you've got that and then you can do so many concurrent workflows it's kind of hard to keep track. Of

30:04James Hartt Yeah.

30:05Abrar Mahmood What this thing was actually supposed to do One of the real problems like this kind of comes back to vibe coding a little bit as well because engineering is not the bottleneck and you can build so Much I think the drawback of that is there's so much more opportunity for things that aren't Good quality work to come out so you can build ten different things but because these are all just ten different ideas You've had this morning There might not necessarily all be to the level of that amount of time spent on one feature in terms of Like put aside all the technical part of it but like actually like really thinking about what this feature is going to do and how It's going to benefit your users essentially like it's now easier than ever to build sloppy stuff

30:59James Hartt Yeah.

31:00Abrar Mahmood So I think that's like one of the key takeaways as well so like at GradeDrive I think that's something we've noticed as well like I think like This like this year like when we like when we first started GradeDrive like we were we were aware of this like now because the Enablement is there like it's more easier than ever for us to have so many competitors but I wonder How many of our competitors are actually coming from good engineering Principles. And coming from a place of Actual good quality work

31:33James Hartt Yeah

31:34Abrar Mahmood And real experience because there's a lot of competitors out there and to be honest I'll try them out and they're all like really bad

31:42James Hartt Right

31:43Abrar Mahmood But

31:43James Hartt Yeah

31:44Abrar Mahmood But the

31:44James Hartt Barely

31:44Abrar Mahmood Issue is

31:44James Hartt Functional

31:45Abrar Mahmood Yeah

31:45James Hartt No.

31:46Abrar Mahmood Like one of them like you couldn't even like log into the to log into the actual website like literally logging just did not work but the issue with that is like The so I think this is kind of where like the whole product engineer thing comes into play you really just get to a point where like because the market is Saturated by such poor quality such poor quality competitors anyone who like if it's a new Market so we're working in relatively new market of like AI-powered exam marking like if anyone

32:14James Hartt Mm-hmm.

32:15Abrar Mahmood Tries a competitor it's very easy for our customers to assume that this is kind of the bar

32:21James Hartt Right,

32:21Abrar Mahmood And

32:22James Hartt Yeah.

32:22Abrar Mahmood Like basically they're all like really bad and like it kind of that it kind of like hurts us in that way like if because like People would just generally get a general consensus of like oh this type of product does not work because I've tried one and it was Really bad but

32:39James Hartt Yeah

32:40Abrar Mahmood Because the enablement is in place now and it's really easy to like build poor quality software it Makes it really hard and harder than ever to market yourself to say you're actually doing this really well and what you're building comes from a place of Experience

32:57James Hartt Yes.

32:57Abrar Mahmood So that's one of the drawbacks as well and then coming into a company where like doing this at a feature level it's easier than Ever to build lots of different features that probably don't really make sense for your product so I think that's something I've noticed as well but I Think that kind of like I think like if that could be categorized as experiment versus a feature That we actually want to ship

33:23James Hartt Yeah

33:23Abrar Mahmood I think that's like But if we've got that distinction, then it's all right.

33:27James Hartt It's that I think people it's trying to. Sort of get over the killing of one's babies like the fact that if you can Like said very hands-off get something to work you do almost need a you need an extra step of like shall we shall we get rid of this in its Entirety like that should always be the question is I do like we can and it does it like very rarely skin off Anyone's back you can you can get rid of it like just the amount of closed pull whereas yeah closed pull request in the old world Was like a pretty big failure you know what I

34:02Abrar Mahmood Yeah.

34:02James Hartt Mean like it would like almost never happen and like there would be a good reason for it like oh someone like just got Like two devs got assigned the same ticket for some ridiculous reason one started working on it and the other started working on it they raised PRs in the same Day and one was better than the other like it's like I'm trying to think of examples where you would often see a closed pull request but That's what needs to happen you need to see a lot more closed pull requests

34:30Abrar Mahmood Yeah.

34:31James Hartt Like it's sort of front-loading the work means that you yeah you the taste comes right at the end and you need to yeah be Ruthless like way more ruthless than you than you were before.

34:46Abrar Mahmood Just coming back to The role that I feel like I'm in now at the places I work, engineering is kind of, for me, Engineering is enablement for the product team. So that's the thing I start with. And Then once I've got that to a good place, I then gravitate towards product.

35:10James Hartt Yeah.

35:11Abrar Mahmood Product and I think that's actually that's it depends on the domain so that in terms of getting domain expertise For whatever you're working on in some ways that's actually really hard because say for example like I'm I'm Working with a fintech company and there's probably a lot of domain expertise there that the layman is not going to be able to meaningfully Contribute towards

35:36James Hartt Yeah.

35:37Abrar Mahmood So I think one of the challenges I'm finding is in examples like that how do I as an engineer provide Value to the business when I've already enabled the product team And without spending five years becoming a domain expert in finance how do I do that and One of the ways I've been been exploring a little bit is like the idea of using agents to do that and I think like this is Where your point about taste gets like really gets to the really gets to the crux of it because agents I'm Finding are really difficult in my experience agents are not great at providing taste and like really It's going to give you like slop when you ask it for taste so that the taste still needs to come from us And I think like it's really important to understand what agents can provide and what value they can provide and I think in my Experience it's doing a lot of what I would describe as grunt work so research for example providing you've got A particular item or no sorry

36:48James Hartt The

36:48Abrar Mahmood Providing

36:48James Hartt Correct

36:48Abrar Mahmood You've

36:48James Hartt Question,

36:48Abrar Mahmood Got a

36:49James Hartt Yeah.

36:49Abrar Mahmood Yeah providing you've got the correct questions it will go and do the research exactly as you would want it to and give you back really good Research findings however what it's really bad at doing is coming up with that question

37:03James Hartt Yeah. Yeah.

37:03Abrar Mahmood To

37:03James Hartt Yeah.

37:03Abrar Mahmood Research so like you can't I'm finding it I'm finding it really tricky for it to do the innovation so to speak And I think like that's a good thing for me because that's something I enjoy doing

37:16James Hartt Yeah this

37:16Abrar Mahmood And like

37:17James Hartt Is

37:17Abrar Mahmood I think

37:17James Hartt The thing that gets me out of bed for sure

37:19Abrar Mahmood Yeah absolutely and I think a lot of people are of course quite worried about especially in the engineering space a lot of people Are worried about it like what it means for their role and things like that oh no like AI is going to be coding like better Than I can and all of that sort of stuff

37:37James Hartt And that's

37:37Abrar Mahmood But

37:37James Hartt Been the

37:37Abrar Mahmood Like

37:37James Hartt Case for a good few months now.

37:43Abrar Mahmood For me personally like I find coding a way of expressing creativity Like coding is just a tool it's just an abstraction to talk to machines like what you're coding is still is still a Question that we answer ourselves

38:01James Hartt Yeah

38:02Abrar Mahmood And I think like coding is just like I kind of think of it as like an abstraction now right so I think we've gone through a few so like Especially in JavaScript right so we've got our we've got JavaScript which is running on like V8 and Node or whatever and then You've got your c++ runtime of that and then I think like you've kind of got like almost like a it's not even a DSL but it's kind of like a natural language layer above what what you want your JavaScript to be

38:32James Hartt Yeah.

38:32Abrar Mahmood And then I think that's kind of where like our spec-driven development and all of that sort of came into it where like you've written your specs and then like your actual JavaScript code is kind of like an artifact of what that is and then it goes down through the goes down a little bit more lower level each and every time so it's kind Of

38:47James Hartt Yeah.

38:47Abrar Mahmood Like we're just going abstraction layers back so but the thing is the actual problem you're trying on itself is still a Tangible thing that AI is not going to give you and that's kind of like what we what I see myself sort of working towards And that's that for me is still very interesting it's coming up with these questions and how we want to Solve these problems without having to deal with like okay like why is there a syntax error In this bit of code right that's not something I need to worry about

39:23James Hartt Yeah.

39:23Abrar Mahmood But similarly like I don't need to worry about why there's a syntax error when writing C++ or like Debugging this assembly code or like hex editing an executable right that's just like it's just abstraction layers

39:39James Hartt Mm-hmm.

39:40Abrar Mahmood And I think like one thing That I personally think is really interesting so I think back in like or I think like GPT-3 came Out maybe And like there was a whole big thing about like it's written a massive newspaper article and it's done an Absolutely amazing job of it like it all reads really well and like this is like comes off as a genuine news article and like no one's ever Seen anything like it and that was that was that really eye-opening for AI and I think

40:14James Hartt Thank

40:14Abrar Mahmood Like fast forward to

40:16James Hartt You.

40:17Abrar Mahmood I don't think if you were to give AI that task of doing that same thing again in my opinion I mean like I'm not an AI expert in terms of like actual AI development I'm a consumer of AI and I Implement AI but like my understanding is if you were to get like Opus Or whatever right now to carry out that same task I wonder how much better it's going to be

40:46James Hartt Yeah

40:46Abrar Mahmood At providing taste to that newspaper article

40:50James Hartt Yeah

40:50Abrar Mahmood So that although like it's got at the moment we've got breakthroughs in terms of like solving these Very complex mathematical problems I think I don't think the value it's providing in terms of taste Is increasing at that same at that same sort of pace

41:10James Hartt Yeah.

41:10Abrar Mahmood And I think that's really like where like where we can still provide value and like that's like That's kind of where we need to be gravitating towards in terms of of engineering I would have thought because engineering ultimately Is although something I enjoy like I think it's kind of like it's engineering best practices that need to go in and it's More a higher level sort of architecture that needs to be focused on like providing that's in place agents are just Getting better and better at recognizing what what that means

41:45James Hartt Making the tool calls yeah like

41:48Abrar Mahmood Yeah

41:48James Hartt Figuring out out where the gaps are, yeah.

41:51Abrar Mahmood Yeah so. I think we need to be moving towards just solving these creative problems and I Think it's like taking that engineering and guiding the agents through that but once you've got that in place the agents are going to run with it So I think yeah that's That's kind of where things are going, really.

42:14James Hartt Sort of talk about how that role is maybe particularly like it's a bit more well it's a lot more exhausting than the old way of Coding

42:22Abrar Mahmood Yeah.

42:22James Hartt Like the sort of haranguing agents for an eight hour nine hour day like I wouldn't I don't think I'd feel capable of It I don't think I'd feel capable of it when if I was like Is it's like yeah and that's kind of yeah we'll talk a little bit about how yeah the role to do that Is like do that is just miserable and you do hear of these sort of nine to six 996 startups that are just hammering agents and you Just I just don't think they I don't think you can do that meaningfully like you can't

42:56Abrar Mahmood Yeah.

42:56James Hartt Add add you like you said you You can't add value by doing that kind of work, Just by increasing your output.

43:05Abrar Mahmood Yeah so I think that kind of ties into a point you made quite a while ago which is like these like roles in this space Are really moving towards being a aren't they it's just it's ultimately you set off an agent to work on something you come Back to it and it kind of it now more than ever fits around your lifestyle I think like hammering and agent nine to Five is just not really a good use of time

43:32James Hartt Yeah. You actually

43:34Abrar Mahmood I think

43:35James Hartt Do still need to do the thinking

43:37Abrar Mahmood Yeah

43:37James Hartt The sort

43:37Abrar Mahmood Yeah

43:37James Hartt Of going away and being like ooh yeah

43:39Abrar Mahmood Yeah.

43:41James Hartt You still do need that space for critical thinking

43:44Abrar Mahmood Yeah and I think there's yeah I think one of the things I'm struggling with is that critical thinking is actually really is actually really difficult I'm more often than not faced with edge cases and product questions that I don't really Consider Until I've actually built something normally and I more often than not find it difficult to conceptualize what some of these product Questions actually are so for example if like you do a plus B plus C But then you need to consider edge case X Because a B and C aren't built what does X look like but

44:29James Hartt Yeah.

44:29Abrar Mahmood I think that agents are obviously really difficult to keep all of that in context and actually understand all of that but like it's actually quite difficult and Quite taxing mentally to really understand what that means

44:41James Hartt Mm-hmm.

44:42Abrar Mahmood So I think like a lot of the work I'm having to do on the day-to-day is everything before the building and the Building is kind of like a checkpoint okay this is ready to go let's build it and that's where you set the agent off and get the pull request progressed quite A bit later like I was talking about but like everything that sort of feeds up to it is more often than not just the product questions and I think Full stack for me now is like well before agents it was of course like full stack is purely defined from An engineering perspective

45:15James Hartt Yeah

45:15Abrar Mahmood Whereas engineering now has just been contracted to the end

45:21James Hartt Yeah.

45:21Abrar Mahmood And full stack is very much problem that the end user has whether that's from your customers or That's the sort of business idea to start with that

45:34James Hartt Yeah.

45:34Abrar Mahmood The product and going through everything in order to make sure you've got a fully fleshed spec of what it is you're working on And I think that's where effort really is going and I think trying to do that in a is actually quite difficult But I think yeah I think I've personally found it be very much you use you spend your time thinking about That and then you basically just have your spec and then you send that off to work on and you kind of come back to it after a While right And that's just kind of the way things are going really

46:11James Hartt Yeah I use heavy use of like remote controlling things remotely basically and then waiting for a ping to come back And going

46:19Abrar Mahmood Yeah

46:20James Hartt Off and doing something else yeah

46:22Abrar Mahmood Yeah so I think in terms of like the way I actually do that at the moment so I've got a I've got a Ubuntu box where I've just got Isolated environments within that for the different things I work on and it is very much just remoting into it and More recently that I've been using the Claude desktop app

46:42James Hartt Yeah

46:42Abrar Mahmood Which which supports remote quite well

46:44James Hartt Yeah

46:45Abrar Mahmood And it's just basically

46:46James Hartt The remote control stuff.

46:47Abrar Mahmood Yeah it's just basically working on that really and it's Like you can just do it from your phone or like just do it from the desktop app, which I find Works quite well. I've got a very modest MacBook here that I essentially just use as a remote control, have barely anything installed on it And then just remote into my into my box where I do the work. But I think like because you're now front-loaded with So many product questions And doing that in parallel for lots of different features or lots of different areas is actually mentally draining

47:23James Hartt Exactly yeah You're not even

47:25Abrar Mahmood Yeah

47:25James Hartt Context switching you're just you've just yeah context overloaded yeah

47:30Abrar Mahmood Yeah,

47:30James Hartt But yeah

47:31Abrar Mahmood So

47:31James Hartt I

47:31Abrar Mahmood That's

47:31James Hartt Guess

47:31Abrar Mahmood Kind of where things are at the moment for me.

47:37James Hartt Should we wrap up

47:38Abrar Mahmood Yep.

47:38James Hartt So yeah I mean I guess we've sort of talked about like just what like where I think people will hit the wall with all these sort of workflows I could

47:48Abrar Mahmood Yeah.

47:48James Hartt Like is there yeah anything that you wanted to sort of add on to that like

47:54Abrar Mahmood So I think that in terms of hitting the wall, I think like the high level areas is reviewing the outputs of agents and finding a good way to do that. And I think some of the approaches that I've taken for that is having an environment spin up that kind of just shows what It is,

48:11James Hartt Yeah.

48:11Abrar Mahmood Having really concise descriptions of what the agents have worked rather than a massive wall of text that Agents love throwing at you also I think this is the review of it from there's the technical review and Then there's the non-technical side of it as well which is answering the product questions I think the technical review is slimming Down more and more every day

48:37James Hartt Yeah.

48:38Abrar Mahmood And I think like a bit of I guess just for context like I've actually not opened a code editor for

48:44James Hartt Yeah.

48:44Abrar Mahmood At least like six months

48:46James Hartt Yeah. I

48:47Abrar Mahmood And

48:47James Hartt Mean that's a typical experience for most

48:49Abrar Mahmood Yeah

48:49James Hartt Devs yeah

48:50Abrar Mahmood And I think like the thing I think like that's fine and that's quite relatable but the other thing I've unfortunately found myself doing Is I'm now going a step further which is I've not found myself Having a look at the database

49:07James Hartt Yeah right not even. Looking at the schema yeah

49:10Abrar Mahmood Yeah I'm not looking at the schema either but I'm kind of trusting the agents to get through it so I think like there's a Little bit of that which is a bit of a wall to that you hit because you've kind of got to decide like are we still looking At this stuff or are we trusting the agents and

49:29James Hartt Yeah.

49:29Abrar Mahmood Just letting it run with it the other the other part of it is of course the whole context overload that's the wall that you would be hitting and you kind of Just gotta weigh up how much of the feature you want to Refine before Before implementation so you basically want to pre -define what's out of scope and then it's going to come up with all these wedge weird edge Cases and you've just got to decide what's in scope what's out of scope rather

49:57James Hartt Yeah.

49:57Abrar Mahmood Than really trying to like iron out every single wedge case because it's just gonna it's just going to take ages I think that another thing is Like initially when we when we first like got into this whole vibe coding and agent assisted engineering the general Consensus was this means we can output faster oh this ticket that we were that we Originally timed pre-AI would take three days it now takes me minutes

50:25James Hartt Yeah.

50:26Abrar Mahmood Because I just get the agent to build it but what I've really found is overall an entire project does not actually Take that much longer or that it's not actually changed that much

50:39James Hartt Right, yeah.

50:39Abrar Mahmood Because what happens now if you're doing it right it takes a similar amount of time but your output is of a higher Quality

50:48James Hartt Yeah.

50:48Abrar Mahmood Because the agents have come up and presented you with edge cases you've not thought about previously so that original ticket get you timed as like two days Which now takes minutes although you might have built that quickly in order to define the actual spec of what you're doing it Still takes long and like to QA it still takes its time and to validate it's a sensible product Improvement and it's solving a product problem still takes a bit of time as well

51:16James Hartt Yeah.

51:17Abrar Mahmood So I think for me the real innovation is going to be trying to solve those problems and I think I mentioned earlier one of the things I'm trying to do is How do we use agents to solve product problems like we're all aware it can solve engineering really well but how do you get it To solve product problems which is kind of a wall I've hit at the moment and that's

51:37James Hartt Right.

51:37Abrar Mahmood Really where I feel like we have innovation that we move towards

51:42James Hartt Well it's all things that we want to do at AgentSee Is like figure out these new workflows and present them to clients so yeah I think that's a great place to stop thank you very much for that, Abrar that was a Great I really enjoyed that

51:57Abrar Mahmood Thanks, James.

51:58James Hartt And yeah we'll see you on the next one I've got some great episodes plans I hope to get some guests on fairly soon and yeah like Look forward to seeing you next time

52:07Abrar Mahmood All right, thank you all.

52:08James Hartt Bye

52:09Abrar Mahmood Bye. Thank you.