Running our own mail server, on purpose

00:06Abrar Mahmood Hello and welcome to another, episode of AgentSee podcast. I'm Abrar and together I'm with...

00:12James Hartt James, hi, nice to meet you. Hi.

00:14Abrar Mahmood Nice to meet you. So today we're going to be talking about running our own email server. Let's get straight into it. Okay, so let's start with why mail at all. So how is mail useful for us And why have we decided to start with emails

00:31James Hartt So yeah, obviously when you start a business, Email is one of the first things that you can think of apart from a website. We already did the website with like Cloudflare pages

00:41Abrar Mahmood Yep

00:41James Hartt And we had Cloudflare email routing, which does work quite well for a really quick and dirty way to sort forward on Things. Things. But yeah, the entire premise of agency is to see what agents can Enable someone that has high tastes and wants to have a good working solution, but sort of lacks Maybe the kind of really fine grain Knowledge that you need to say run an email server and run it to a standard Most businesses that are going to obviously stake their delivery of important information through email for. So yeah, that's why I kind of chose to like try and host our our own email servers.

01:32Abrar Mahmood And maybe just a quick one -liner just for our viewers in case they don't know in terms of our current offering we are hosting with Cloudflare for the website Itself

01:40James Hartt Yeah.

01:41Abrar Mahmood And you mentioned we do email forwarding or we support email forwarding with Cloudflare we can do that but in terms of what We are actually doing it's is it more the responses and like the two -way

01:53James Hartt Well,

01:53Abrar Mahmood Communication or

01:54James Hartt When we first Set up the website, I had the Cloudflare email routing,

02:01Abrar Mahmood Yes

02:01James Hartt But now it is fully going to our email server. So all the Cloudflare

02:06Abrar Mahmood Nice

02:06James Hartt Routing stuff has been taken away.

02:09Abrar Mahmood Cool.

02:09James Hartt So, yeah, does that answer your question? Yeah.

02:12Abrar Mahmood Yes it does yes

02:13James Hartt Yeah. Perfect.

02:14Abrar Mahmood Cool So if we go into sort of some of the traditional routes that people take, so we've spoken a little bit about Cloudflare. We've said that we've gone all the way and tried our own email server in terms of other potential offerings That people typically go for. Have we looked at any alternatives or anything

02:39James Hartt Yes

02:39Abrar Mahmood That we might

02:40James Hartt Absolutely

02:40Abrar Mahmood Have

02:40James Hartt So

02:40Abrar Mahmood Considered?

02:41James Hartt Yeah I mean obviously

02:42Abrar Mahmood That it?

02:42James Hartt You can choose anything from sort of Google Workspace

02:45Abrar Mahmood Yeah.

02:46James Hartt The actual cloud provider that I chose to run the VPS on was Infomaniak which are a swiss outfit and They have and basically a sort of microsoft suite I considered them as well just because like I quite Like all the principles of Infomaniak like they're running on nice swiss like servers the sort of swiss care about their like Their privacy and data integrity and sovereignty

03:12Abrar Mahmood Ciao.

03:13James Hartt And they're also all run on nice renewable energy so there was was just lots of like they're I'm pretty sure they're a B Corp as well like that there's just There were lots of things that were just like it's not a it's not a ridiculous hyperscaler that I have to worry about sort of and yeah just yeah Not it was just a nice kind of routine like they all they support open stack and yeah got to use OpenTofu And lots of lots of other things that I've kind of wanted to try over the years and yeah kind of play around so

03:44Abrar Mahmood Nice and in terms of those details and the VPS Did you what did you have like a bare bones VPS that you're working with or that because you

03:54James Hartt Yeah

03:54Abrar Mahmood Mentioned OpenTofu

03:56James Hartt Yeah

03:56Abrar Mahmood And a few other tools did

03:58James Hartt Yeah.

03:58Abrar Mahmood That sort of come with it or was it pretty bare bones and you had to sort of bring that in

04:03James Hartt No everything's been instantiated by OpenTofu yeah so

04:07Abrar Mahmood Chill.

04:08James Hartt There was obviously when I was setting up Stalwart which was the email server that we used like we didn't vibe code email server or anything Like that would have been absolutely nuts yes stuck it up on basically yeah OpenTofu took care of a lot of it including Yeah and it's that's all on our GitHub sort of showing how that's finally put together but yeah it it was a lot of it a lot of those Decisions obviously all the all the Terraform files were written by the agent

04:36Abrar Mahmood Yeah.

04:36James Hartt All of the sort of DNS settings I had sort of put in various amounts of tokens for Cloudflare for changing DNS routing like I Had to obviously do a decent amount of click ops to get quite like the secrets working got it all injected with 1Password CLI

04:54Abrar Mahmood Nice

04:55James Hartt Yeah kind of did things how I would love to do it if I had sort of an infinite amount of time to work on a project like this

05:02Abrar Mahmood Yeah

05:03James Hartt And I would have predicted in the old world like this would have taken me several weeks to

05:08Abrar Mahmood Yes

05:08James Hartt Do and it would have like quite a lot of it would have been miserable like

05:12Abrar Mahmood Yes.

05:12James Hartt It would just be making config files and the kind of stuff that makes you want to like run off a cliff so

05:19Abrar Mahmood Yeah.

05:20James Hartt And it genuinely so so yeah it was quite it could genuinely took me a couple two afternoons to complete and along on with Yeah sort of proper backups and at like a small amount of monitoring like I could probably put in some more monitoring to make sure that It's a little bit more reliable but yeah it's backups and if the backups don't go up swimmingly then I'll get told that's not working two different backups So yeah it was overall the process was good there's great assets being created and I feel like confident enough in its Security

05:53Abrar Mahmood Yep.

05:53James Hartt That like I'm telling telling the internet about it

05:59Abrar Mahmood So I guess just taking a step back so we've got a VPS

06:04James Hartt Yeah

06:04Abrar Mahmood We're using OpenTofu and I guess for those of the viewers that don't know what OpenTofu is do we want to just have a quick one -Liner of like

06:13James Hartt Well,

06:13Abrar Mahmood What

06:13James Hartt Yeah.

06:14Abrar Mahmood That is

06:14James Hartt So

06:14Abrar Mahmood Yes

06:15James Hartt Yeah, OpenTofu is like the open source drop -in replacement for Terraform. It's a way of sort of defining your infrastructure, Defining DNS, defining

06:24Abrar Mahmood Yes.

06:25James Hartt Basically everything that you need to connect up digital services. And yes, you're writing the same Terraform that you would do if you were working with that

06:37Abrar Mahmood Yep

06:38James Hartt So yeah it's it's a way to fully define your infrastructure so with relatively ease if for some reason you need to move Your Infomaniak onto another provider that wouldn't be hard

06:51Abrar Mahmood Yeah.

06:51James Hartt Yeah just a matter of sort different tokens and a and a and another plan and redeploy yeah just kind of it's the way that I Certainly like making services is if I feel like I'm clicking around in a console like again I'm just like I Will forget this

07:09Abrar Mahmood Yeah.

07:09James Hartt I will forget how this has been done I will forget all the menus I'll forget Like basically forget it so

07:17Abrar Mahmood Yeah.

07:17James Hartt If it's written down and it's reproducible

07:18Abrar Mahmood Yeah.

07:19James Hartt And you can spin it up spin spin it down, Like that's great, like suits me to a T.

07:28Abrar Mahmood So we'll go into a little bit about some of the areas where we found it a little bit more challenging.

07:35James Hartt Yeah.

07:35Abrar Mahmood Of course this principle of infrastructure with code in theory has been really good and we've spoken about some areas where

07:42James Hartt Yeah.

07:42Abrar Mahmood It's done really well

07:43James Hartt Yeah.

07:44Abrar Mahmood We'll go into some areas where it's a little bit more challenging but I think it will be good to so we've established the foundation of OpenTofu What it is how it's helped us maybe let's go into sort of the next sort of level which is what are some of the things That we had to set up in order to get

08:07James Hartt Yeah.

08:07Abrar Mahmood Our Stalwart email server running so like what did that look like as a package

08:14James Hartt So,

08:14Abrar Mahmood Was it like a Docker container

08:16James Hartt Yes.

08:16Abrar Mahmood Image sorry or was it

08:18James Hartt Yeah,

08:19Abrar Mahmood Another

08:19James Hartt Like,

08:19Abrar Mahmood Format

08:20James Hartt Yeah, it was a Docker image with the store binary. Binary, And yeah, as soon as as soon as like, I was sort Of like, again, mainly using, in fact, for the initial setup, until I was sort of confident that we were just making tweaks, I had, I had the SSH key, Like, behind a passphrase so that the agent couldn't do any SSH commands. Commands and that kind of kept me at least in the loop a little bit Until I was happy enough that we were just doing sort of tweaks and then I removed the passphrase and it was able to sort of just do that was when I Really had Stalwart set up but yeah to sort of go on to your point about the sort of click ops side like there was Unavoidable click ops inside Stalwart and that could have been avoided in hindsight if I'd like brought in the docs a Bit better into the agent

09:13Abrar Mahmood Sure.

09:13James Hartt Maybe into the context and the CLI for Stalwart is a little bit clunky like it's specifically clunky around Like what like kind of what routes there are to get the information that you need and to mutate the information that you need like. It was hallucinating a bit and it especially found it wasn't easy to Iterate with the CLI, the Stalwart CLI, because it would often, as soon as you added like an extra little bit of a domain That you needed to get the information you needed, you would then get the error saying, oh, you're missing this or it's linked to this and therefore you can't remove that. That so it just it felt like the CLI maybe wasn't the Stalwart CLI wasn't built for an agentic Workflow

09:59Abrar Mahmood Show.

09:59James Hartt So yeah that those that was the pain and that was where

10:02Abrar Mahmood Yeah.

10:02James Hartt I spent basically all of my time was just messing around with Stalwart getting our emails set email accounts set up and just yeah looking at Sort logs of CLI logs and trying to sort of get them around and in into the agent to kind of like speed Things up

10:19Abrar Mahmood Yeah,

10:20James Hartt So yeah does that does that answer your question

10:23Abrar Mahmood It does. I think it would be good to

10:24James Hartt Yeah

10:24Abrar Mahmood Talk a little bit more, talk in a little bit more detail about your interaction and how you've tried to, some of The paths you've tried to take to overcome the click ops

10:35James Hartt Yeah

10:35Abrar Mahmood So of course you've mentioned the CLI which

10:38James Hartt Yeah

10:39Abrar Mahmood Is a it's one mechanism we can use

10:43James Hartt Yeah

10:43Abrar Mahmood To try and automate things that aren't directly in Terraform and it's kind of like provisioning your instance After it's already had the infrastructure so that you're setting up your config and all of that so we've got the CLI Which was perhaps a little bit clunky does Stalwart have like a management http API which is probably what what the CLI is using Anyway.

11:08James Hartt Yeah, potentially, I think that's what it's all based on. It's

11:11Abrar Mahmood Yeah.

11:11James Hartt All based on the sort of the admin panels

11:14Abrar Mahmood Yeah.

11:16James Hartt Like that you're mutating. So yeah, it would make total sense. And

11:19Abrar Mahmood Yeah.

11:21James Hartt Possibly why it was sort of hallucinating a little bit because you can see the routes inside the URLs for each of these Configuration panels and maybe a few here were wrong in the docs, wrong on the box, wrong like changed, Moved around. I certainly noticed Some things that had been sort of moved around between versions and that's the kind of stuff that can really mess up a an LLM workflow is that

11:48Abrar Mahmood Absolutely

11:48James Hartt That sort of little bit of mutation between how it used to work and everything's trained on to then What it is now and then

11:56Abrar Mahmood Yeah Yeah.

11:57James Hartt And that becomes a sticking point that it then needs to somehow get the context to figure out that that's what's going on

12:03Abrar Mahmood Interesting.

12:04James Hartt Because because it's yeah and that it you got a hint of that's what was happening But as with working with LLMs and Agents, like there's always walls of text to read and you're always going to miss something that's really subtle, and, just not Part of it. So yeah, like that was the, that was the sort of the worst part of the experience, but ultimately when it was done, It was, it was was okay, like I

12:30Abrar Mahmood Yeah

12:30James Hartt Was happy enough.

12:34Abrar Mahmood Interesting you mentioned the point about the sort of mutations across versions because it's something I've hit personally quite A lot in my own projects so an example which is somewhat relevant is I'm using cloud for in a couple of projects of my Own and I've I've had situations where the LLM would kind of give you a step -by -step list of what to do In the cloud fair console but of course more often than not what happens is the steps the LLM is Giving you are essentially telling you to click buttons that don't exist

13:11James Hartt Yeah absolutely

13:12Abrar Mahmood So so you often find yourself in a situation where hang on a minute like you're telling

13:17James Hartt That

13:17Abrar Mahmood Me

13:17James Hartt That

13:17Abrar Mahmood To

13:17James Hartt Path

13:17Abrar Mahmood Click

13:18James Hartt Yeah you're telling me

13:18Abrar Mahmood Yeah

13:18James Hartt To go to settings this panel

13:20Abrar Mahmood Yeah

13:20James Hartt That panel that panel and And it's

13:21Abrar Mahmood Yeah

13:21James Hartt Like, that doesn't exist.

13:23Abrar Mahmood Yeah so I think I think like absolutely infrastructure as code and having that all sort of Reproducible

13:32James Hartt Yeah.

13:32Abrar Mahmood In code is the best way like we can't really rely on uis for click ops and I guess like that's sort of the point where you Hit pretty quickly and like that's kind of where it fell apart a little bit as well but of course we've got there in the end so I think it'll be good to Go into a little bit about so we've got there we've done the click ops what have we got now at this point in time In terms of a representation presentation of the click ops And... What do you think we can do to move forwards to sort of Make sure we've got something that's reproducible going forwards

14:09James Hartt Well, yeah. Yeah that's where like again the sort of brute force of agents you can kind of you can one If you have gone through a hard spot you can still do the cleanup really quickly and that's what saw eventually once I've wrestled enough context Together it basically just it I was able to make a really quick snapshot script that did take all of the like our Entire store was set up and was able to back it up Or to commit it even because it's not it's not got any secrets in there it's not got anything And then yeah therefore we can we can deploy an instance of Stalwart that's exactly as we would want to want to have Minus yeah minus sort setting up accounts again you would want to put a new app password in if you if you Set it up again So,

15:02Abrar Mahmood Thank

15:02James Hartt Like, and just, and rotate credentials and stuff. So, yeah, I was, And again, that's the kind of drudgery work that would have been miserable. Beforehand, like making sure you've got a reliable, and yeah, During this process, I set up a, just a throwaway dry instance with this configuration just to check that it would take it,

15:22Abrar Mahmood Yes.

15:24James Hartt And, yeah, that was approved as part of it so

15:26Abrar Mahmood Nice.

15:26James Hartt Yeah yeah just sort of those The drudgery work of ops is just gone away and I think that's probably what like we're both developers that Level of like obviously in large orgs you have you had an ops team that would do a lot of this for you or help you or like Maintain it whereas I just feel like when like the balance is shifted like you can you can you can just do this drudgery really Quickly and

15:55Abrar Mahmood Yeah.

15:56James Hartt And like be sure that like or at least quickly test that it's gonna work

16:02Abrar Mahmood Yes.

16:03James Hartt Rather than spending like this again to go back like this would have taken me weeks and I would

16:08Abrar Mahmood Yes.

16:08James Hartt Have hated every second of it like

16:11Abrar Mahmood Yeah.

16:11James Hartt So yeah and I feel like yes with agents like sometimes you lose you lose a little bit of the understanding but I felt like I did all have an understanding of all the different email terms and like your DMARCs your DKIMs Your like yeah the glossary of terms like was re yeah I kind of revised it I think this would have been a lot harder if I'd never come Across these terms before

16:37Abrar Mahmood Yeah

16:38James Hartt And like just didn't know what they meant

16:42Abrar Mahmood Personally email of course for me email is something that I really don't enjoy working with

16:50James Hartt Yeah I mean

16:51Abrar Mahmood And

16:51James Hartt It's

16:51Abrar Mahmood I'm

16:51James Hartt A

16:51Abrar Mahmood Sure it's a shed

16:52James Hartt It's a

16:52Abrar Mahmood Yeah

16:53James Hartt It's a year old protocol isn't it

16:55Abrar Mahmood Yeah.

16:55James Hartt So it's ancient

16:57Abrar Mahmood Yeah and it's also the kind of thing you set up once

17:01James Hartt Yeah

17:01Abrar Mahmood In an organization and never revisit

17:04James Hartt Forget

17:05Abrar Mahmood Until

17:05James Hartt Yeah hopefully

17:06Abrar Mahmood Yes. Until there's that time where something's not working

17:10James Hartt Yeah

17:11Abrar Mahmood And whoops there was this key or whatever I needed to recycle because it had expired whatever it is and that only that is When you need to refresh yourself and revise

17:22James Hartt Yeah

17:22Abrar Mahmood All your detail about how emails work in order to know what it is

17:26James Hartt And if

17:27Abrar Mahmood That needs to be done

17:28James Hartt If you're executing. Agents to do that like that's what we're like no one I don't think is everyone is ever going to say like I'm going to have to Sort of SSH into this box or to fix a small configuration issue that's stopping it from working like That I would hope hope those days are behind us so it means that I think when we do encounter like an issue with the with email I think yeah we've got a couple of open problems with spam detection because obviously if you set up a new email account you don't have enough information to Set up your spam like that will

18:02Abrar Mahmood You.

18:02James Hartt Take some kicking I'm sure there'll be like if we ever have to send out mailing lists I'll have to upgrade the outgoing Track because I just shoved in SMTP to go as our outgoing and like signed the headers correctly for them and Yeah I'm sure if we're sending out more volume we'll either have to pay them or move I think one of the original parts of the the agent -led Plan was to use SES and I was like I don't I don't want to get an amazon account just for SES and have to go through the manual process of of Like because I I've set up SES before in the past and you

18:37Abrar Mahmood Yeah

18:37James Hartt You actually have to have a human review to make sure you're not like spam basically

18:42Abrar Mahmood Yeah which

18:42James Hartt So

18:42Abrar Mahmood Is actually in my experience not easy to

18:46James Hartt No

18:46Abrar Mahmood Overcome as well especially when you're a very small business

18:49James Hartt Yeah

18:49Abrar Mahmood It's not easy to get the correct approvals that SES need to

18:54James Hartt Yeah

18:54Abrar Mahmood Use

18:54James Hartt Yeah

18:54Abrar Mahmood In production like you remain in the sandbox quite long

18:58James Hartt Absolutely cool

19:03Abrar Mahmood So we've spoken a lot about the OpenTofu setup on the VPS next layer up is the Docker service after that we've Got the click ops that we did on the Stalwart service we've done spoken about the snapshot and the restore but I think one area That people are particularly nervous about when self -hosting is the monitoring and alerting when

19:26James Hartt Oh, yeah.

19:26Abrar Mahmood Something goes wrong so I think it would be good to talk a little bit about what we have in place at the moment what we deem Necessary given our current state our agency and then moving forward what we'd like to introduce to ensure that We've got our correct monitoring and alerting in place to give ourselves confidence with the infrastructure and lastly a part of that Could be okay so these are the particular requirements what I guess based on prior experience how Much other limbs will help us with that I think I'll be good to go into that a little bit

20:02James Hartt Yeah absolutely so obviously the only kind of monitoring that I have are failed backups and they're sort of they're going to two places they're going to a Sort of Infomaniak S3 clone and they're going to a Cloudflare R2 And both of those are monitored for whether or not they're happening or not so that would obviously count as if the actual the whole service would go Down but that's probably not like great like you would like to know whether incomings and outgoings are working potentially Like whether inboxes are full like sizes and stuff so there's definitely some outstanding work to do but those are things That you're not like certainly like inboxes becoming full like you could that could happen if someone wanted to send us an absolutely Enormous amount of spam but like we'll deal with that if anyone throws that our way but yeah it's I think those sort of things will Happen like as and when we need to and I think would they would be very discreet operations to happen like from an agent it Wouldn't be anything that would be particularly onerous to do so I think it's kind of like as like as and when is Needed for that and then yeah it'll be something to do when I've got some downtime I guess like as a little follow -ups and yeah we were able to Like track those loose ends quite well

21:23Abrar Mahmood Nice.

21:23James Hartt Using so yeah It's none of these things like would ever be absolutely perfect if you're going to do it yourself with agents you're you're not going to Have the monitoring that say like Gmail would have

21:37Abrar Mahmood Yes.

21:37James Hartt So yeah there are still open ends But yeah, well, I guess we'll just have to, that is, You're always going to have to find these sort of things when you're setting up services yourself.

21:51Abrar Mahmood Yeah

21:52James Hartt You're never going touch every single base when

21:55Abrar Mahmood Yeah

21:55James Hartt You stand stuff up.

21:59Abrar Mahmood Another thing that I personally would be quite interested to know because I don't have a huge amount of Email experience so in terms of reputation

22:10James Hartt Yeah.

22:11Abrar Mahmood How does that play into all this because I assume if you for example are using Google

22:16James Hartt Yeah.

22:17Abrar Mahmood You I would have thought you'd have a reasonable level of reputation from the sender

22:22James Hartt Yeah.

22:23Abrar Mahmood How does how is it impacted

22:25James Hartt So

22:25Abrar Mahmood When

22:25James Hartt Obviously

22:25Abrar Mahmood Running around mail server

22:26James Hartt Like receiving emails is just that's no problem that's just

22:30Abrar Mahmood Yes

22:30James Hartt Just an open port to the internet but like sending that's why I used sm SMTP2GO for the outgoing is that yeah if you're Trying to send outgoing mail from your own server good luck with that everyone will treat you as spam there's just. No there's no other way of going about it like you do need to borrow the reputation from someone

22:50Abrar Mahmood Right.

22:50James Hartt And yeah so that's why I mentioned yeah SES was the first agent suggested option and I just immediately saw Through it and just thought for the first go we need to just go with someone that's got good enough rep yeah good enough reputation to just sort of Piggyback up off a free service like with as long as we're not sending a thousand more than a thousand emails a month which I think is highly unlikely For a good while like that will keep us going and again and I think even if we want to just keep exactly the same setup I think it's Dollars a month single digit dollars a month to get some more capacity but yeah it's basically it's SMTP to go's rep And they obviously are a business that don't want to be known as spammers so

23:36Abrar Mahmood Yes.

23:37James Hartt They're they're quite like I would trust them well enough and Again it would it would be wouldn't be hard to find another outgoing provider and Set it off to change set off an agent to change it

23:54Abrar Mahmood I also want to talk a little bit. So just like slightly sort of like bringing together the entire sort of technical part of it. In terms of the mail server itself and running on the VPS. Like what does that contain like does it like have its own database or is it like is it a stateless service that Persists or is it ephemeral or

24:19James Hartt I

24:19Abrar Mahmood Like

24:19James Hartt Think

24:19Abrar Mahmood What does that look like

24:20James Hartt I think from what I remember seeing again I didn't have to deal with the gubbins of Stalwart

24:25Abrar Mahmood Yeah.

24:25James Hartt But I think it has its own its own like it's got rock in the name I think but it's their own flavor of MySQL or

24:32Abrar Mahmood Right.

24:32James Hartt Postgres or something so

24:34Abrar Mahmood Okay.

24:34James Hartt It is all on the and that is in fact what the backup is the backup of the DB there's no state

24:40Abrar Mahmood Cool.

24:40James Hartt Of there's no state of the machine or anything like that's we hope again that's all covered by the by the IaC and the and the Outputs of the Stalwart CLI config so yeah did that answer your question

24:53Abrar Mahmood Yeah it does yeah

24:54James Hartt Yeah

24:54Abrar Mahmood No that's

24:54James Hartt Yeah

24:55Abrar Mahmood Really good yeah I think like in my experience like I typically try to aim for ephemeral runtimes

25:02James Hartt Yeah

25:03Abrar Mahmood That are of course stateless

25:04James Hartt Yeah

25:04Abrar Mahmood So just it was it was just quite good to

25:06James Hartt Yeah

25:07Abrar Mahmood Sort of

25:07James Hartt And

25:07Abrar Mahmood Clarify

25:07James Hartt I

25:07Abrar Mahmood That.

25:08James Hartt I think obviously you could you could experiment with like changes to the db config if you wanted to

25:16Abrar Mahmood Yeah.

25:16James Hartt But I thought it was fine I think again we'll have to see whether this like It's obviously running on the on the local storage of the of the VPS we'll have to see when that starts becoming a problem if it does ever Like yeah we could find that people send us sort of yeah an enormous amount of spam email and we'll have to figure out what we do But like yeah I think for regular business use for sort of like it should be fine for a good long while and then we can deal with It when it starts that's becoming a problem.

25:51Abrar Mahmood We've spoken about quite a wide range of things both technically and operationally What sort of comes to mind if you were to do this again are there particular things you'll do a little bit differently or

26:05James Hartt Yeah.

26:06Abrar Mahmood Anything that you would want to call out to people that are going to try it

26:10James Hartt I think on the whole, I would recommend it in our context.

26:14Abrar Mahmood Yeah

26:15James Hartt Would I recommend it to any business probably not Like this really is very specific to our selling point is that we want to use these kind of Workflows when developing other people's products we have to eat our dog food and see what the problems Become and know them intimately for the most core thing for a business like we needed this so that I could sample The social networking things to publish this podcast so like it it's it will be Like it's highly contextual for us

26:54Abrar Mahmood Yes.

26:54James Hartt Would I still recommend it for every business to go along this way absolutely not don't I don't do it like but just for us it Was it is also something that we can move away from very quickly if it becomes an issue and that's a Follow -up question that we will answer if and when we do move away from self-hosted email we'll tell you about it

27:18Abrar Mahmood I think looking at looking at it like slightly more from an abstract perspective so rather than Email specifically I guess this is an example where agency agency is experimenting with the idea of Taking a problem that we need to solve where we usually go for a a out -Of -the -box solution I .e Gmail but we are taking this opportunity to take a self -hosted solution and host it Ourself

27:50James Hartt Yeah.

27:51Abrar Mahmood What would I guess like would you say that this is something that we would be doing like I guess like Sorry what would you suggest we look for the next time we do something like this because of course that one of the main challenges was of course A click ops of the actual configuration

28:10James Hartt Yeah.

28:10Abrar Mahmood Are there any particular details of like the next candidate we look at

28:14James Hartt Well, that's

28:15Abrar Mahmood Or

28:15James Hartt I

28:15Abrar Mahmood Any

28:15James Hartt Think,

28:15Abrar Mahmood Particular

28:15James Hartt Yeah,

28:16Abrar Mahmood Tools

28:17James Hartt I think I would definitely attempt to Gather more context for For third party applications that may not be particularly well trained. That's definitely how I would do it again is That just make sure that they've like make sure during the agent workflow like have we got all of the docs are we sure How this CLI works can how can we avoid more manual work that could have because I Yeah I do ultimately think my click ops inside Stalwart could have been avoided if I had just pushed a little bit bit harder and said Look the we've we want to set up a couple accounts a bit of like a mailing a mailing list for like hello agentsee.work and a Few other things like this can't this can't be that difficult try

29:07Abrar Mahmood Yeah.

29:07James Hartt And do it without me clicking on a on a panel so yeah I would definitely push for more autonomy push for yeah for stuff like that So yes I agree like the click ops still needs to be minimized to for these workflows to really shine

29:23Abrar Mahmood Yeah

29:24James Hartt And yeah I mean you would want that in the old world right but that was like that was tough

29:29Abrar Mahmood Yes

29:30James Hartt It was tough

29:30Abrar Mahmood I think

29:30James Hartt And

29:31Abrar Mahmood Yeah I think in the old world though it's because we didn't have everything before it it automated

29:37James Hartt Yeah

29:37Abrar Mahmood It was relatively a small piece of work but because

29:40James Hartt Yeah

29:41Abrar Mahmood We get to that so much quicker now it feels more painful having that last bit

29:46James Hartt Yeah

29:46Abrar Mahmood Because relative to everything else you're doing it's now much more work

29:51James Hartt Yeah yes

29:51Abrar Mahmood So it's kind of highlighted in a way In terms of the actual click ops part of it I was just wondering if you've played with or have any thoughts on On like using AI And LLM tools for like browser automation or anything like that.

30:12James Hartt Yeah

30:12Abrar Mahmood I know there's significant drawbacks with doing that which is reproducibility, et cetera.

30:16James Hartt Yeah,

30:17Abrar Mahmood But have you played with that at all? And we

30:19James Hartt I

30:19Abrar Mahmood Thought

30:19James Hartt Mean

30:19Abrar Mahmood So.

30:19James Hartt I haven't used, it's part of the thing that sort of the Anthropic tools are lacking a little bit is that sort of computer use Aspect that sort of say some of the OpenAI tools and I I think I think for this specific instance I did Want a little bit more control over what was and That was happening that I I kind of at least understood some aspects of it I don't think that would be the case for every workflow and I think yeah Having the ad the admin panel be navigated by an agent could have could have worked

30:54Abrar Mahmood Yeah.

30:55James Hartt For sure I just think I still needed some level of like of understanding of of the config

31:02Abrar Mahmood Yeah.

31:02James Hartt At least whilst it was happening but yes I think yeah what tell me more about your experience of like

31:08Abrar Mahmood Yeah.

31:08James Hartt Because yeah obviously OpenAI products for computer use are a lot better currently

31:12Abrar Mahmood Yeah.

31:13James Hartt And they can actually click through and it's only available on Mac as well right

31:17Abrar Mahmood Oh, I didn't know that.

31:19James Hartt I

31:19Abrar Mahmood Oh okay

31:19James Hartt Thought I thought it was

31:20Abrar Mahmood Yeah

31:21James Hartt That might be very recent

31:22Abrar Mahmood Yeah

31:23James Hartt Recent changes

31:24Abrar Mahmood Perhaps yeah

31:25James Hartt Yeah

31:25Abrar Mahmood So yeah I have used it quite a bit both Claude and Codex and I often I found it works Extremely well for doing what you're actually asking it to do

31:37James Hartt Yeah.

31:38Abrar Mahmood So in this scenario I would probably have a Claude Code chrome extension I use chrome

31:44James Hartt Right, yeah.

31:45Abrar Mahmood And I would say open a tab on this port I need you to set up the email server with this find. Your way around the menus and it will do it and

31:56James Hartt Yeah.

31:56Abrar Mahmood It will do it really well

31:58James Hartt Yeah.

31:58Abrar Mahmood And you do I I I imagine there's probably a way to probably record the browser session to sort of see what it's doing like perhaps from a reproducibility Perspective but I think I think the reproducibility of that is probably the issue for me although you can probably record it I've never Recorded doing any sort of click ops before but like it's

32:17James Hartt You wouldn't want to watch it as well that'd

32:19Abrar Mahmood Yeah

32:19James Hartt Be even worse doing it yourself

32:21Abrar Mahmood Yeah

32:21James Hartt Watching

32:22Abrar Mahmood So

32:22James Hartt Someone Someone else click around,

32:24Abrar Mahmood Yeah

32:24James Hartt Yeah,

32:24Abrar Mahmood And like

32:24James Hartt And

32:24Abrar Mahmood Quite

32:24James Hartt Someone.

32:25Abrar Mahmood Often you see it like clicking in like different sort of like coordinates before it actually gets a button

32:30James Hartt Finds

32:30Abrar Mahmood And whatever

32:30James Hartt The button,

32:31Abrar Mahmood But

32:31James Hartt Yeah.

32:32Abrar Mahmood It does work really well and I think the really good thing is that if you set it a goal like it might take a while to get there but it does get There and having a clear test case to tell it when it's done is really important as well and I think Overall my experience with it has been very positive and I've found it works really well to tackle these sort of click ops

32:55James Hartt Yeah.

32:55Abrar Mahmood Styles of tasks I think two other things I've personally found to be really useful for again click ops Is so there's a number of I think there's a number of sort of like open source tools out there for this but like when using skills there's a concept I think there's a skill a particular one which is called which is the concept of building a wizard

33:18James Hartt Yeah.

33:18Abrar Mahmood And you basically invoke the skill and it will like basically figure out what a step by step looks Like in order to go through all your click ops tasks so

33:30James Hartt Yeah.

33:30Abrar Mahmood This is one I've used several all times and it works really well because it because it does as much as it can so for example if you're doing A whole bunch of click ops on like Google tag manager for example and setting up GA it will use the APIs that are available so it Will tell you go step one click into the console and log in step two go to API keys step three create an API key and then you Shove that API key into your wizard and then it will automate as much as it can using the API key and then if there's anything else left that's click ops it will Tell you what to do so it

34:03James Hartt Nice.

34:03Abrar Mahmood Kind of guides you through it

34:05James Hartt Yeah.

34:05Abrar Mahmood And it creates it as a shell script so it is reproducible which is good

34:10James Hartt Yeah

34:11Abrar Mahmood Of course there are some

34:12James Hartt Well

34:12Abrar Mahmood I mean

34:12James Hartt What

34:12Abrar Mahmood Like

34:13James Hartt How is it like actually Yeah what do you know like any of the underlying packages like of how it's interacting with the browser it's all written by like

34:23Abrar Mahmood So

34:23James Hartt Yeah do

34:24Abrar Mahmood So

34:24James Hartt You

34:24Abrar Mahmood There's

34:24James Hartt Know

34:24Abrar Mahmood Two

34:24James Hartt Any

34:24Abrar Mahmood Different

34:24James Hartt Yeah

34:25Abrar Mahmood Two different things there so the Claude Chrome extension is interacting

34:29James Hartt Yeah

34:29Abrar Mahmood With the browser course and that's

34:30James Hartt Yeah Yeah,

34:31Abrar Mahmood Just Anthropic building the wizard thing I spoke about is purely a skill that will go out and discover APIs available for Whatever you're trying to build a wizard for

34:41James Hartt Right,

34:41Abrar Mahmood So for

34:41James Hartt Yeah,

34:41Abrar Mahmood The GA GTM use case it will discover what APIs are available figure

34:47James Hartt Yeah.

34:47Abrar Mahmood Out what it can and figure out what needs to remain as click ops and then it will save everything like that Entire wizard is basically saved as a shell script that you can rerun

35:00James Hartt Right, yeah.

35:00Abrar Mahmood Of course one of the real drawbacks which I've not gone into to is whether it will like if you create a wizard to Set up your email server right now and then in two years time we need to reset it up

35:14James Hartt Yeah.

35:15Abrar Mahmood And you do it at a different version I imagine it's probably not going to consider the fact

35:19James Hartt Yeah.

35:19Abrar Mahmood That you're running it at newer version

35:21James Hartt Or anything

35:22Abrar Mahmood However

35:22James Hartt Has changed in the meantime,

35:23Abrar Mahmood Yeah

35:23James Hartt Yeah.

35:24Abrar Mahmood However the good thing is that you if you commit that into your repo you probably can infer what sort of time you run it Out so you can probably get an idea of what version that wizard was built for

35:35James Hartt Yeah

35:35Abrar Mahmood So perhaps you might be able to migrate it to work with a later version so I found that really useful and of course the third thing that we've not really touched On too much is MCPs like perhaps if it had an MCP endpoint

35:47James Hartt Yeah yeah

35:49Abrar Mahmood You could shove that

35:49James Hartt I

35:50Abrar Mahmood In yeah

35:50James Hartt That's a good but I think I did see some rumbling links potentially of a Stalwart paid MCP but that might be me just forgetting but yeah there Were there are some paid options inside Stalwart if you get a license so

36:03Abrar Mahmood But yeah I found that really useful as well for click up style stuff but I think I think with agents we are at a point where like Brute force you can get things done pretty quick and pretty autonomously it's just the whole reproducibility of it which Gets a little bit sketchy when when doing config for your click ops.

36:24James Hartt Yeah when you definitely want to do it again I think that's the bar that I tried to meet is like how like could I do this Again and

36:33Abrar Mahmood Yeah.

36:33James Hartt Quickly

36:33Abrar Mahmood Yeah.

36:34James Hartt And yeah not like I feel like if I really did let a let a computer use agent free I just wouldn't have even remotely have understood That it done what I'd asked it to do so yeah but I think for certain tasks I'll just have to play around with that a bit more and see If it does still meet that bar

36:52Abrar Mahmood Cool

36:52James Hartt Yeah nice

36:53Abrar Mahmood So if anyone wants to sort of see the journey that you've gone through the repo is public so I think have you committed Your sort of a document

37:02James Hartt Okay,

37:02Abrar Mahmood As to

37:03James Hartt Yeah,

37:03Abrar Mahmood What you've

37:03James Hartt All

37:03Abrar Mahmood Done

37:03James Hartt The terraforms up there, I believe the Configs are up there. You'll see that there's a 1Password environment that obviously doesn't have any of the secrets in it. And yeah you kind of get to see the whole the whole journey Really so yeah it's all up there

37:23Abrar Mahmood All right. Well, anything else before we close off?

37:27James Hartt No other than yeah carry on looking out for another more podcasts we'll

37:31Abrar Mahmood Yep.

37:31James Hartt Be going into sort of some other like maybe not technical things like some of the sort creative stuff that I've had to do for Setting up podcast and if you like we're gonna once these are these are out we'll definitely be wanting to get some guests on to get out of our bubble And yeah talk to other people and see their experiences so yeah I'm looking forward to it

37:52Abrar Mahmood Absolutely well thank you very much for having us and we'll catch you next time

37:57James Hartt Cheers bye

37:58Abrar Mahmood Cheers bye