N8N and PHP With Edward Barnard  ## Metadata - Author: [[YouTube]] - Full Title: N8N and PHP With Edward Barnard - Category: #articles - Summary: In this episode, Scott talks with Edward Barnard about his column in PHP Architect magazine and his current series on using N8N with PHP to create reusable workflows. Links: N8N - https://n8n.io/ Beyond Prompt Engineering - https://leanpub.com/beyond-prompt-engineering Lean Startup Book - https://www.amazon.com/Lean-Startup-Entrepreneurs-Continuous-Innovation-ebook/dp/B004J4XGN6/ displace.tech - https://displace.tech/ Our Discord - https://discord.gg/aMTxunVx Buy our shirts - https://store.phparch.com/products/community-corner-podcast-t-shirt Scott's Social Media: Website - https://scott.keck-warren.com/ Bluesky - https://bsky.app/profile/scottkeckwarren.bsky.social LinkedIn - https://www.linkedin.com/in/scott-keck-warren-91689810/ Mastodon - https://phpc.social/@scottkeckwarren - URL: https://www.youtube.com/watch?v=LtayJIei6o0 Hello developers and welcome to the PHP Architect community corner where we have conversations with members of the web development community. I'm your host Scott Kekworn and today we're talking with Ed Bernard about uh all of his different things that he works on including his article in PHP Architect magazine and his recently released book on AI. Edward W. Bubernard brings unique implementation experience from programming Cray supercomputers at the hardware level, debugging systems at classified facilities and solving critical Y2K infrastructure problems. His 20 plus years at Cray research includes hands-on work with assembly language operating systems and IO subsystems. His cross-dommain experience from NSA troubleshooting to Saudi America inst AMRico installations provides practical insights into applying advanced problem solving techniques across diverse technical environments. After a second 20-year career focused on PHP applications, Edward has began developing a new area of artificial intelligence collaboration based on supercomputing skills developed during the Cold War era. He has written more than 100 articles for PHP Architect magazine. PHP remains his language of choice. Thank you, Ed, for finding some time to talk to us today. >> That was Saudi Aramco in the country of Saudi Arabia. Aramco standing for Arabian American oil company and um that's the nature of the of the US Saudi relationship is is um has a whole lot to do with the oil history. So yes, that's it. >> Okay. Awesome. So, so, um, so I've like come across you, um, several times over the years. Like I, you were at, um, I think Midwest PHP. You did like a whole like >> tutorial track about teaching people how to do PHP, which I like missed for, um, cuz that wasn't my my my like interest level. Um, but then you're also one of the columnists in PHP Architect magazine, which you've I didn't know you've written over 100 articles, which is impressive. Um can you can you talk to us a little bit about what it is that your column generally um constitutes? It's the the back office column. >> Yes. Um I tend to write odd material and that's coming from um I guess best way to describe it is uh trying to show more of a big picture view. Here are other things that um a breadth of experience can bring to you. Um and uh very often it's well this is the problem we had last week and uh this is how we solved it uh you know problem plus solution and in PHP and then report the following month that that didn't go well but that's that's PHP that that's normal website development especially with legacy systems. Um so the back uh the back office is um focused on backend considerations as opposed to front-end and user interface considerations um originally the idea was to um which I did uh more in the way of data warehousing concepts um because um with PHP sites we normally don't do data warehousing uh not not in the normal enter enterprise sense so that's a where it that's where it came from. Um and uh let's see. No, I put them away. Um my mascot was a carrier pigeon as it as because I I figured well back office use use carrier pigeons as your transport layer. Um there is an RFC for that by the way as some people will will know. Um, so, um, within the back off back office column then what I'm currently doing and it might end up being actually a six-month series. I'm um, today I'm writing part four. Uh, the last magazine had part three um, and dealing with workflows. Uh, and uh, specifically using NN8N as a workflow engine. Um, should I just keep talking about that? Is that okay? >> Mhm. >> Okay. Um so N8N is seems to be rapidly becoming more and more adopted in our industry and not necess and not just PHP. Um I'm using it from a uh personal project standpoint. And so um NADN allows a community edition where I install it on my own laptop. I register with them. I just give my email address and then I have an unlimited license for unlimited anything. um a as as a local installation. So it's um uh very good for personal projects in that way and um also learn some things. Um, NAN is um integrated with Node um which means JavaScript and I've often said that you know comp um computer software and websites run really really fast unless they're JavaScript and I usually make sure I'm at arms length when I say that to keep myself safe. But um what I found was if you are looping through a series of records that's that was my workflow is I was going through a whole data set and um extracting information well if you are doing a loop that means a new invocation of the node each time and there's a very slow startup cost for that node. So it turned out that uh and so doing it as as a single my SQL query which is why I showed in the column versus doing it as a loop went from um three and a half hours to two and a half seconds. And um uh so uh the and so part of what I talk about is the idea of look at these from a bigger picture standpoint, from an architecture standpoint. Um it um in that in this case it what is the constraint? Uh in this case it was my my lack of patience. Um and and so can I think about this differently? And it turned out that uh training it as a data warehousing process was the right answer. Um these were problems that have long since been solved by other people and I like to read a lot of They're back there. Yeah, there we go. I I read I read a lot of book read a lot of outside books. So, um I would much rather take the time to read how somebody else did it than take the weekend to figure out the hard way that I don't know how to do it. >> Right. >> Um so, um uh yeah, let me uh let me talk to about workflows in general briefly. Um >> yeah, >> we all we all have Go ahead. Go ahead. before we kind so like and and I I want to make sure that like I'm I'm like for people who don't know what like NAN is so like and so the idea is like you have like a you design a workflow and like it has multiple kind of like steps involved in it. So if you haven't read like the articles I highly recommend finding finding Ed's articles and kind of going through them and so like there's there's kind of like little I don't know discrete chunks basically where like each chunk has like an input and an output and they just kind of like work their way through the system. And so, and I know that like it's become like it's become like a big deal. Like I feel like in the last five months maybe like it's I feel like I've heard more and more about it, but like realistically like I've seen a lot of stuff where it's like this is a no code solution for people. And I think Ed's approach that's the approach that you've you've kind of taken to it is like well you can use it and then also implement your own little chunks in there to like because we know PHP, we know MySQL and so you can like kind of have those those pieces. So, and I think I think that's really like the thing that fascinated me about the articles is like oh yeah, like we can extend this because we're programmers and we have like more more functionalities essentially. So, is that like a good encapsulation of that or >> Yes. Um and and another analogy is um something my son do he's now Dr. Jacob Barard, professor of computer science over in North Dakota, um noted is I was doing things where I was using like five different tools to do my work a workflow related to well actually rel related to writing a book. And he was you uh and this was during his PhD dissertation and he kept trying to find the one perfect tool that did that did everything. and um uh uh Eric Reeves who talk who is the author of the lean starter I had it handy I highly recommend it by the way um for anyone doing uh PHP development it's it's got some very good ideas in it Eric Reese the lean startup it's about 10 or 12 years old um uh one thing he mentioned is yes when you are doing a prototype If you're trying to put all the features in and and do everything for everybody, what you've got is QuickBooks that nobody can use because it's it's just got it's it's got he actually was talking about QuickBooks. Um, and so, um, what I really like about N8N is I can take this tool for this, that tool for that, that tool for that, and then I can uh, this is actually one of my workflow sets of workflow notes. And so I I've got myself little notes, and then each of those would become a step in the N8 workflow. And so the ability to orchestrate a bunch of steps, some of which could be manual, but um do it in a way I know best um in uh it will do direct SQL queries. So that for my data import ETL process, part of it was my SQL and then the actual extraction I did in PHP. Well, it doesn't have a direct a way to directly call PHP, but um it does have a way I'm I'm running on on a Mac MacBook Pro, which is Unix, uh uh similar to to Linux, of course. Um and so, uh Laravel, for example, uh has very good support for creating a command line is you know, PHP, Artisan, whatever. um as do the other frameworks um because we can run PHP scripts at the command line um PHP- CLI is the um correct interface and so I just create something in PHP as a PHP script or in this case doing I'm doing it inside Laravel that way I've got access to Laravel models and and the entire infrastructure and then from nan um I simply call the command real simple and so um the idea of building a workflow that is using the best tool for the job in each case. Um and uh it does natively support um handcoded JavaScript. Handcoded JavaScript and I don't get along very well in 8 N8N. And the reason is in N8N it uh you have one step then the next step then the next step and so there's a data flow between each step and how that is structured is not at all obvious and it's very hard to identify on the so what I did was um uh on NAN I take the results put them in my SQL table I know how to deal with my SQL tables and so do you >> and I can look at the result and then the X1 reads that table and and uses the next table as the output. And so I I don't care about the communication. I only care about the control step one, step two, step three. And then I can uh and so that that work. So as a way of orchestrating its um it works very well. And in fact, in my in my view, that's kind of its its its purpose is um orchestrate somebody else's stuff. Uh, pardon me a moment. >> No problem. >> Okay, excuse me. Um, so um, uh, N8N has a very good UI and so you can look at the at the steps and I have screenshots in magazine articles and and what you have is the top top level flow. It's it is literally the sticky notes that you would put on a whiteboard if you were designing something and it's just right there and it will do it for you. And uh, so that's very nice. and then you can look into each one. Um, so yes, I'm finding it to be a very useful tool. Um, but I need to point out the idea of losing the forest for the trees versus losing the trees for the forest. Um, with workflows, you need to be aware of both of both aspects. What is it that I'm trying to accomplish? And that's where I step back. Well, the loop wasn't working, but what I'm trying to accomplish actually looks a whole lot like a data warehousing ETL, extract, transform, load, uh, business process. Um, why don't I why don't I do it the way it was if it was a business process and it went two, you know, 100 times faster. Who knew? Um, so, uh, that's the big picture part. And then once I understand the big picture, then the individual trees, they all fit. Um, and I literally lay things out by hand. Um, oh, never mind the phone number. Lay things out by hand. And, um, that works. And so, um, so my workflow is part NADN is part sticky notes, is part um, database. And, um, um, another Yes. And a very another thing we always worry about with in in the PHP world meaning website is okay I've got I'm I'm using this new tool is the company that produced that new tool going to be around a while and so the fact that it appears that NAN adoption is going up rather rapidly and so there's a growing community and meanwhile I'm on their mailing list and so I get NAND announcement mailings now and then. Those are all good signs that it's still going to be around next year. >> And we'll be back after this word from our partners. Building PHP applications is your passion. Managing cloud infrastructure shouldn't be your headache. Displace is your partner in cloud infrastructure orchestration, giving solo developers and small teams the tools and automation to deploy enterprisegrade Kubernetes clusters without enterprisegrade complexity or cost. Their CLI tools streamlines everything from local development to fully managed cloud deployments on AWS, Azour, GCP, and more. Skip the steep learning curve of Docker, Kubernetes, and Terraform. Let our automation free you to focus on your core business without losing control over your infrastructure. Get started at displace.te and discover how the right tools can eliminate the need for a full DevOps team. Use a um we use a sim similar product at work. Um, it's not NN it and and I think like realistically like the it it wasn't my it wasn't my pick. Somebody else picked it like a long time ago. Um, but like they have they have the problem of like it's a for-profit company and like we don't know if like that company is going to disappear, but like this does have a community edition where theoretically like because it seems like that that is a functional thing for us if we could just say but we want to use this thing that's free and open source and now we can just kind of run it. um which is a huge benefit to to all those pieces I think. Um >> um good examples are uh Rabbit MQ is run by a company but there's a free version um Elastic Search um and so I take that as a good sign meaning that the c the for the profit company is going to be around um uh Laravel itself has now become a for-profit company but they we've got all sorts of promises that um uh we'll still have our own versions. So, um, but that also brings up a good point, and that's again keep in mind that that is always a consideration, um, the more you can make yourself not locked into something, um, the better. >> Yeah. And I think like we the the way it kind of gets used is like it's it's its own like segment and then it dumps data into into another product. Um, and so I think like the thing I I've been really fascinated by is like cuz we have like some custom stuff. I don't remember what the language is, but there's like custom pieces in there sometimes to do the transforms that we need needed to do. And like I like the like your approach of saying, well, here's like the different steps and then all those like those like more complicated bits can exist within my PHP code. because I'm guessing like there's probably like a lot of times where like you don't like we have to touch the the workflow a little bit sometimes and like when we find bugs or whatnot but like all the bugs could exist within like like if it's just doing the like the steps essentially and all the all the actual logic is being handled by PHP and MySQL then we can like put all that in our source code repository and like have it tested and you know version and everything which I think is a really like a much better approach to all this. Um so yeah >> and there's a there's a principle there that you're coming very very close to which is an ETL principle and that is make the step as small as possible with one single responsibility and uh that you can clearly see for debugging purposes what is the input to that step and then what is the output from that step and don't do a whole lot of processing. It's far better to do uh 20 little tiny steps than it is one big step that does 20 things um for debugging reliability and the chances of getting it right the first time. Um with NAN there's such a thing as workflows. We call them sub routines or functions in PHP. And so I can have uh I can have a main runner which uh the main runner would run this sticky note and then it would run this sticky note. Well, I'm in the middle of development. There's actually a third sticky note out here. And so all when I as I develop that third sticky note of work of 10 10 steps, then I just add it. I just call it and and so um the idea of just adding methods and classes, same very similar idea. Um so you know these are these are universal architecture type things and um um uh I've also found that working with AI uh AI is strongly inclining to do absolutely as much as possible in a single step which is the opposite of what's useful um because the idea is just click the button and then everything happens but um yeah that that's usually like spilling a carton of milk. It's It's Yeah. Everything happened. >> Yeah. And I I feel like the the especially with like coding and a and large learning mod large language models. Um it's a lot of like reiteration of like what we're trying to do, right? Like it's it's you know like and that's that's the I always have a hard time with like large language models and workflow automation because there's always like it never is just like here's the output and it's in good shape, right? There's always like some follow-up step. So I'm I'm kind of I don't know. Yeah. Yeah, I'm so I'm curious that I don't think I I think I missed that part. I know that like you're planning on piping your So like the the workflow that you're building takes a bunch of documents and then like pipes them through um I'm sorry I'm blinking and through through like a large learn large language model. Is that is that kind of like the gist of of what you're doing with with the article? >> You're actually correct. Um my problem is that I have been saving transcripts of conversations. I have more than five million words referring to English language words, English language text of transcripts and um there was for for one person that's that's a lot also for passing it through a large language model that's a lot um to to pro and so um I was collecting things in batches to to to look for things and um ultimately trying to reprocess all of that um didn't prove necessary, didn't make sense, and was just too much for that use case. On the other hand, um as I'm thinking back, well, I want to go back to a certain area or conversation by um I've got those transcripts spread across um a doz about two dozen different directories and so finding what I wanted. Well, this this workflow allowed me to actually gather everything into suitable chunks that are organized by calendar date and by um um each time I had a new conversation, I started a new folder. So, organized in that sense. Um and so I've given myself a usable ret repository. And then what I do is I put that repository inside PHP Storm. PHP Storm has a very good text search. Since the transcript is of my talking, I you know I was either Claude or I did the typing. I I have a good enough memory that I can kind of remember what it was I was talking about. And so uh that gets everything into one place where I have my own search engine which is PHP Storm which is really lame when I'm using AI to format in the first place but at the same time uh processing 6 million words of transcripts takes um it's under it's under a minute and that's just on a laptop um to doing it as a loop just using it using proper data warehousing ETL processes it it's under it's underneath it's under a minute to get everything gathered in and that's with the PHP extraction steps and everything. Um did I sort of answer your question? Yes. Um so um a a key limit there's uh large language models. I should I should back up be be clear. I have decided to focus on a single large langu language model and gain a depth of experience with one as opposed to a breadth of experience with lots of them. So, I'm speaking specifically of Claude 3.7 anthropic uh Claude 3.7, which is the one with um all of the uh copyright lawsuits going on because they like to steal authors. Well, we won't go into that. That's that's for the lawyers. I agree with with the lawyer side, not not Claude's side on that. Um but uh two key limitations are uh the large language model is based on input. Input is the training data and so um in this case the training data cut off I believe was the end at or near the end of 2024 and therefore Claude has no knowledge of anything that happened after that point. And in fact if I refer to a date that was yesterday Claude says oh this is sometime in the future. are you sure you meant that? And so I I've learned to add, okay, and and today's date is u September 30th. Um so that's one of the key limitations. The other key limitation is um the input window. And if you're going depending on what application you use um there's ways of um adding files uh as a kind of as a uh GPT calls it a project and you can add files and so you basically have your own knowledge base. Uh the way I'm using it it's just uh current immediate input which means you're limited to um um uh everything gets tokenized and then you're limited to how many tokens can fit in that window. Well, if you're holding a conversation for for a week or more, um, and so the conversation transcript runs to 500 p, uh, PDF pages, if you were silly enough to do that, um, you're well beyond the context window. And so, um, I've just I didn't realize that was a problem for everyone. I just did took I just automatically used the same techniques I used 40 years ago when I was working with supercomputers with very very small spaces to work with. And um same techniques work. Who knew? Um uh and that's and that's why I put the career research material in my in my introductory bio is I'm I'm all of a sudden that experience is becoming more relevant than my PHP experience. Who knew? >> Yeah. I I I find that as like an interesting piece of like your your articles is like you do have like all those weird I don't know weird is not the right word. You all you have all this like this history with like computer computers and like I I've never worked with like a cray computer. I've never worked with like I I have I have a computer engineering degree. So I did some like hardware debugging at university but never anything as depth as like that. >> That is a circuit board from the last Cray one ever made. Oh, okay. That's cool. >> That's a super So, that is a Cray one supercomputer board. Um, and you also and you know it is not large scale integration. Those are individual chips. Each of those chips is a uh one, two, or four gates. That's it. Um, and there there was reasons for that, but um yeah, that's that's literally ancient history previous millennium. >> Yes. Yeah. Yeah, it's you know was it I think that thing's said 84 on it, right? Is that what the Is that what >> uh Yes, that was that was actually the Saudi Aramco Cray one uh installed 1984. Um uh I and family went out uh as part of the site support and it was decommissioned in 1990. >> Okay. So it's like that was that's a pretty good long time run for run for a computer I guess you know for 1996 years. Yeah. Uh yes and um uh they used it for oil oil exploration. Um uh the primary application for that one was um what they call reservoir simulation and you know there's you know Ford did uh engine simulation and so on. Um, but they would take everything they know about a given oil field and say, and then they would consider, well, if we pump, if we take this empty well and pump high pressure seawater down it, could we recover more oil over here? And they run the simulation and with all sorts of different data sets and and possibilities. And of course, they're dealing with the largest oil fields in the world at that point. Um, while I was there, up to a certain point, every single year, they discovered more oil than they pulled out of the ground that year. Their reserves went up every single year until a certain year. Carrie, guess what happened that that year? And this would have been around um >> the first Desert Storm. >> No, it was before Desert Storm. I think it was 1986, actually. uh if I'm remembering correctly, so it's uh Desert Storm, Desert Shield happened in fall of 1990 and then Desert Storm was late January of 1991. Um no, why they stopped why the amount of known reserves didn't exceed the amount they stopped looking that year. >> Oh, >> they simply stopped looking and that's the only reason it stopped going up. Right. Well, that's a good reason to stop. >> Yeah. So, I I really unfortunately I have to call like bring this to a close. It's really nice like chatting with you about all these things. Um and I do recommend like everyone check out your articles in in the magazine. Um they're they're always like informative and interesting. There's always like some um even if it's not technology related like I know you like you were talking about Hedi Lamar for a while um and some of her stuff. Um but is there anything that you want to direct our listeners to before I let you go for today? This is where I get to tell you about my book up uh on AI which I'm developing um as a PHP person. And so in the magazine, the current the current article, the magazine that just came out at the end of my article, the next page is the link to my book um about going beyond pro standard prompt engineering. >> Okay. And that should be that should be the September ep issue and I will put a link in the correct in the episode description so people can find find the book and I'm gonna go take a look at it. Um I I'm I'm it's unfortunate like I have a a subscription to the magazine so I don't so I have those things but I don't always get um all the pieces cuz because I read like the preprints because I want to know like what's going on in the magazine before people before people are are actually published. So I miss some of I miss the the that those things. So but I really do appreciate you finding some time to talk to us today. Thank you, Scott. >> Yeah, absolutely. I have to say another heartfelt thank you to Ed for all of his time as well as you for listening. If you're not already a subscriber, make sure you subscribe with whatever your preferred method is. We have both a audio version on the PHP Architect website as well as a video version on our YouTube channel. You can maybe share this with your co-workers so they can learn a little bit about what Ed's been working on and maybe learn about NAN. This is Scott Gag for the PHP architect community corner signing off to remind you keep listening, keep coding and keep reading., N8N and PHP With Edward Barnard, YouTube, 01k8rskqxav1ngg7sta9c59cg2.