924: Sync Engines and Local Data  ## Metadata - Author: [[Syntax - Tasty Web Development Treats]] - Full Title: 924: Sync Engines and Local Data - Category: #podcasts - URL: https://share.snipd.com/episode/7b0f7bc4-828e-4d86-8d0b-49e2609ad7f8 ## Document Notes ## Highlights - **Local Data Boosts App Speed** - Local data storage drastically improves app speed and user experience. - Many top apps use local storage to feel instantaneous and responsive. Transcript: Scott Tolinski Now, local data can be done in a number of very low file., and I mean lo-fi in terms of not the coined, you know, lo-fi local first, but lo-fi in terms of you're saving local data, you're Handling updating yourself, you're, you know, you're sending that data along, whatever, you're managing it yourself. It can be handled in very much those sort of ways with index DB. Some people use something like dexy for that you could throw it in all kinds of ways storing that data and managing yourself locally but what we're going to be talking about in this episode Is going to be more of the managed and sync side of things where some of these solutions take your entire stack, essentially, except for your view layer, which is whatever you may be. And sometimes they take the database as well, and sometimes they just sit in front of there. And so what I hope to get out of this by the end of this episode is for you to understand who are the players and how they might fit into what you need. Because local data, as we've talked about many times on this show, is a really, really, really exciting way that people are working these days. Some of the best apps that you use, use data locally. So we just interviewed the folks from Superhuman. Their platform is incredibly fast because of local data people talk all the time about how great linear is local data and the way these apps feel is they feel instantaneous and you might Think my app feels instantaneous i'm saving uh records to a database via an api it's all running locally on my computer, but it feels very fast. Wes Bos Yeah, it feels instant when your server is literally running on the same machine as your browser. But when you're calling APIs back and forth, it probably doesn't feel that fast, especially like I'm up at the cottage right now and we've got Starlink, which is pretty fast, but you Scott Tolinski Certainly notice it versus like a hard piped fiber connection yeah and anytime you have data loading or mutations 300 500 600 milliseconds feels like a lifetime and what doesn't feel Like a lifetime is like 10 milliseconds it feels like the darn instantaneous stuff and many people often account how native apps feel so much better than web apps. Why? Sometimes it's just because of UI frameworks and things that feel more native to the phone itself. And other times it's frame rate. But many times it is saving and storing data, things like that. A lot of iPhone apps, yeah, they're storing things to what? A SQLite database on the phone, and then they're handling a sync process in many different ways. So if your mental model is, I don't understand why local data makes sense, just think about the best apps that you use. And chances are they're really good because they're super fast, they feel native or they are native and they're saving data locally. I ([Time 0:01:58](https://share.snipd.com/snip/e7546e29-fa0e-48ae-8c26-479ab51b55be)) - **Next React Could Be Sync Engine** - The next major front-end breakthrough could be a local-first sync engine, not just a framework like React. - This reflects a fundamental shift in app architecture rather than just UI updates. Transcript: Wes Bos Got a question for you before we even get into this. I saw a tweet the other day that says the next React won't be like React. It will be a local data first sync engine that adds it on. And they said, like, when we moved from jQuery to React, those two things didn't have a ton of overlapped in how they work, right? Obviously, you're still wiring up clicks and fetching data and whatnot. But it was such a major change. And I think part of the reason we've never seen like a real React killer. Like we have lots of alternatives, right? We have Vue and Svelte and Angular and whatnot. But nothing has sort of like dethroned React similar to how nothing dethroned jQuery. And what will actually dethrone that will be sort of like a new generation. Do you agree with that where it's going to be something that is this stuff is baked right in? Scott Tolinski So as much as myself and some others have been talking about local data, I mean, you can check the receipts. I've been talking about this stuff for a long time. I still feel like ([Time 0:05:08](https://share.snipd.com/snip/a5a99b0b-3a55-4c54-8f4e-208b768b68b1)) - **Livestore Uses Event Sourcing** - LiveStore uses an event sourcing model, storing user actions as an event log. - This allows full undo/redo and reactive local SQLite database projections of state. Transcript: Scott Tolinski And this is from Johannes, who is behind Prisma. He's been around the Data game a long time. And we've talked a little bit about Local First with Johannes pre this project. So this is called LiveStore. And LiveStore is an interesting solution to the Local First platform that tries to do a lot for you. In fact, this is one of those ones that really takes over in a good way where you aren't necessarily bringing this to your current project. Some of these other ones, you got your own backend, you have your own database, you have things that are like kind of like a normal setup and you can kind of plug a sync engine in or your own Process or your conflict resolution stuff in there this is not that this is really something that takes over and it's an entirely different approach to storing data in a way so this is Called live store it's at live store.dev there's currently adapters for react there's adapters for Vue Solid. Somebody's putting together a Svelte one right now. There's a PR for it. There's React, I mean, with Expo. That's one of the options. They have stuff that's a Tori adapter, electron adapter. So it does seem to be something that is going to be, you know, pretty widely supported. Now, the way that this is interesting is that this all works in SQLite. Now there's a SQLite browser in your client, or a SQLite database in your browser via WASM using Livestore. So it's an embedded SQLite in your app that saves the information locally. And on your server side, there is a SQLite database as well. And unlike some of these other ones, what's being saved in your database is not necessarily data. It's events. So the way that Livestore functions is it's an event log, very similar to like, you might think of like Git or something like that, right? It's an event log. It's here's everything that happens. Last right wins. And it's a really interesting approach. So the entire thing is like you have a to do that to do is created, task completed, whatever. Those are events. They're saved as events and like what the information is with those events. These events are the source of truth. And the current state of the app is a projection of the event log, materialized into a local reactive SQLite database. Wes Bos Wow. So instead of saving the data locally, like let's say if you had like a to-do list and you changed, you added an item or you deleted an item or you changed one of them, what would normally Happen is that data would then sync to the backend and then hopefully you don't have any conflicts. And if there are conflicts, there's these CRDTs that will allow you to hopefully resolve them, right? But this will, instead of that, will store what you did, very similar to like Redux, it will just log out what you have done. And then I, I guess that sends that to the backend and it replays it in order to modify your data. Scott Tolinski Yes. It's interesting. And a quote from David Koresh, and you know, he loves this kind of stuff. Events are the most accurate representation of state. Live store gets it right. I have been goofing around with this a little bit, specifically trying to just get like this felt version of it working, which I haven't been because it's, you know, still a draft PR. But from browsing the documentation, from really getting into it and actually writing code, there's a lot of things I really generally appreciate about this, and it follows along Kind of the same things I appreciate about ZeroSync, which we'll talk about next, is that it is a nice API. Managing versions of data. I'm not having to think about my manual writing to the database, which, you know, sometimes you want to think about that. But if I'm starting a project from scratch, I want it to all just kind of work, but still give me access to everything. This is a very different approach. And I think it is an interesting one whether or not long term it proves to be like the best approach i don't i have no idea um they do have a thing on their live store verse redux live store Shares a lot of similarities with redux in the sense that both are based on event sourcing so you nailed that. So there's also even like dev tools for this though already. So I do think it's an interesting project. Their demos are very exciting. You got to love an exciting demo. They do have an llms.txt in case you want to, you know, actually plug this into your... Yeah, right. And for the most part, it handles off in a way like uh jwt based off very similar to how zero does it you know you're able to kind of step in there and get access to the current user and do your Wes Bos Your auth checks that way there there's a lot here there is offline support so does this replaces like let's take a look at like what this replaces this does not replace react right because You would use it with your framework of choice correct react ([Time 0:08:36](https://share.snipd.com/snip/92ba12b2-3306-4f39-96b7-d74c9fb1e558)) - **LiveStore Replaces State Management** - LiveStore replaces client state management, backend ORM, and sync with one system. - Use it with your UI framework but avoid overlapping tools it combines. Transcript: Wes Bos Does this replaces like let's take a look at like what this replaces this does not replace react right because you would use it with your framework of choice correct react result or whatever But it does replace any like state management you would do client side right it does replace your your back end orm so you wouldn't use like drizzle with this or whatever it's included With all of that and it does replace like you're saying JWT sessions and stuff like that so it does replace your like your user system like how far does this go oh well there is no JWT based Scott Tolinski Off inside of here you would bring your own off you're just verifying your JWT um and that and they can give you like a server side. ([Time 0:14:21](https://share.snipd.com/snip/4adcbcba-ae02-44b4-8736-35168b7dd709)) - **ZeroSync Links Postgres and Client** - ZeroSync bridges client and server databases using your existing Postgres backend. - It synchronizes data via patch messages for offline and real-time sync. Transcript: Scott Tolinski Now I've talked about zero sync a whole lot. Zero is kind of the second version of replica cache in a way where replica cache was a product that came out and handled that sync layer and that local data, local cache really well. But you still had to do a lot of work on the back end. So when zero came out, it was like, what if we take replica cache, we handle that back end, that versioning, that database side of things for you? And with Xero, it is a client side and server side library. It's a sync engine and it does a lot for you, but you can bring your own Postgres database and you can bring your own Postgres data. So if you have a Postgres database, and you wanted to turn it to be like a local type of data syncing project, you could theoretically throw zero on there without much strife, which is Awesome. Zero because of that also works anywhere like you have Postgres databases, like you can connect Zero to like Supabase or something like that. It works really well in those ways. And the way Zero handles it is it sends patch messages of the data back and forth when things change. And your data is stored more traditionally in your index DB on your local database as messages of the data that should be there. And when data is changed, it then has the opportunity to sync with the database offline, or sync with the database that's remote, not offline, the remote database. And anytime there's changes to the remote database, it pulls them into your local data. Anytime there's changes to your local data, it pushes them to your remote database. They stay in sync. And load times on this thing are phenomenal. I've built a ton of projects now in Xero, even though it's still in alpha. And the thing I like the most about working in Xero, it feels like it replaces your backend and your ORM. And in that you're, you're doing RPC calls, what look like that on the client side. And it just takes care of the rest. And to me, that's like these ones that are like taking care of all of the minutia of, you know, having an API endpoint and things like that. Really, really great. Wes Bos Yeah, because then you're not setting up any server endpoints. You're not setting up resolvers or anything like that. You just plug and play. You're ready to go. You got the client side thing as well as the server setup. Scott Tolinski Yeah. And so with Xero, you typically, you're having your sync server, which exists as a server. You have your database, which exists as a Postgres database. And then you have your application, which exists as an application could just be client side even. And that's pretty much it. Xero is really full featured in the replicash folks have been doing this for a long time. So they've hit a lot of the edges. They know a lot of where the pain is, and they've really smoothed that out in a degree. But one thing I really appreciate about ZeroSync is that they give you a ton of control, whether that is through how you're writing data, how you're handling permissions. All of these things, give you complete access. They let you step in there, and so it will block things on the server, block things on the client if permissions aren't there. And for the most part, the entire system feels like once you get it set up, it feels like you're working in something like Firestore or something like that with still owning your database, With it not changing how that data is stored, and with the client side experience, like I said, feeling like an ORM on the client side. ([Time 0:18:03](https://share.snipd.com/snip/9189ac46-9c42-4215-ae2d-ac783ad2b64c)) - **InstantDB Simplifies Development** - InstantDB offers the easiest developer experience with built-in authentication. - It's suitable for fast prototyping but mostly used as a hosted service. Transcript: Scott Tolinski Now InstantDB describes itself you want to get a handle on what instant db is like instant db describes itself as a fire base it's a fire store fire base which one is it is it both uh fire Store is firebases database okay it is a it is an open well it and it's a Firebase alternative is really what it is. But it is local data-based. And this one is fantastic because specifically it is the one that does the most for you. It does everything for you. You have to do the least amount of work to use this. Now, unfortunately, the way you're going to be using this is most likely not through self-hosting it. It's going to be signing up for a service. You get that API key. You download their library. Next thing you know, you've got a full auth system, though. You've got MagicLink login. You've got Normal login because this comes with authentication baked in, like some of these others, like we mentioned, you got to roll your own auth or bring it from better auth or any Of these things. This comes with auth baked in. And I will say the client side library for this one is probably the nicest in terms of like, it feels like Firebase, right? It feels nice and easy. You can theoretically self-host this. There is no documentation on that. I think you would have to dive into the code to figure that out. ([Time 0:22:01](https://share.snipd.com/snip/eb274931-d27b-45c1-af1a-501f6e9eb51d)) - **Convex: Promising Sync Platform** - Convex is a sync platform with backend and auth features and an easy startup path. - Its local-first features are still developing but it's well integrated and mature in sync. Transcript: Scott Tolinski It's way more similar to Supabase in its current iteration than it is something like zero sync or instant DB. So just a quick note there. There's also convex, which you know what, a lot of people love convex, and I haven't used convex, which makes me think that I need to really give this a fair shake because Convex also happens To do a lot of these things. Convex is just a really interesting platform. It handles the sync for you. It handles the back end. When I've been talking about InstantDB or Firebase or Superbase or any of these, people often say, give Convex a look. Convex is a paid product. Free and starter projects are free. Professional teams, $25 per developer per month. Wes Bos The per developer per month always kills me. The Instant gives you 10 devs to log in. Scott Tolinski Let me tell you about Con self-hosted convex you can run self-hosted on your own infrastructure if you want um they have documentation on how to do that so that's better than instant Db on that there is just a docker file for that and they even have docs on hosting with fly railway those types of things you can get this going yourself. So if you are that type of person, you can do that. Wes Bos Tanner Lindsley, he's sponsored by Convex, which you have to say that ahead of time. But he's been telling me, man, Convex is actually sick. People are sleeping on it. I certainly should. Scott Tolinski I think people are sleeping on it. I am too. For Convex, there's authentication baked in component typescript building blocks for your back end there's yeah man over 80 oauth integrations already existing for this thing resend Integration presence man so you can get like real-time sync presence detect it man maybe i am just really highly sleeping on Convex and maybe Convex is that thing. But maybe I should give this a rip to fraud. I'll just do like the paid or the free hobby one first. And then if I like it, then I'll figure out the because, yeah, there's so many interesting things going on in Convex. And every time I look at this website, I say, ooh, that's the one I'm going to try next. And then I reach for the more techie ones for some reason. But Convex is beautiful, and I think this seems like a great, great option. Electric SQL, ([Time 0:25:29](https://share.snipd.com/snip/35a132a2-4f5c-40f8-bdd8-c44b6b04b106)) - **ElectricSQL for Existing Apps** - ElectricSQL works well if you want local data with Postgres and keep your existing API endpoints. - Choose it when you want local-first features in only parts of a large app. Transcript: Scott Tolinski Because a lot of these things so far that we've mentioned are sync systems and data options. Electric SQL is the sync in cache on the front end, but you have your backend. So this does handle local data. It does handle the sync. But just like what we talked about with Replicash, you have to bring your entire backend still. You have to use your backend and you keep your own API endpoints as you typically would. Not a bad option for somebody who doesn't want to leave their API endpoints and their backend behind. If you like your backend, you can keep it. PG Lite is what this uses on the client side which is a wasm based postgres so pglight it's a light version of postgres in wasm in the browser and that's how it stores data on the client side Wes Bos And then on the server i assuming you're just writing you're just writing like resolvers or like what is that called like the code that you write to actually it gives you just an endpoint In this point. You got to save it to your database. Honestly, this is probably... For existing applications, this is probably what people will reach for. Because there's a lot of people that have large existing applications. And their entire app doesn't need to be done this. But maybe there's one part of the app that would be nice to have local first, but you're not about to start put that in something totally different because you need that data in the same Database for the rest of your application to work, right? So this makes a lot of sense that ([Time 0:28:04](https://share.snipd.com/snip/486354b9-a800-45a2-8792-8de27a09d925)) - **Sync Engines Vary by Scope** - PowerSync handles syncing only, leaving data management to developers. - PartyKit offers real-time multiplayer sync but also requires server-side functions. Transcript: Scott Tolinski I think a lot of these things have great use cases for different users. Another one here option is PowerSync. Now, PowerSync is just the sync server. It doesn't handle the local client data, and it doesn't handle the server side of things. It just handles the syncing, which is great if you're looking for a sync server. So another that is a sync server party kit, real time sync, you've used party kit before, Wes, that gives you multiplayer, multiplayer, real time sync, right? Some of the other ones, I guess, in this case, would be like Replicash. Replicash is more similar to Electric SQL, where it handles the client side data caching, and it handles the real-time syncing. But what it doesn't handle is how that data gets resolved on the server. So you have to write an endpoint for pushing, write an endpoint for pulling. You have to manage the versioning. You have to manage the sending of the patch messages. But ([Time 0:29:43](https://share.snipd.com/snip/d6ddfa87-d30c-4802-8396-cd444808ae33)) - **Fastest Way to Build Local-First Apps** - For rapid app builds needing local-first sync, InstantDB or Convex enable the fastest deployments. - Managed services cut out hosting and schema pains common with other sync engines. Transcript: Wes Bos I want to build, let's say I want to build an app where I want to use groceries. Let's call this like a grocery app, right? And me and my wife can add stuff to it. It's a list and you should be able to categorize it. There's AI in it. So it could be able to figure out like in which order should I pick these things given the layout of my store. Maybe there's offline because I'm in a grocery store. Like sometimes our Costco I can't get service in because of the weird roof on it. Let's say you want to build that app. It seems that's a relatively simple app, but it needs to have all of the it needs real time, but also needs local data. It needs sync and it needs a database i want to build that as fast as humanly possible which one of these are you reaching for which one am i reaching for not given the fact that you want always Scott Tolinski Reach for the one you haven't tried oh yeah because that was going to be my answer no not you but like someone listening yeah vu yeah as fast as possible if you're talking fast as possible I think you will get app to hands the fastest with instant db that not that that's what i would pick long term or something but just how fast it is to get i i built just fussing around. Wes Bos You don't care about pricing or anything. Scott Tolinski I list in that in like five seconds, like no time at all. If you're talking about a more established platform, Convex. It looks great. You're clicking a couple buttons. You're having the sync. One of the biggest barriers to entry to getting like app to your hands in a real world use case is figuring out like hosting of the sync server, hosting of your database, picking an ORM In many cases. Because like with zero sync and some of these, you still need to handle migrations. You still need to handle like your actual database. Like when I'm working in zero, I have to write my schema in drizzle and migrate that database in drizzle so that my Postgres database is ready for that local data sync, right? So if you have something that has a hosted version of it, like convex, you click a button, you get an endpoint or whatever, and then you can just connect it. That is going to be the fastest way. So you don't have to worry about where you're hosting that sync server. With Xero, again, it's an alpha. You're going to hit rocky spots around the way. Hosting the sync server, like with SST, can be a giant pain in the ass. You can always just host it anywhere you host a Node app, or if you're good with Docker, that's fine. But if you're talking 0 to 100, I think a managed service like Convex or InstantDB are going to be the fastest. I think Livestore is going to be very fast, too. I haven't gotten a version of it up and working yet because of the Svelte integration. But the fact that you can host it all on Cloudflare infrastructure, I could see that being you start a starter kit up, you push it up to Cloudflare, you got your data, you should be good To go on that one. You're not having to write migrations for that even. So beautiful. Wes Bos Yeah. Scott Tolinski So a lot of good options here. ([Time 0:35:42](https://share.snipd.com/snip/272d343e-e9de-4620-9806-0bc09689e141))