Never. Let. AI. Write. Your. Tests.  ## Metadata - Author: [[The Changelog: Software Development, Open Source]] - Full Title: Never. Let. AI. Write. Your. Tests. - Category: #podcasts - URL: https://share.snipd.com/episode/cf3b2d0c-8ba4-440c-960d-a116a9801d5b ## Document Notes ## Highlights - **Never Let AI Write Tests** - Never let AI write your tests; they encode your true intentions and edge cases. - Use AI only for test planning, scenario suggestion, and debugging, but guard your test code zealously. Transcript: Jerod Santo Never let AI write your tests. Developer DeWank's field guide to a new way of building software starts off as a pretty typical, here's how to be productive coding with AI, but then he says something near the end, and Emphatically so, that I haven't heard anybody say. Quote, now we come to the most important principle in AI-assisted development. It's so important that I'm going to repeat it in multiple ways until it's burned into your memory. Never let AI write your tests. Tests are not just code that verifies other code works. Tests are executable specifications. They encode your actual intentions, your edge cases, your understanding of the problem domain. High performers excel at both speed and stability. There's no tradeoffs. Tests are how you achieve both. End quote. DeWank says AI can help with test planning, suggest test scenarios, debug, and analyze test features, but that it should never touch test files, write test code, or modify test expectations. Quote, your tests are your specification. They're your safety net. They're the encoded wisdom of every bug you've fixed and every edge case you've discovered. Guard them zealously. End quote. I'm not sure if I agree or not. ([Time 0:00:37](https://share.snipd.com/snip/f9fba122-5440-42f4-833c-6e7e257a9976)) - **Rise of Judgment Over Skill** - AI commoditizes technical skills, making judgment the critical skill to develop. - The value lies in choosing what tasks or perspectives to focus on, not just technical ability. Transcript: Jerod Santo The rise of judgment over technical skill. Ever since ChatGPT launched our current AI madness, developers have been asking ourselves and each other what it all means in the long term. We still don't have that answer yet, but I can confidently say that at least in the medium term, it means we must move up the value chain because the once cherished technical skills we've Acquired are being commoditized at a blistering pace. There's nothing new under the sun. Quote, in 1995, musician and producer Brian Enno made a profound observation about computer sequencers that has become increasingly relevant in our AI-powered world. Quote, the great benefit of computer sequencers is that they remove the issue of skill and replace it with the issue of judgment. With Cubase or Photoshop, anybody can actually do anything. And you can make stuff that sounds very much like stuff you'd hear on the radio or looks very much like anything you see in magazines. So the question becomes not whether you can do it or not, because any drudge can do it if they're prepared to sit in front of the computer for a few days. The question then is, of all things you can do now, which do you choose to do? End quote. You know what? Adam and I had a similar conversation about digital photography while on a photo walk in New York City years ago. It was my contention then that the skills required to take great pictures were trending towards zero, and when we get to that point, which we're pretty close to now, the only thing that Would matter is taste, which is just another form of judgment. In other words, it's a way of answering the question of all the perspectives you can now capture, which do you choose to capture? ([Time 0:03:12](https://share.snipd.com/snip/0196ef3e-c487-4a5b-8658-3ce825ec5aaa)) - **Claude Code is My Computer** - Peter Steinberger uses Claude Code daily, saving an hour each day without issues. - Claude Code integrates deeply with the command line, improving productivity beyond IDE plugins. Transcript: Jerod Santo Clodcode is my computer. Here's Peter Steinberger, quote, I run Clodcode in no prompt mode. It saves me an hour a day and hasn't broken my Mac in two months. The $200 per month max plan pays for itself, end quote. This echoes the sentiment that Steve Yege impressed upon us on last week's show. After recording that, I took Steve's advice and gave Claude Code the old college try at writing a few scripts that I procrastinated because they were just too much work for their perceived ROI. Color me impressed. The first script Claude wrote was delivered so well on my specs that I decided to vibe code the second one and didn't even look at the code itself. Worked great. Peter says this about Cloud Code. Quote, Cloud Code shines because it was built command line first, not bolted onto an IDE as an afterthought. The agent has full access to my file system, if you're bold enough, can execute commands, read output, and iterate based on results. End quote. I think that's right. I like Cloud Code more than I like Claude inside of Zed. It's even more natural in my terminal than it is in my editor for some reason. ([Time 0:06:10](https://share.snipd.com/snip/2a5066c7-2b91-49ca-a3ef-5eedd5727551)) - **Memvid's Memory Claims Debunked** - Memvid claims AI memory management through video encoding, but tests show it's inefficient. - Its usage causes higher disk space consumption and slower searches than traditional vector databases. Transcript: Jerod Santo So here's a nice balancing story. A graduate created a software project that got a lot of attention online. Like a lot of attention. Its pitch, quote, Memvid revolutionizes AI memory management by encoding text data into videos, enabling lightning-fast semantic search across millions of text chunks with sub-second Retrieval times. Unlike traditional vector databases that consume massive amounts of RAM and storage, Memvid compresses your knowledge base into compact video files while maintaining instant Access to any piece of information. End quote. Now on his face, that sounds amazing, but it also sounds kind of weird. Why would encoding text into video use less disk space or make anything faster? Well, it turns out it doesn't. Quote, testing shows this library's performance is the opposite of what the ReadMe claims. Your text will take 100x more disk space. Searches will be 5x slower. Setup will take hours, not minutes. This library will cause serious problems at production scale. The Readme's performance claims are backwards, end quote. That was posted as an issue on the repo. On the heels of this discovery came a new contribution, a proposal for Memvid 1.0, the universal, streamable, self-contained AI memory format. Does that sound ambitious? Does it sound sloppy? One commenter sure thinks so. Quote, GitHub is now infested with AI slop, AI generated repo with obvious overhead and no practical usages, people that has AI replaced brains, giving stars to this, and AI generated Issues. Perfect. End quote. I guess the AI slopping will continue until morale improves. ([Time 0:07:28](https://share.snipd.com/snip/183c7610-c289-4c0c-acda-0e2c9ea470fb))