How 50+ Organizations Built Valkey by Roberto Luna Rojas

How 50+ Organizations Built Valkey by Roberto Luna Rojas 🔗 **Source:** [How 50+ Organizations Built Valkey by Roberto Luna Rojas](https://www.youtube.com/watch?v=vau8ck97Xho) > [!note]+ Document Notes > tags: [valkey, open-source-databases, in-memory-database] > author: Roberto Luna Rojas > source: Conference Talk > reference: https://www.youtube.com/watch?v=vau8ck97Xho > date created: 2024-12-20 11:30 > > # 🏷️ Title: How 50+ Organizations Built Valkey >  ## 🧰 Action Inbox - [H1] Evaluate Valkey as Redis alternative — 20% memory reduction, 2x performance improvement #database-migration - [H2] Implement vector similarity search for personalization — cosine distance with HNSW algorithm #ai-features - [H3] Test hash field expiration for session management — expires individual fields vs entire keys #performance-optimization ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Valkey GitHub](https://github.com/valkey-io/valkey) - [Linux Foundation](https://www.linuxfoundation.org/) ## 📝 Summary - Roberto Luna Rojas demonstrates how 50+ organizations collaborated to build Valkey, an open-source Redis-compatible in-memory database - Key improvements include 1M+ requests/second performance, 20% memory reduction, and new features like vector search and JSON support - Technical Steering Committee includes AWS, Google Cloud, Ericsson, Alibaba, Huawei, and Tencent - Valkey 9.0 introduces atomic slot migration, multiple databases in cluster mode, and hash field expiration - Live demo shows e-commerce personalization using vector embeddings, bloom filters, and JSON operations ## 📖 Main Content Valkey represents a significant community-driven fork of Redis 7.2.4, maintained under the Linux Foundation with BSD-3 license. The project emerged from collaboration between major cloud providers and technology companies, each contributing specific enhancements based on their operational experience. The performance improvements stem from architectural changes including enhanced I/O threading that leverages multiple CPU cores while maintaining the single-threaded processing engine's simplicity. Memory optimizations through redesigned hash tables provide substantial space savings for common key-value patterns. New capabilities include vector similarity search for AI applications, JSON document support with path queries, and bloom filters for probabilistic set membership testing. The upcoming version addresses long-standing clustering limitations with atomic slot migration and multi-database support in distributed deployments. ## 🗒️ Notes #### ⏱️ [00:01:30](https://www.youtube.com/watch?v=vau8ck97Xho&t=90) — [H1] What is Valkey - Key idea: Open-source high-performance key-value store fully compatible with Redis 7.2.4 - Evidence: "Supports caching, session store, message queues, and can be used as primary database" #### ⏱️ [00:03:45](https://www.youtube.com/watch?v=vau8ck97Xho&t=225) — [H1] Technical Steering Committee - Key idea: Six major organizations lead project direction with AWS, GCP, Ericsson, Alibaba, Huawei, Tencent - Evidence: "Built by contributors from open source community, vendor neutral under Linux Foundation" #### ⏱️ [00:05:12](https://www.youtube.com/watch?v=vau8ck97Xho&t=312) — [H2] Migration Compatibility - Key idea: Drop-in replacement requiring only container/endpoint changes - Evidence: "Same API, commands, REST protocol, client libraries - you don't have to change the client" #### ⏱️ [00:07:30](https://www.youtube.com/watch?v=vau8ck97Xho&t=450) — [H1] Valkey 8.0 Performance - Key idea: Achieved 1 million requests per second per node with 2x replication improvement - Evidence: "Enhanced I/O threading using extra CPU cores for network operations" #### ⏱️ [00:10:45](https://www.youtube.com/watch?v=vau8ck97Xho&t=645) — [H2] E-commerce Demo Setup - Key idea: Personalized product recommendations using vector similarity search - Evidence: "Indoor geek gets gaming headphones, outdoor fanatic gets rugged headphones for same search" #### ⏱️ [00:15:20](https://www.youtube.com/watch?v=vau8ck97Xho&t=920) — [H2] Memory Reduction Techniques - Key idea: 20% memory reduction through redesigned dictionary storage for 16-byte keys/values - Evidence: "6.3 million keys using 693MB reduced to 13% in 8.0, 20% in 8.1" #### ⏱️ [00:18:15](https://www.youtube.com/watch?v=vau8ck97Xho&t=1095) — [H1] Enhanced I/O Threading - Key idea: Multi-core utilization without affecting single-threaded processing engine atomicity - Evidence: "1.2 million requests per second - 230% improvement with 90% latency reduction at P100" #### ⏱️ [00:22:30](https://www.youtube.com/watch?v=vau8ck97Xho&t=1350) — [H2] Community Contributions - Key idea: Different organizations contributed modules based on their specific needs - Evidence: "Percona provided LDAP authentication requested by users but not prioritized by cloud providers" #### ⏱️ [00:25:45](https://www.youtube.com/watch?v=vau8ck97Xho&t=1545) — [H3] JSON Document Support - Key idea: Nested data structures with path queries eliminate full document serialization - Evidence: "Up to 128MB documents, 127 levels depth, atomic updates prevent race conditions" #### ⏱️ [00:30:12](https://www.youtube.com/watch?v=vau8ck97Xho&t=1812) — [H2] Bloom Filters Implementation - Key idea: Probabilistic set membership with configurable false positive rates - Evidence: "99.2% certainty, no false negatives, 90% memory reduction for tracking user-product interactions" #### ⏱️ [00:35:30](https://www.youtube.com/watch?v=vau8ck97Xho&t=2130) — [H1] Vector Similarity Search - Key idea: HNSW algorithm with cosine distance for AI-powered recommendations - Evidence: "Hierarchical navigable small world reduces layers, 1024 dimensions, hybrid search with keywords" #### ⏱️ [00:42:15](https://www.youtube.com/watch?v=vau8ck97Xho&t=2535) — [H1] Clustering Architecture - Key idea: Horizontal scaling through CRC16 hashing across 16,384 slots - Evidence: "Like library shelves - each node responsible for subset of data, client knows where to fetch" #### ⏱️ [00:45:30](https://www.youtube.com/watch?v=vau8ck97Xho&t=2730) — [H2] Hash Field Expiration - Key idea: Expire individual hash fields instead of entire keys - Evidence: "Coupon codes expire at midnight without recreating entire product key" #### ⏱️ [00:48:45](https://www.youtube.com/watch?v=vau8ck97Xho&t=2925) — [H1] Multiple Databases in Cluster Mode - Key idea: Logical partitioning with horizontal scaling for multi-tenancy - Evidence: "Mastodont servers can use separate databases while scaling across nodes" #### ⏱️ [00:52:10](https://www.youtube.com/watch?v=vau8ck97Xho&t=3130) — [H1] Atomic Slot Migration - Key idea: Append-only format prevents migration downtime for large keys - Evidence: "Broken since 2013 cluster mode creation - 10GB keys with millions of items migrate in batches" ## 📚 References - [Valkey Project](https://valkey.io/) - [Linux Foundation](https://www.linuxfoundation.org/) - [Ollama](https://ollama.ai/) - [HNSW Algorithm](https://arxiv.org/abs/1603.09320) ## 🗣️ Paraphrase - Valkey emerged as a community-driven alternative to Redis, maintaining full compatibility while adding significant performance and feature improvements through collaboration between major technology organizations - Performance gains come from intelligent multi-threading that uses additional CPU cores for I/O operations while preserving the atomic guarantees of single-threaded command processing - New data structures and algorithms enable modern use cases like AI-powered recommendations through vector search, efficient probabilistic filtering with bloom filters, and complex document operations with JSON support ## 📰 Newsletter Discover Valkey: The Community-Built Redis Alternative Delivering Major Performance Wins The open-source community has delivered something remarkable with Valkey - a Redis-compatible database that's not just a fork, but a significant leap forward. With 50+ organizations collaborating under the Linux Foundation, Valkey achieves over 1 million requests per second while reducing memory usage by 20%. Key benefits for developers include drop-in Redis compatibility (just change your container), enhanced I/O threading that actually uses your extra CPU cores, and powerful new features like vector similarity search for AI applications. The upcoming version 9.0 fixes cluster migration issues that have existed since 2013. Whether you're building high-performance caches, session stores, or AI-powered recommendations, Valkey offers a vendor-neutral path forward with impressive community backing from AWS, Google Cloud, Ericsson, and others. Resources: [Valkey GitHub](https://github.com/valkey-io/valkey), [Documentation](https://valkey.io/docs/), [Community Slack](https://valkey.io/community/) ## ✅ Action Items #### DevRel - Developer advocacy: Conference talks showcasing migration path, performance benchmarks, community success stories - Education: Workshops on vector search implementation, bloom filter use cases, JSON document patterns - Developer experience: Migration tooling documentation, compatibility testing guides - Developer marketing: Case studies from major adopters, performance comparison content - Community management: Slack moderation, GitHub issue triage, contributor onboarding #### OSPO - Policy & Governance: Contribution guidelines for multi-organization collaboration model - License Compliance: BSD-3 license compatibility review, dependency scanning - Security Management: Vulnerability disclosure process, security audit coordination - Software Inventory: Track Valkey adoption across organization, SBOM generation - Developer Support: Internal training on Redis migration, cluster configuration - Community Engagement: Technical Steering Committee participation model - Strategic Alignment: Open source database strategy alignment with Valkey roadmap #### Community - Maintainers: Multi-organization governance templates, decision-making processes - Contributors: Feature development coordination, testing infrastructure - Sponsors: Transparent resource allocation, engineering contribution tracking - Ecosystem: Client library compatibility, tooling integration #### WordPress - Core Performance: Evaluate Valkey for object caching, session storage improvements - Core AI: Vector search integration for content recommendations, semantic search - Meta: .org infrastructure caching evaluation, performance testing - Hosting: Cross-host compatibility testing, migration documentation - Community: Meetup presentations on high-performance caching strategies ## 🧪 Hypotheses - Organizations prefer community-governed database forks over vendor-controlled alternatives when performance parity exists - Multi-threading I/O improvements can achieve 2x+ performance gains without breaking single-threaded atomicity guarantees - Vector similarity search in key-value stores enables cost-effective AI personalization compared to specialized vector databases - Atomic slot migration resolves decade-old clustering limitations through append-only replication techniques ### 📊 Methodology - Performance benchmarking: Compare Valkey vs Redis across request throughput, memory utilization, latency percentiles - Migration analysis: Measure downtime, compatibility issues, performance changes during Redis-to-Valkey transitions - Feature adoption tracking: Monitor usage patterns of JSON, bloom filters, vector search across different use cases ### ❓ Questions - How do governance decisions scale across 50+ contributing organizations without creating bottlenecks? - What are the practical limits of horizontal scaling with the 16,384 slot architecture? - How does vector similarity search performance compare to dedicated vector databases at scale? ### 🔮 Future Research - Multi-cloud deployment patterns for Valkey clusters across different providers - Integration strategies with Kubernetes operators and service mesh architectures - Performance optimization techniques for mixed workload scenarios ### 💡 Implications - Practice: Drop-in Redis replacement enables immediate performance gains without application changes - Policy: Open source governance models can coordinate complex multi-vendor collaborations effectively - Security: Memory-resident databases require enhanced data protection and access control strategies - Education: Training programs needed for new features like vector search and advanced clustering - Funding: Community-driven development model demonstrates sustainable alternative to vendor-controlled databases ### 🧠 Learning Styles Assessment Reading level estimate: Flesch-Kincaid Grade Level 12 #### 👀 Visual learners - Present: Architecture diagrams, performance charts, slot distribution graphics - Improve: Vector space visualizations, memory layout diagrams, cluster topology maps #### 🔊 Aural learners - Present: Live conference presentation with Q&A - Improve: Podcast-style deep dive on governance model, technical interview series #### 🗣️ Verbal learners - Present: Technical terminology explanations, analogies (library shelves, dictionary storage) - Improve: Glossary of database terms, comparison matrices with Redis #### 🧩 Kinesthetic learners - Present: Live coding demo, hands-on e-commerce application - Improve: Migration lab exercises, performance testing worksheets, configuration templates #### 🧠 Logical learners - Present: Performance metrics, architectural decisions, algorithm explanations - Improve: Decision trees for feature selection, capacity planning calculators #### 🧑‍🤝‍🧑 Social learners - Present: Community collaboration stories, multi-organization governance - Improve: Discussion forums for use cases, peer review processes for implementations #### 🧘‍♀️ Solitary learners - Present: Technical documentation, code examples - Improve: Self-assessment quizzes, independent project templates #### 🌿 Naturalistic learners - Present: Ecosystem thinking about database communities - Improve: Sustainability considerations for infrastructure choices ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive link text - Improve: Alt text for architecture diagrams, data table headers #### Keyboard navigation - Present: Standard video controls, clear focus indicators - Improve: Slide navigation shortcuts, code example tabs #### Contrast and readability - Present: High contrast slides, readable font sizes - Improve: Color-blind friendly charts, increased line spacing #### Text resizing and reflow - Present: Responsive slide layout - Improve: Text-only versions of complex diagrams #### Media and captions - Present: Video presentation with audio - Improve: Synchronized captions, downloadable transcript #### Forms and interactions - Present: Clear demo application interface - Improve: Keyboard-accessible interactive elements ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global technology companies represented (US, Europe, China) - Improve: Broader geographic representation in governance, localized documentation #### Inclusive language - Present: Technical terminology appropriately defined - Improve: Plain language summaries, avoid cultural idioms #### Neurodiversity support - Present: Structured presentation, clear sections - Improve: Content warnings for rapid technical demos, alternative pacing options #### Representation - Present: Focus on technical merit and collaboration - Improve: