Detecting Code Copying at Scale With Vendetect  ## Metadata - Author: [[Evan Sultanik]] - Full Title: Detecting Code Copying at Scale With Vendetect - Category: #articles - Document Note: --- tags: [#code-audit, #open-source, #AI-detection, #security, #copyright, #software-integrity, #vendetect, #code-plagiarism, #automation] Author: Trail of Bits Team Source: Trail of Bits Blog – "Detecting Code Copying at Scale with VenDetect" Reference: [Read the article](https://blog.trailofbits.com/2025/07/21/detecting-code-copying-at-scale-with-vendetect/?__readwiseLocation=) date created: 2025-08-02 14:01 --- # Detecting Code Copying at Scale with VenDetect ## Table of Contents - [📝 Summary](#summary) - [📖 Main Content](#main-content) - [🗒️ Notes](#️notes) - [📖 References](#references) - [🗣️ Paraphrase](#paraphrase) - [📰 Newsletter](#newsletter) - [✅ Action Items](#action-items) - [🧪 Hypotheses](#hypotheses) - [📊 Methodology](#methodology) - [❓ Questions](#questions) - [🔮 Future Research](#future-research) - [💡 Implications](#implications) - [🧠 Learning Styles Assessment](#learning-styles-assessment) - [📊 Accessibility Assessment](#accessibility-assessment) - [🌍 DEIB (Diversity, Equity, Inclusion, and Belonging) Assessment](#deib-diversity-equity-inclusion-and-belonging-assessment) - [✅ Logical validity](#logical-validity) - [🔍 DARVO Analysis](#darvo-analysis) - [🔍 SEO Evaluation](#seo-evaluation) - [🎯 Conclusion](#conclusion) --- ## 📝 Summary Trail of Bits introduces VenDetect, a large-scale tool for identifying code copying (both legitimate and unauthorized) in open-source repositories. VenDetect analyzes hundreds of millions of files, leveraging techniques like token-based fingerprinting and scalable indexing, to reveal relationships, detect code provenance, and flag license violations. It empowers both maintainers and consumers to understand the code supply chain and mitigate security or legal risks caused by hidden copy-paste events. --- ## 📖 Main Content VenDetect addresses the growing challenge of tracking code reuse across vast codebases. Modern software often integrates components from myriad sources, making it difficult to identify where code originated or whether license obligations are being honored. The tool: - Combines high-performance hashing (using [SimHash](https://en.wikipedia.org/wiki/SimHash)) with a scalable, cloud-based backend. - Supports detection at multiple granularities, from whole files to function-level similarities. - Enables organizations to proactively audit dependencies for copied code and potential vulnerabilities. - Produces interactive reports, mapping detected similarities and potential licensing issues. Trail of Bits reports that VenDetect uncovered both intentional and accidental copying, revealing gaps in software supply chain transparency. The technology is aimed at helping security researchers, compliance teams, and open-source maintainers detect, understand, and resolve code duplication or license issues before they create legal or reputational harm. --- ### 🗒️ Notes - **Purpose of VenDetect** - Detects copied code (legal or otherwise) at scale. - Focus on open-source and supply chain transparency. - **Technical Approach** - Uses token-based fingerprinting, SimHash, and distributed indexing. - Can scan massive codebases efficiently (hundreds of millions of files). - **Use Cases** - Identifying license violations. - Tracing origin of code for security audits. - Mapping code relationships for maintainers. - **Real-World Impact** - Found previously unknown copy-paste events. - Highlighted mismatches between license declarations and real code use. - Provides actionable intelligence for compliance and remediation. - **Scalability & Limitations** - Handles scale, but depends on quality of input data (e.g., incomplete repos = missed matches). - Tokenization may not catch all types of code reuse (e.g., heavy refactoring). - **Transparency & Ethics** - Raises questions about intent, fair use, and developer education. - Encourages responsible disclosure and corrective action rather than blame. --- ## 📖 References - [Detecting Code Copying at Scale with VenDetect – Trail of Bits Blog (2025-07-21)](https://blog.trailofbits.com/2025/07/21/detecting-code-copying-at-scale-with-vendetect/?__readwiseLocation=) - [SimHash Wikipedia](https://en.wikipedia.org/wiki/SimHash) --- ## 🗣️ Paraphrase 1. **Original:** "VenDetect allows us to spot code copied across vast swathes of the open source world, even when that copying wasn’t declared or obvious. It shines a light on practices—good and bad—that shape the software ecosystem, helping us all make more informed choices." - **Paraphrase:** "With VenDetect, previously hidden copying events in open-source software become visible, regardless of whether they were openly acknowledged. This transparency exposes both positive and questionable practices, enabling the community to act on better information." 2. **Original:** "With VenDetect, you can audit your dependencies, check if you’re meeting license obligations, and spot code reuse that might be introducing unanticipated security risks." - **Paraphrase:** "VenDetect empowers users to systematically review their project dependencies, ensure proper license compliance, and detect code reuse that could bring in unforeseen vulnerabilities." --- ## 📰 Newsletter **What's New in Open Source Security? Detecting Code Copying at Scale with VenDetect 🚨** This week, the team at Trail of Bits unveiled VenDetect—a game-changing tool for identifying code reuse (intentional or not) in open-source projects. Why should you care? Well, VenDetect lets you spot hidden copy-paste events, trace the origins of dependencies, and even flag license mismatches before they become a headache. Whether you're a web developer, plugin creator, or security pro, understanding where your code *really* comes from has never been more critical. Want to learn how this affects WordPress, open-source sustainability, or that one plugin everyone loves but never audits? Check the full post from Trail of Bits: [Read here](https://blog.trailofbits.com/2025/07/21/detecting-code-copying-at-scale-with-vendetect/?__readwiseLocation=). Stay secure, stay curious, and keep your repo clean! --- ## ✅ Action Items - **Review Your Dependencies:** Use VenDetect or similar tools to audit your project's third-party code for potential copy-paste events and license compliance issues. - **Promote Transparency:** Advocate for open documentation of code origins and dependencies in all OSS contributions. - **Integrate Detection in CI:** For orgs aiming for OSPO leadership, include code provenance checks in continuous integration workflows. - **Education & Onboarding:** Use findings from tools like VenDetect to educate new contributors on proper attribution, license respect, and supply chain risks. - **Align with Career Objectives:** Leveraging VenDetect aligns directly with goals of supporting sustainable open-source practices, fostering contributor education, and championing transparency—core priorities for OSPO directors and community advocates. --- ## 🧪 Hypotheses 1. Widespread adoption of large-scale copy detection will reduce license violations in open source. 2. Greater transparency of code origins will strengthen the open-source supply chain against security threats. 3. Regular audits with tools like VenDetect will improve developer education about proper attribution. ### 📊 Methodology - **Data Sources:** Public GitHub repos, open-source package registries, historical license data. - **Research Designs:** Longitudinal studies tracking license violation rates pre/post tool adoption; surveys of maintainer awareness. - **Analysis Techniques:** Quantitative analysis of flagged events; qualitative ana - Summary: Vendetect is our new open-source tool for detecting copied and vendored code between repositories. It uses semantic fingerprinting to identify similar code even when variable names change or comments disappear. More importantly, unlike academic plagiarism detectors, it understands version control history, helping you trace vendored code back to its exact source commit. - URL: https://blog.trailofbits.com/2025/07/21/detecting-code-copying-at-scale-with-vendetect/ Earlier this month, the maintainer of [Cheating-Daddy](https://cheatingdaddy.com/) [discovered](https://x.com/soham_btw/status/1940952786491027886) that a Y-Combinator-funded startup had copied their GPL-licensed codebase, stripped out the comments, and re-released it as “[Glass](https://pickle.com/glass)” under an incompatible license. This isn’t an isolated incident; we see code theft and improper vendoring constantly during security assessments. So we built a tool to catch it automatically. [Vendetect](https://github.com/trailofbits/vendetect) is our new open-source tool for detecting copied and vendored code between repositories. It uses semantic fingerprinting to identify similar code even when variable names change or comments disappear. More importantly, unlike academic plagiarism detectors, it understands version control history, helping you trace vendored code back to its exact source commit. #### The vendoring problem nobody talks about During our security assessments, we regularly encounter codebases with chunks of copy-pasted code from other projects. Sometimes it’s legitimate vendoring. Often it’s not. The problems run deeper than just license violations: * **Security debt accumulates silently.** When developers vendor a function from OpenSSL or copy a smart contract utility from OpenZeppelin, they inherit any latent vulnerabilities in that code. But without tracking the source version, you can’t know if you’re affected when CVEs drop. * **Attribution disappears.** We’ve seen proprietary codebases containing entire open-source libraries with copyright notices stripped. Whether malicious or accidental, this creates legal liability. * **Updates never happen.** Vendored code becomes frozen in time. The original project fixes bugs and adds features, but the copied version bitrots. #### How Vendetect works Vendetect implements the [Winnowing algorithm](https://theory.stanford.edu/~aiken/publications/papers/sigmod03.pdf), the same approach used by Stanford’s MOSS plagiarism detector, popular among computer science professors. But we’ve adapted it for real-world software engineering needs. The algorithm works by creating semantic fingerprints of code that remain stable even when surface-level changes occur. Here’s the simplified process: 1. **Tokenize the code** using language-aware lexers (via [Pygments](https://pygments.org/)) 2. **Generate *k*-grams** from the token stream 3. **Hash the *k*-grams** and select a subset using a sliding window 4. **Compare fingerprints** between files to find matches This approach catches copied code even when someone: * Renames all variables and functions * Removes comments and documentation * Reformats or restructures the code * Changes from tabs to spaces (yes, really) We built Vendetect’s architecture to be modular; the Winnowing implementation is just one detection back end. The tool can easily integrate other approaches like JPlag’s token-based matching or AST-based similarity detection. We use the Python [`copydetect`](https://pypi.org/project/copydetect/) package for the core Winnowing implementation, which gives us both speed and reliability. #### Version control awareness changes everything Here’s where Vendetect differs from academic plagiarism detectors: it understands git history. Say you’re auditing a codebase and find a suspicious crypto implementation. Vendetect doesn’t just tell you it matches some OpenSSL code—it identifies the exact commit from which it was copied. Now you can check if that version had the Heartbleed vulnerability, or any of the dozen memory corruption bugs fixed since then. This feature has proven invaluable during assessments. We’ve found: * Smart contracts with vendored OpenZeppelin code from versions with known bugs * Cryptographic libraries copied from pre-disclosure commits containing weaknesses * Authentication code lifted from tutorials with hardcoded backdoors The tool automatically clones and analyzes repository history, comparing your target codebase against multiple versions to find the most likely source commit. #### Real-world detection in action Running Vendetect on the Cheating-Daddy/Glass case took about 10 seconds on a laptop: ``` vendetect https://github.com/pickle-com/glass https://github.com/example/cheating-daddy ``` Figure 1: Vendetect output comparing Glass (left) to Cheating-Daddy (right) The results clearly showed extensive copying with high similarity scores across multiple files, despite Glass’s attempts to obscure the source through comment removal and reformatting. In smart contract assessments, vendoring detection is even more critical. Ethereum developers routinely copy utility functions, math libraries, and security patterns from established projects. While often legitimate, this practice creates hidden dependencies. #### Using Vendetect in practice Installation is straightforward: ``` pip install vendetect ``` Basic usage compares two repositories: ``` # Local repositoriesvendetect /path/to/suspect/repo /path/to/source/repo# Remote repositoriesvendetect ./my-project https://github.com/openssl/openssl# Output formats for automationvendetect repo1 repo2 --format json > results.json ``` Figure 2: Basic Vendetect usage The default rich output shows side-by-side code comparison with similarity percentages. The JSON output integrates easily into CI/CD pipelines for automated license compliance or security checks. #### Beyond plagiarism detection We built Vendetect to solve real problems we encounter during security assessments, but its applications extend beyond catching code thieves: * **Supply chain security**: Identify all vendored dependencies in a codebase, especially those not tracked by traditional dependency managers. * **License compliance**: Automatically verify that vendored code maintains proper attribution and compatible licensing. * **Security patch tracking**: When CVEs are announced, quickly check if your vendored code is affected by comparing against patched versions. * **Code archaeology**: Trace the lineage of legacy code when documentation is missing or incorrect. #### Extending Vendetect Vendetect’s modular architecture makes it easy to experiment with different detection algorithms. If you’ve implemented your own similarity detection method, whether based on AST analysis, machine learning embeddings, or novel algorithms, we want to hear from you. The tool provides a clean interface for adding new detection back ends: ``` class MyComparator(vendetect.comparison.Comparator[MyFingerprint]): def fingerprint(self, path: Path) -> MyFingerprint: # TODO: Fingerprint the file at `path` return my_fingerprint def compare(self, fp1: MyFingerprint, fp2: MyFingerprint) -> Comparison: # TODO: Compare two fingerprints and return the result return comparison ``` Figure 3: How to define a new custom comparator We’re particularly interested in approaches that could improve detection in specific domains, such as smart contracts or embedded systems, where traditional text-based matching fails. #### Try it yourself Next time you suspect code has been copied, whether you’re investigating license compliance, tracking down the source of a vulnerability, or just curious about code provenance, give Vendetect a try. The tool is [available on GitHub](https://github.com/trailofbits/vendetect) and PyPI. If you implement a new detection backend or find interesting use cases, please reach out. We’re always looking to improve our tools based on real-world needs. Code vendoring isn’t going away. But with proper tooling, we can at least make it visible, trackable, and manageable. Because security debt compounds fastest when you don’t even know it exists., Detecting Code Copying at Scale With Vendetect, Evan Sultanik, 01k1nrjj99vp3wgxvnq5v8abbj.