How Binary Dependencies Work Across Different Languages

How Binary Dependencies Work Across Different Languages 🔗 **Source:** [How Binary Dependencies Work Across Different Languages](https://vlad.website/how-binary-dependencies-work/) > [!note]+ Document Notes > tags: [FFI, binary-dependencies, programming-languages] > author: Vlad > source: Vlad's Website > reference: [https://vlad.website/how-binary-dependencies-work/] > date created: > > --- > > # 🏷️ Title: How Binary Dependencies Work Across Different Languages >  ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - Linux dynamic linking and ELF basics - Language FFIs (Rust, Python, Ruby, Node.js) - libffi, ctypes/cffi, nanobind/pybind11, cgo --- ## 📝 Summary - What happened: The article explains how code in higher-level languages calls into precompiled C libraries via dynamic linking, dynamic loading (FFI), and extension modules, with a brief note on Go’s cgo. - Why it matters: Binary dependencies are often “phantom” (not declared in manifests), creating security, observability, and sustainability blind spots. - Key stakeholders: Application developers, security teams, OS vendors, open source maintainers of low-level libraries. - Risks and tradeoffs: Hidden dependencies, ABI mismatch, data-conversion cost, performance vs. complexity in extension modules. - Bottom line: Understanding how binary dependencies work is a prerequisite for better tooling, security policy, and sustainable funding for critical C libraries. --- ## 📖 Main Content - The author defines a binary dependency as a compiled library (often C) your program calls without compiling it together with your code. These are common whenever Python, JS, Ruby, or Rust talk to C libraries like OpenBLAS or libcurl. - Binary dependencies are often not recorded in package manifests, making them “phantom dependencies” with implications for supply-chain security and funding critical infrastructure libraries. - General pattern for calling a C function in a shared object (`c_lib.so`): - Load the library file. - Know the function’s signature (name, return type, argument count and types). - Convert host-language data to C representations (e.g., null-terminated strings). - Find the symbol’s address inside the shared object. - Generate a call following the platform ABI (e.g., System V on x86_64: arguments in `rdi`, `rsi`, `rdx`, return value in `rax`). - **Dynamic linking (compiled hosts like Rust):** - At compile time, the binary records that it depends on a shared library (e.g., `libcurl.so`) and needs a symbol (e.g., `curl_easy_init`). - At runtime, the dynamic linker consults these records, searches the required shared libraries, and patches in the function address. - In Rust, you use `#[link(name = "c_lib")]` and `extern "C"` declarations with C-compatible types (`c_char`, etc.), then the compiler and kernel handle symbol lookup and call generation. - **Dynamic loading (interpreted hosts like Python, Ruby, Node.js):** - There is no compile step, so linking must happen at runtime. - OS APIs like `dlopen` and `dlsym` open a shared library and look up symbols by name. - You still must generate calls with the correct ABI, so most languages rely on `libffi` (or an equivalent) to construct calls at runtime. - Python’s `ctypes`/`cffi`, Ruby-FFI, and Node’s FFI libraries are thin, language-specific layers over `libffi`, plus convenience type-conversion helpers. - **Dynamic linking with extension modules (C modules embedded in higher-level runtimes):** - You write C (or C++) code that: - Knows how to construct and interpret the host language’s data types (e.g., Python `PyObject*`). - Calls C libraries directly via normal dynamic linking (`#include <c_lib.h>` and link against `c_lib.so`). - The host runtime special-cases imports like `import ext` / `require("ext")` to load compiled extension modules (e.g., `ext.so`). - Binding generators (nanobind, pybind11, Boost.Python, nbind) can inspect your C/C++ function signatures and autogenerate the glue. - Performance win: You can do many operations inside C, converting between host types and C types only when needed, instead of on every FFI call. - The article closes with Go’s `cgo`, which acts like a lightweight, integrated extension-module plus FFI system, including its own call machinery rather than relying on external `libffi`. --- ### 🗒️ Notes - Binary dependencies let high-level languages reuse highly optimized C code without reimplementing it, but often escape manifest-based dependency tracking. - Core steps to call into a C shared object are consistent across methods: know the library, know the signature, convert data, locate the symbol, and perform a call that respects the ABI. - Dynamic linking (compiled languages): - The compiler emits references to undefined symbols and records needed libraries. - The runtime linker resolves these at load time, using information like: “program depends on `libcurl.so` and needs symbol `curl_easy_init`.” - Result: minimal manual work; you only annotate foreign functions and use C-compatible types. - Dynamic loading (interpreted languages): - You explicitly load libraries (`dlopen`) and fetch symbols (`dlsym`) at runtime. - But generating ABI-correct calls is hard, so an FFI library like `libffi` provides a generic “call foreign function” mechanism given argument types, return type, and pointer to code. - Language-level wrappers (Python `ctypes`/`cffi`, Ruby-FFI, Node FFI) depend on compiled components in the interpreter to bridge to `libffi`. - All approaches must grapple with type representation mismatches: for instance, C strings vs. Python or Rust strings; integers and pointers with different sizes; structs and ownership semantics. - Extension modules: - Move much of the FFI complexity into C modules that speak both the host language’s API and C’s ABI. - Host code sees a normal module (`import ext`) that exposes high-level functions; inside, extension code calls C libraries via ordinary C linking. - Binding generators read C++ declarations and emit the glue to build host-language objects and back. This centralizes type conversion logic and can often auto-update when C APIs change. - Data-conversion strategy affects performance: - Repeated cross-boundary calls (host ↔ C) are costly. - Extension modules can keep work in C longer, reducing boundary crossings. - Go’s `cgo` implements its own small FFI and extension-like scheme, consistent with Go’s self-contained toolchain philosophy. - The author frames this as groundwork for later research into security and sustainability, including how “phantom” binary dependencies complicate SBOMs, auditing, and funding models for low-level libraries. --- ## 📚 References - How Binary Dependencies Work Across Different Languages — Vlad’s Website: https://vlad.website/how-binary-dependencies-work/ - libffi project: https://sourceware.org/libffi/ - Python `ctypes`: https://docs.python.org/3/library/ctypes.html - Python `cffi`: https://cffi.readthedocs.io/ - pybind11: https://pybind11.readthedocs.io/ - nanobind: https://github.com/wjakob/nanobind - Ruby-FFI: https://github.com/ffi/ffi - Node-FFI: https://github.com/node-ffi/node-ffi - Go `cgo`: https://go.dev/blog/cgo --- ## 🗣️ Paraphrase - Paragraph A: Many languages call precompiled C libraries instead of recompiling them; these binary dependencies are often invisible in dependency manifests, which raises security and sustainability issues. - Paragraph B: There are three main mechanisms—dynamic linking, dynamic loading/FFI, and extension modules (plus Go’s special case)—which all follow the same fundamental steps for locating symbols, converting data, and making ABI-correct calls. --- ## 📰 Newsletter Binary dependencies quietly power much of the modern stack—including WordPress hosting platforms and performance-critical plugins. Vlad’s deep dive into dynamic linking, FFIs, and extension modules is a handy primer for anyone shipping PHP, JS, or Rust that leans on C libraries. For WordPress builders, this is a reminder to document your native dependencies, track them in SBOMs, and understand how Node-based build tools or PHP extensions talk to C. Pair this article with resources on PHP extensions, WebAssembly in the browser, and modern JS tooling to tighten your supply-chain story across themes, plugins, and hosting environments. --- ## ✅ Action Items #### DevRel - Map common binary-dependency paths (PHP extensions, Node toolchains, image/SSL libraries) in your ecosystem docs. - Prepare a talk/demo explaining FFI vs. extension modules for plugin/theme authors. - Add examples and quickstarts for safe native-module usage in starter repos. - Run a content campaign around “hidden dependencies” and supply-chain visibility. - Survey your community about where they rely on native extensions and what tooling they lack. #### OSPO - Require recording binary dependencies in SBOMs and internal dependency registries. - Establish review processes for adopting new C/C++/Rust libraries via FFI or extensions. - Define a policy for ABI compatibility checks and security advisories on native libs. - Maintain an inventory of critical shared libraries across products and build pipelines. - Fund or support upstream maintainers of core binary libraries you rely on. #### Community - Host clinics on auditing projects for undocumented binary dependencies. - Offer maintainer office hours focused on extension modules and FFIs. - Provide onboarding guides for contributors who want to work on native bindings. - Share case studies on performance wins and pitfalls from moving logic into C. #### WordPress - Core: Document and track all PHP extensions and OS-level libraries core assumes (e.g., image, XML, crypto). - Core Performance: Profile where native libraries could help vs. where FFI overhead would negate gains. - Core AI: Define guidelines for native dependencies in AI inference/integration plugins. - Core Program Team: Align release processes with dependency and SBOM updates for native components. - Mobile: Track binary dependencies in iOS/Android builds, including media and crypto libraries. - Playground: Explore demos of PHP+native libraries via WebAssembly or virtualized environments. - Community: Add binary-dependency questions to event and meetup checklists. - Training: Create lessons on “How WordPress uses native libraries” for dev education. - Marketing: Communicate supply-chain diligence (including native deps) as a trust and security story. - Support: Add runbooks for debugging issues caused by missing or mismatched system libraries. - Design: Ensure UX around system-requirement checks is clear for hosts and users. - Documentation: Centralize a list of required/recommended PHP extensions and system libraries. - Photos/TV/Openverse: Track and document any native codecs or media libs in ingestion/transcoding. - Polyglots: Note any binary dependencies in translation tooling or build steps. - Accessibility: Ensure any native accessibility tooling interaction is documented and testable. - Meta: Surface binary-dependency health in .org infrastructure dashboards. - CLI: Document native tools used by `wp-cli` commands and their required versions. - Hosting: Provide consensus checklists for system libraries required for WordPress performance and security. - Tide: Include static information about native dependencies in scoring where relevant. - Themes/Plugins: - Require declaration of binary dependencies for plugins/themes that depend on PHP extensions or Node-native modules. - Encourage SBOM-inclusive releases for complex stacks. #### The WPOCC - Fund microgrants for maintainers of critical C libraries heavily used in the WordPress ecosystem. - Run documentation and governance sprints focused on native-dependency policies. - Pilot matched grants tied to improving SBOM and dependency transparency in popular plugins/themes. ### FAIR - Promote tools that locally record and verify binary dependency provenance and signatures. - Encourage projects to publish funding statements and long-term support expectations for critical native components. - Offer governance templates that explicitly cover native-dependency review, adoption, and deprecation. --- ## 🧪 Hypotheses - H1: Projects using extension modules rather than vanilla FFI for hot paths achieve significantly better throughput at equal or lower CPU load. - H2: Most production applications with binary dependencies lack complete manifest-level records of those dependencies. - H3: Teams with explicit ABI-compatibility policies experience fewer production incidents related to native libraries. - H4: Auto-generated bindings (nanobind/pybind11/etc.) reduce FFI bugs compared to hand-written bindings, without materially increasing runtime overhead. --- ### 📊 Methodology - For H1: Benchmark representative workloads (e.g., numerical kernels, image processing) across FFI-only vs. extension-module implementations in Python/Node/Ruby, comparing requests per second and latency. - For H2: Sample a set of open-source and internal services, compare runtime-loaded libraries (via `ldd`, `lsof`, `procfs`) to declared manifests and SBOMs. - For H3: Survey organizations on ABI policies and correlate with incident postmortems involving native libraries over a multi-year period. - For H4: Analyze historical bug reports and CVEs involving FFI layers in projects using auto-generated vs. manual bindings; re-benchmark call overhead for both. --- ### ❓ Questions - How can package managers incorporate binary dependency discovery and tracking automatically? - What’s the best developer experience for safely adding C libraries to a high-level project? - How should SBOM standards evolve to represent runtime-discovered dependencies cleanly? - Can WebAssembly meaningfully replace traditional FFIs for a slice of these use cases? --- ### 🔮 Future Research - Tooling to automatically detect and list all dynamically loaded libraries at runtime, integrated into CI/CD. - Comparative studies of performance and ergonomics across FFI approaches (dynamic loading vs. extension modules vs. Wasm). - Funding and governance models specifically tailored to low-level C libraries that underpin multiple ecosystems. - Safer, higher-level ABI description languages that can generate bindings across many targets. --- ### 💡 Implications - Practice: Teams must treat binary dependencies as first-class citizens—tracked, versioned, and tested. - Policy: Security and compliance frameworks should explicitly require visibility into native libraries. - Security: Phantom dependencies are a major attack surface; FFI paths need special scrutiny. - Education: Developer curricula should cover dynamic linking, FFIs, and extension modules as supply-chain topics. - Funding: Critical-path C libraries deserve sustainable funding, given their cross-language impact. --- ### 🧠 Learning Styles Assessment - Reading level: Roughly upper-intermediate technical (equivalent ~Flesch-Kincaid Grade 11–13) due to jargon and systems concepts. #### 👀 Visual learners - Present: Conceptual diagrams (described), code snippets for Rust/Python/Ruby/Node. - Improve: Concrete architecture diagrams showing process, shared objects, and call flows. #### 🔊 Aural learners - Present: None noted. - Improve: Short audio walkthrough or podcast-style explainer of each method. #### 🗣️ Verbal learners - Present: Clear definitions (binary dependencies, dynamic linking/loading, extension modules) and analogies across languages. - Improve: A small glossary and side-by-side comparison table of terminology per language. #### 🧩 Kinesthetic learners - Present: Implicit exercises (try Rust extern "C", Python cffi, etc.). - Improve: Step-by-step labs: “call this C function from Python/Rust/Node” with guided commands. #### 🧠 Logical learners - Present: Systematic breakdown of steps (type definition, conversion, symbol finding, call generation). - Improve: Explicit flowcharts or state diagrams for each mechanism. #### 🧑‍🤝‍🧑 Social learners - Present: Invitation for feedback at the end. - Improve: Discussion prompts, GitHub repo for examples, and suggested pair-exercises. #### 🧘‍♀️ Solitary learners - Present: Text format suitable for deep solo reading and experimentation. - Improve: Reflection questions on their own stack’s binary dependencies. #### 🌿 Naturalistic learners - Present: Light framing around ecosystem and sustainability. - Improve: Stronger links to environmental impact (e.g., performance/efficiency ties) and ecosystem health. --- ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Reasonably structured headings and paragraphs. - Improve: Ensure all diagrams have descriptive alt-text and logical heading hierarchy. #### Keyboard navigation - Present: Standard web article behavior. - Improve: Confirm focus order, skip links, and that code samples are easily tabbable/selectable. #### Contrast and readability - Present: Depends on site theme but likely adequate for text. - Improve: Check contrast ratios and line spacing for long technical sections. #### Text resizing and reflow - Present: Likely responsive layout. - Improve: Verify at 200% zoom and support reduced-motion preferences. #### Media and captions - Present: Static images only. - Improve: Add text descriptions for diagrams and downloadable examples. #### Forms and interactions - Present: Minimal or none. - Improve: N/A beyond standard contact/comment forms. #### Standards reference - Map improvements to WCAG 2.2 AA, especially 1.4.x (contrast), 1.4.10 (reflow), 2.1.x (keyboard). --- ### 🌍 DEIB Assessment #### Cultural breadth - Present: Globally neutral technical content. - Improve: Examples from diverse ecosystems (not only Linux/x86_64) and broader tooling. #### Inclusive language - Present: Generally neutral and clear. - Improve: Add more plain-language summaries for dense sections. #### Neurodiversity support - Present: Sectioned content; repeated patterns (steps per mechanism). - Improve: More frequent subheadings, bullet summaries, and highlighted key points. #### Representation - Present: Not strongly person-focused. - Improve: Include diverse attributions in examples or further-reading suggestions. #### Psychological safety - Present: Respectful, non-derisive tone, invites corrections. - Improve: Explicitly welcome questions from beginners; normalize incomplete understanding. --- ### ✅ Logical Validity #### Strong reasoning - Solid causal explanation: type representations and ABIs are the reason FFIs and extension modules are complex. The author walks from C strings to register-level calling conventions stepwise. - Good analogical structure: shows the same call pipeline across dynamic linking, dynamic loading, and extension modules, making similarities and tradeoffs clear. #### Fallacies - None obvious; the article is descriptive and cautiously worded, noting uncertainty around Go and inviting corrections. #### Tone of voice - Tone is explanatory, humble, and collaborative: phrases like “hand-wavy pseudocode,” “I’ll try to answer these questions,” and “my understanding could be improved” set a non-authoritarian, learner-friendly mood. --- ### 🛡️ DARVO Analysis - Deny: No denial or minimization of others’ concerns; the piece foregrounds problems (phantom dependencies). - Attack: No attacks on people or groups; criticism is directed at technical limitations, not actors. - Reverse victim/offender: Not present; the author does not portray tool users or maintainers as victims of critique and instead discusses shared constraints. --- ## 🎯 Conclusion Binary dependencies sit at the intersection of performance, security, and sustainability. By unpacking dynamic linking, dynamic loading, and extension modules across multiple languages, the article equips you to reason about how your code actually calls into C, where costs and risks arise, and why manifest-level visibility is incomplete. Next steps: audit your own stack for hidden native libraries, improve documentation and SBOMs, and choose FFI strategies that balance ergonomics, performance, and safety. --- ## 🧱 Refactor Appendix (append after 🎯 Conclusion) ### 📚 Literature Split Pad - Purpose: Stage Konik CEE units that become per-source Literature notes. - Split level: **H3**. - Include verbatim quotes under 25 words and a [view highlight] link. ### [H1] Binary dependencies underpin multi-language ecosystems > [!quote] [How Binary Dependencies Work Across Different Languages](https://vlad.website/how-binary-dependencies-work/) by [[Vlad]] via [[Vlad's Website]]. Published 2026-01-16. Accessed . > “Binary dependencies are interesting because they most frequently occur as phantom dependencies — dependencies that we don’t know about.” Two lines: Use to motivate SBOM and supply-chain work in cross-language stacks. Helpful for security and OSPO discussions. [view highlight](https://vlad.website/how-binary-dependencies-work/#H1) #xref #articleseed ID:rw_[MISSING] ### [H2] Extension modules optimize cross-language data conversion > [!quote] [How Binary Dependencies Work Across Different Languages](https://vlad.website/how-binary-dependencies-work/) > “Because your extension module can create both Python/Ruby/Node.js data structures and C data structures, you can perform data conversion whenever is most performant.” Two lines: Use when arguing for moving hot loops into C via extension modules instead of naive FFIs. [view highlight](https://vlad.website/how-binary-dependencies-work/#H2) #xref ID:rw_[MISSING] --- ### 🔖 Atomic Split Pad (Zettel Candidates) - Purpose: Promote reusable claims into Atomic notes. - Split level: **H4**. - Each item: one idea, present tense, specific, testable. #### [Z1] Binary dependencies often remain phantom in manifests - Many applications use C libraries via FFIs or extensions that are not declared in package manifests. - Pointer: article sections on phantom dependencies and NumPy/OpenBLAS. - Links: [[Software-Supply-Chain]] [[Binary-Dependencies]] #### [Z2] Extension modules shift work across language boundaries - Extension modules let developers decide where to keep data in C vs. host types for performance. - Pointer: sections on dynamic linking with extension modules and data conversion. - Links: [[FFI-Design]] [[Performance-Tuning]] --- ### ✏️ Fleeting Queue (Extract by Selection) - Purpose: Capture quick actions or sparks for Fleeting notes. - Extraction: **Select one line** → “Refactor: Extract selection to new note (content only).” - Keep each line short and self-contained. - [F1] Audit my core projects for undocumented binary dependencies — context/https://vlad.website/how-binary-dependencies-work/ #fleeting #inbox - [F2] Prototype a demo comparing Python FFI vs. extension module performance — context/https://vlad.website/how-binary-dependencies-work/ #fleeting #inbox - [F3] Draft internal guideline on choosing between FFI and extension modules — context/https://vlad.website/how-binary-dependencies-work/ #fleeting #inbox ---