How to utilize profiles for AI use cases 🔗 **Source:** [How to utilize profiles for AI use cases](https://www.youtube.com/watch?v=v3MpcW05yvk) > [!note]+ Document Notes > tags: [AI, data-management, software-security] > author: Linux Foundation > source: YouTube > reference: https://www.youtube.com/watch?v=v3MpcW05yvk > date created: 2024-12-28 14:30 > > # 🏷️ Title: How to utilize profiles for AI use cases > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Document AI training and production datasets in SBOMs — standards compliance requirement #AI-governance - [H2] Implement monitoring systems for dynamic AI behavior changes — production safety measure #AI-monitoring - [H3] Create transparency requirements for AI model cards and datasets — auditing capability #AI-transparency ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [SPDX 3.0 AI Profiles](https://spdx.dev) - [Partnership in AI Database](https://partnershiponai.org) ## 📝 Summary - Industry experts are developing SBOM standards for AI applications to address the "Wild West" of AI licensing and security - AI applications are dynamic and require new fields in SPDX 3.0 including hyperparameters, data preprocessing, and model explainability - Current AI applications like ChatGPT lack transparency in documentation, creating audit risks for enterprise adoption - New monitoring approaches are needed since AI behavior changes continuously in production without traditional release cycles - The effort aims to bring the same vulnerability management benefits of SBOMs to AI systems ## 📖 Main Content A collaborative effort between IEEE, ISO, and NIST experts is developing standardized documentation requirements for AI applications through Software Bill of Materials (SBOMs). The initiative recognizes that AI applications present unique challenges compared to traditional software - they're dynamic, continuously learning systems that can change behavior in production without new releases. The working group has identified critical new fields needed for SPDX 3.0 to handle AI use cases, including documentation of training vs production datasets, hyperparameters (ChatGPT has 3 billion), data preprocessing techniques, and model explainability requirements. Current industry examples show major gaps - neither ChatGPT nor major datasets provide adequate transparency for enterprise auditing. The effort addresses incident reports that go beyond traditional cybersecurity to include ethical issues, bias problems, and transparency concerns. Recent examples include ChatGPT generating malicious code due to compromised open source libraries, and autonomous vehicle failures due to insufficiently diverse training data. ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=v3MpcW05yvk?t=0) — [H1] AI SBOM Standards Development Initiative - Key idea: Year-long collaboration between IEEE, ISO experts to document AI dataset fields - Evidence: "Team of about 10 to 12 individuals around for the last year trying to figure out how we could document the fields that you need for AI" #### ⏱️ [02:30](https://www.youtube.com/watch?v=v3MpcW05yvk?t=150) — [H2] AI Applications Are Dynamic Systems - Key idea: AI applications change behavior continuously without traditional release cycles - Evidence: "AI applications are not static, they're Dynamic and the whole behavior of an AI application can change" #### ⏱️ [04:15](https://www.youtube.com/watch?v=v3MpcW05yvk?t=255) — [H2] Current Industry Documentation Gaps - Key idea: Major tech companies create proprietary documentation instead of standardized formats - Evidence: "Google model cards, IBM fact checking cards - there needs to be a generic place where you go for the fields" #### ⏱️ [06:45](https://www.youtube.com/watch?v=v3MpcW05yvk?t=405) — [H3] AI Incident Reports Beyond Cybersecurity - Key idea: AI incidents include ethical issues, bias, and safety problems beyond traditional vulnerabilities - Evidence: "Incident report in the AI space can be cyber security, ethical like bias in data, automated car autopilot models kill people" #### ⏱️ [09:20](https://www.youtube.com/watch?v=v3MpcW05yvk?t=560) — [H2] Open Source AI Models Gaining Ground - Key idea: Open source AI models are outpacing proprietary ones according to leaked Google memo - Evidence: "Google leaked note says open sourced model is actually better than Bart which is their proprietary model" #### ⏱️ [12:10](https://www.youtube.com/watch?v=v3MpcW05yvk?t=730) — [H1] SPDX 2.3 Already Supports Basic AI Fields - Key idea: Core AI documentation capabilities exist in current SPDX version - Evidence: "In 2.3 we actually worked with Kate and the SPDX team to get the core things that an AI application would need" #### ⏱️ [14:30](https://www.youtube.com/watch?v=v3MpcW05yvk?t=870) — [H2] New SPDX 3.0 AI-Specific Fields - Key idea: Hyperparameters, data preprocessing, and model explainability fields being added - Evidence: "ChatGPT has three billion hyperparameter options, pre-process your data, model explainability aspect" #### ⏱️ [17:45](https://www.youtube.com/watch?v=v3MpcW05yvk?t=1065) — [H3] Real-World Dataset Transparency Assessment - Key idea: Current open datasets lack sufficient documentation for audit purposes - Evidence: "World dataset has very little information about build time, release time - transparency is low" #### ⏱️ [20:15](https://www.youtube.com/watch?v=v3MpcW05yvk?t=1215) — [H2] Data Collection and Processing Documentation - Key idea: New fields capture data sources, size, noise, biases, and sensitivity levels - Evidence: "How did you collect data, ChatGPT uses publicly available internet data, GM gave 2 billion rows of car data" #### ⏱️ [23:30](https://www.youtube.com/watch?v=v3MpcW05yvk?t=1410) — [H3] GPT-3 Documentation Gaps Analysis - Key idea: Major AI models fail to provide required transparency information - Evidence: "GPT-3 model card, all new data fields are red because they're not documented anywhere" #### ⏱️ [25:50](https://www.youtube.com/watch?v=v3MpcW05yvk?t=1550) — [H2] Custom AI Licensing Challenges - Key idea: AI models use novel licensing approaches that need standardization - Evidence: "New licenses associated with models like OpenAI, Microsoft owns all rights behind ChatGPT API" #### ⏱️ [28:20](https://www.youtube.com/watch?v=v3MpcW05yvk?t=1700) — [H2] Separating Models and Datasets - Key idea: Models and datasets must be documented separately due to reuse patterns - Evidence: "People use data for multiple models and models use multiple different datasets so we have to separate them" #### ⏱️ [30:45](https://www.youtube.com/watch?v=v3MpcW05yvk?t=1845) — [H3] Poisoned Dataset Remediation Challenge - Key idea: Need mechanisms to track and remove compromised training data from models - Evidence: Q&A discussion about reverting models when datasets are found to be "poisoned encrypted" #### ⏱️ [32:10](https://www.youtube.com/watch?v=v3MpcW05yvk?t=1930) — [H2] Continuous Monitoring Requirements - Key idea: AI systems need real-time monitoring with automated pullback capabilities - Evidence: "Tesla autopilot pulled off market after accidents, need monitoring system for metric anomalies" ## 📚 References - [SPDX Specification](https://spdx.dev/specifications/) - [IEEE AI Standards](https://standards.ieee.org/initiatives/artificial-intelligence-systems/) - [Partnership on AI Incident Database](https://incidentdatabase.ai/) - [AI Vulnerability Database](https://avidml.org/) ## 🗣️ Paraphrase - A working group of 10-12 experts spent a year developing SBOM standards for AI, recognizing that AI systems are fundamentally different from traditional software because they continuously evolve and learn in production - The team identified new documentation requirements for SPDX 3.0 including hyperparameters, data preprocessing methods, and explainability features, while finding that major AI systems like ChatGPT currently lack transparency needed for enterprise auditing - AI incident management extends beyond cybersecurity to include ethical issues and bias problems, requiring new monitoring approaches since AI behavior can change without traditional software releases ## 📰 Newsletter The WordPress community should pay attention to emerging AI documentation standards as they'll soon impact how we build and audit AI-powered features in WordPress. A collaborative effort between IEEE, ISO, and industry experts is developing SBOM (Software Bill of Materials) requirements specifically for AI applications through SPDX 3.0. This matters for WordPress because AI features are increasingly common in themes, plugins, and core functionality. The new standards will require documenting training datasets, model parameters, and bias assessments - going far beyond traditional dependency tracking. Key takeaway: AI systems are fundamentally different from traditional software because they continuously learn and change behavior in production. WordPress developers working with AI features should start thinking about transparency, monitoring, and documentation requirements now. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create AI documentation best practices presentations for WordCamps - Education: Develop workshops on AI transparency requirements for plugin developers - Developer experience: Draft AI feature documentation templates for WordPress projects - Developer marketing: Launch campaign about responsible AI development in WordPress ecosystem - Community management: Moderate discussions about AI ethics and bias in WordPress forums #### OSPO - Policy & Governance: Draft AI model documentation policies for WordPress contributions - License Compliance: Assess compatibility of AI model licenses with WordPress licensing - Security Management: Develop AI-specific vulnerability assessment procedures - Software Inventory: Create SBOM strategy for AI-powered WordPress components - Developer Support: Provide training on AI documentation requirements - Community Engagement: Engage with SPDX working group on WordPress-specific AI needs - Strategic Alignment: Align AI documentation KPIs with WordPress transparency goals #### Community - Maintainers: Develop governance templates for AI feature contributions - Contributors: Create onboarding materials for AI-powered plugin development - Sponsors: Fund development of AI transparency tooling for WordPress - Ecosystem: Coordinate AI documentation standards across major WordPress hosts and services #### WordPress - Core: Establish AI feature review criteria and documentation requirements - Core Performance: Profile performance impact of AI features and document in SBOMs - Core AI: Implement guardrails and evaluation frameworks for AI contributions - Plugins: Develop AI-specific review guidelines and security liaison protocols - Themes: Create AI feature disclosure requirements for theme directory - Documentation: Expand DevHub with AI development and documentation guidelines - Security: Establish AI-specific vulnerability reporting and remediation procedures - Accessibility: Ensure AI features meet WCAG requirements and document bias assessments #### The WPCC - Programs: Fund fellowships focused on AI ethics and transparency in WordPress - Projects: Support documentation sprints for AI feature transparency - Funding: Provide matched grants for AI monitoring and auditing tool development #### FAIR - Technical independence: Ensure AI model provenance tracking and documentation - Policies: Develop funding transparency requirements for AI-powered features - Governance artifacts: Create charters for AI feature governance and ethical review ## 🧪 Hypotheses - WordPress sites using AI features without proper documentation will face increased security and compliance risks - Standardized AI documentation requirements will initially slow AI feature adoption but improve long-term trust - Open source AI models will become more prevalent in WordPress ecosystem due to transparency advantages - Continuous monitoring requirements for AI features will drive development of new WordPress tooling ### 📊 Methodology - Survey WordPress plugin and theme developers about current AI documentation practices - Analyze existing WordPress AI features for SPDX 3.0 compliance gaps - Pilot SBOM generation for popular AI-powered WordPress plugins - Measure impact of documentation requirements on development velocity ### ❓ Questions - How will WordPress handle AI features that continuously change behavior without plugin updates? - What liability do plugin developers have for undisclosed bias in AI training data? - Should WordPress core include built-in AI monitoring and transparency features? ### 🔮 Future Research - Impact of AI documentation requirements on WordPress plugin adoption rates - Effectiveness of automated AI bias detection in WordPress content management - Integration patterns between WordPress and external AI model registries ### 💡 Implications - Practice: WordPress developers must learn AI documentation standards and bias assessment - Policy: WordPress.org may need new review criteria for AI-powered plugins and themes - Security: AI-specific vulnerability classes require new detection and remediation approaches - Education: WordPress training curricula must include AI ethics and transparency modules - Funding: Resources needed for AI monitoring infrastructure and documentation tooling ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade #### 👀 Visual learners - Present: Referenced datasets and model card examples - Improve: Add SBOM structure diagrams, AI pipeline visualizations #### 🔊 Aural learners - Present: Video presentation with clear narration - Improve: Provide 5-minute audio summary of key documentation requirements #### 🗣️ Verbal learners - Present: Technical definitions of SBOMs, hyperparameters, model cards - Improve: Add analogies comparing AI documentation to traditional software documentation #### 🧩 Kinesthetic learners - Present: References to hands-on AI auditing exercises - Improve: Create step-by-step SBOM generation tutorial for WordPress AI features #### 🧠 Logical learners - Present: Clear cause-effect relationships between documentation gaps and risks - Improve: Add decision trees for AI feature documentation requirements #### 🧑‍🤝‍🧑 Social learners - Present: Collaborative working group approach to standards development - Improve: Include discussion prompts for WordPress community AI ethics conversations #### 🧘‍♀️ Solitary learners - Present: Technical specifications and individual assessment frameworks - Improve: Add self-check quizzes for AI documentation completeness #### 🌿 Naturalistic learners - Present: Ecosystem thinking about AI standards and industry collaboration - Improve: Connect AI documentation to broader software sustainability practices ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure and technical term definitions - Improve: Add ARIA labels for technical diagrams, provide alt text for model examples #### Keyboard navigation - Present: Standard video controls accessible via keyboard - Improve: Ensure transcript is keyboard navigable with skip links #### Contrast and readability - Present: Standard video player with adequate contrast - Improve: Provide high-contrast transcript option #### Text resizing and reflow - Present: Responsive video player - Improve: Ensure technical diagrams scale appropriately #### Media and captions - Present: Video content with transcript - Improve: Add closed captions and audio descriptions #### Forms and interactions - Present: Basic video controls - Improve: Add accessible bookmark and note-taking features #### Standards reference - Meets WCAG 2.2 AA requirements for video content with transcript provision ### 🌍 DEIB Assessment #### Cultural breadth - Present: References to global standards organizations (IEEE, ISO) - Improve: Include examples from diverse geographic regions and AI applications #### Inclusive language - Present: Technical language with definitions provided - Improve: Add plain-language summaries, avoid technical jargon <!-- deep-read:sections end -->