Track 2Reference
Deepdive.
The mechanics, the risks, and the frameworks, with sources. Written for practitioners who have to make AI work safely in production.
D1 · Mechanics
How large language models work
Tokens and next-token prediction
LLMs break text into tokens (word fragments, roughly three-quarters of an English word on average) and are trained to predict the next token given everything before it. Generation is that prediction repeated: pick a token, append it, predict again.
Training in stages
- Pre-training on a very large text corpus teaches general language and world patterns. This is where most of the cost and most of the "knowledge" comes from.
- Post-training (instruction tuning and reinforcement learning from human or AI feedback) shapes the model into a helpful assistant that follows instructions and declines certain requests.
The context window
Everything the model can "see" at once (system instructions, your conversation, attached documents, tool results) lives in its context window. Anything outside it doesn't exist to the model. Most practical control over an LLM happens here.
Grounding: retrieval and tools
Retrieval-augmented generation (RAG) searches your approved documents and places the relevant passages in the context so the model answers from them. Tool use lets a model call functions such as search, databases, or APIs. Both reduce guessing, and both widen what the model can reach, which is why they need access controls.
Sampling
Settings like temperature control how much randomness goes into choosing each token. Lower values make output more repeatable; they don't make it more true.
D2 · Reliability
Why models make things up
A model that is trained to always produce a likely continuation will produce one even when it has no reliable basis. Research published by OpenAI in September 2025 argues that hallucinations persist partly because common training and evaluation methods reward guessing over admitting uncertainty: on a scoreboard that only counts right answers, a confident guess beats "I don't know."
What reduces hallucination in practice:
- Ground answers in approved sources (RAG) and require citations you can click.
- Give the model explicit permission to say "not enough information."
- Constrain the output format (fields, choices, schemas) for anything a system will act on.
- Evaluate on your own tasks with known answers, not only on public benchmarks.
- Keep a human review step wherever an error would be costly or public.
D3 · Security
Security risks: the OWASP LLM Top 10
The OWASP Top 10 for LLM Applications (2025 edition) is the most widely used checklist of what goes wrong when LLMs are built into real systems:
| ID | Risk | What it means in practice |
|---|
| LLM01 | Prompt injection | Instructions hidden in user input, documents, emails, or web pages hijack the model. |
| LLM02 | Sensitive information disclosure | The model reveals confidential data it was given or trained on. |
| LLM03 | Supply chain | Compromised models, datasets, plugins, or dependencies. |
| LLM04 | Data and model poisoning | Tampered training or fine-tuning data that plants bad behavior. |
| LLM05 | Improper output handling | Model output passed unchecked into code, queries, or browsers. |
| LLM06 | Excessive agency | An agent has more permissions, tools, or autonomy than the task needs. |
| LLM07 | System prompt leakage | Hidden instructions (and any secrets in them) get exposed. |
| LLM08 | Vector and embedding weaknesses | Flaws in the retrieval layer leak or poison what the model reads. |
| LLM09 | Misinformation | Confident, wrong output that people or systems rely on. |
| LLM10 | Unbounded consumption | Runaway usage that drives cost or denies service. |
Two of these deserve special attention as agents spread. Prompt injection has no complete fix today, so design on the assumption that anything the model reads could try to give it orders. Excessive agency is the control you fully own: least privilege, scoped credentials, approval gates for high-impact actions, and rate and spend limits.
D4 · Governance
Governance frameworks and regulation
NIST AI Risk Management Framework
Released January 26, 2023, the voluntary NIST AI RMF organizes AI risk work into four functions: Govern (culture, roles, policy), Map (context and intended use), Measure (testing and metrics), and Manage (prioritize, respond, monitor). NIST added a Generative AI Profile (NIST AI 600-1) on July 26, 2024, and has said the framework is being revised.
ISO/IEC 42001
Published December 18, 2023, ISO/IEC 42001 is the first certifiable standard for an AI management system: the policies, roles, risk assessments, and continual-improvement loop for organizations that build or use AI. Teams with ISO 27001 experience will recognize the structure.
The EU AI Act
The EU AI Act (Regulation (EU) 2024/1689) sorts AI by risk: some uses are prohibited, "high-risk" uses carry heavy obligations, and many systems carry transparency duties. In 2026 the EU adopted a Digital Omnibus on AI (Regulation (EU) 2026/1744) that moved key deadlines:
- High-risk systems listed in Annex III (such as employment and credit uses): December 2, 2027, instead of August 2, 2026.
- High-risk AI in products covered by EU product law (Annex I): August 2, 2028.
- Transparency marking for synthetic content from systems already on the market: December 2, 2026.
If you sell into or operate in the EU, check these dates with counsel; regulations and guidance continue to evolve.
D5 · The numbers
Where adoption really stands
McKinsey's State of AI survey (November 2025) captures the gap between using AI and getting value from it:
- 88% of respondents say their organization uses AI in at least one business function, up from 78% a year earlier.
- Only about one-third say their organization has begun scaling AI across the enterprise.
- 62% are at least experimenting with AI agents; 23% are scaling agents somewhere in the business.
- 39% report any enterprise-level EBIT impact from AI, and most of those attribute less than 5% of EBIT to it.
Nearly everyone has AI. Far fewer have it under management. That gap, not the model, is where the value is lost.
D6 · Operating model
The AI control stack
This is the operating model we use with clients. It maps to the NIST functions and fits inside an ISO 42001 management system, but it's written for the people who run the tools day to day.
| Layer | Control | Evidence you can show an auditor |
|---|
| 1. Inventory | Every AI tool, model, and agent in use, including "shadow" tools staff adopted on their own. | AI register with owner, purpose, data, vendor |
| 2. Risk tiering | Classify each use by impact: internal drafting vs. customer-facing vs. automated decisions. | Tier per use case and its required controls |
| 3. Access and agency | Least-privilege data access, scoped credentials, tool allow-lists, spend limits. | Permission matrix; approval gates for high-impact actions |
| 4. Evaluation | Test on your own tasks before launch and after every model or prompt change. | Test set, pass thresholds, change log |
| 5. Human oversight | Defined review points, sampling rates, and escalation paths. | Review records and sign-offs |
| 6. Monitoring | Quality, drift, cost, security events, and user feedback in one dashboard. | KPIs with alert thresholds |
| 7. Incident and retirement | A playbook for AI failures and a clear trigger to retune or retire. | Incident log; decommission records |
The pattern behind all seven: treat an AI system like a new team member with superpowers and no judgment. Give it a manager, a job description, limited keys, a probation period, and a performance review.
D7 · Reference
Glossary
- Machine learning
- Software that learns patterns from data rather than following hand-written rules.
- Large language model (LLM)
- A model trained on large amounts of text to predict and generate language.
- Generative AI
- AI that creates new content: text, images, audio, code, or video.
- Token
- The unit of text a model reads and writes; usually part of a word. Usage is often billed per token.
- Context window
- The amount of text a model can consider at once, including instructions, documents, and conversation.
- Prompt
- The instructions and input you give a model. A system prompt is the standing instruction set behind an app.
- Hallucination
- Confident output that is false or unsupported.
- RAG
- Retrieval-augmented generation: fetching relevant approved documents and giving them to the model to answer from.
- Fine-tuning
- Further training a model on specific examples to adjust its behavior for a task.
- AI agent
- An AI system that can plan and take actions with tools, often over multiple steps.
- Human in the loop
- A required human review or approval step before output or action takes effect.
- Prompt injection
- An attack where malicious instructions in input or content take over a model's behavior.
- Model drift
- Degrading or changing results over time as data, usage, or the model itself changes.
- AI governance
- The policies, roles, and controls that decide how AI is chosen, used, monitored, and retired.
Sources
Where these facts come from
- https://oecd.ai/en/wonk/definition
- https://openai.com/index/why-language-models-hallucinate/
- https://genai.owasp.org/llm-top-10/
- https://www.nist.gov/itl/ai-risk-management-framework
- https://www.iso.org/standard/42001
- https://www.orrick.com/en/Insights/2026/07/EU-AI-Act-Update-Digital-Omnibus-Finalizes-8-Compliance-Changes
- https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-2025
Last reviewed October 12, 2026. This page is educational and is not legal advice.